The linguistic similarity hypothesis states that it is more difficult to segregate target and masker speech when they are linguistically similar (Brouwer et al., 2012). This may be the result of energetic masking (interference at the auditory periphery) and/or informational masking (cognitive interference). To provide a rigorous test of the hypothesis and investigate how informational masking interferes with speech identification in the absence of energetic masking, we presented target speech visually and masking babble auditorily. Participants completed an English lipreading task in silence, speech-shaped noise, semantically anomalous English, semantically meaningful English, Dutch, and Mandarin two-talker babble. Results showed that speech maskers interfere with lipreading more than stationary noise, and that maskers that are the same language as the target speech provide more interference than different-language maskers. However, the study found no evidence that a masker that is similar to the English target speech (Dutch) provides more masking than one that is less similar (Mandarin). These results provide some cross-modal support for the linguistic similarity hypothesis, but suggest that the theory should be further specified to address the conditions under which languages that differ in their similarity to the target speech should provide different levels of masking.