Humans use all possible information to gain knowledge of the outside word. The audiovisual speech perception is a good example of this principle. In understanding speech, we see what is difficult to hear and hear what is difficult to see. The syllables pa and ka are acoustically very similar which makes them difficult to separate, e.g., when talking on the telephone. Visually they are quite different, which becomes evident by having a look at the mirror when pronouncing pa and ka. On the other hand, the voiced za and voiceless sa cannot be differentiated by vision but they are easy to separate by hearing.
An exciting example about the influence of seeing on hearing is the "McGurk effect". If the auditory syllable pa is led to the subjects ears but he simultaneously sees the speakers lips to pronounce ka, the subject reports that he heard ta. This perception is clearly auditory in nature and it does not change even when the subject learns that he has been cheated. Almost all adults experience this illusion. However, the effect is not inborn but learned because it is not present in pre-school children.
McGurk illusion provides an interesting possibility to study the perceptual mechanisms in humans because the phenomenon cannot be explained solely on the basis of either auditory or visual features of the stimulus: perception is constructed utilizing information from both modalities. We have used MEG to study in which brain area the heard and seen speech are integrated. The subjects listened to pa syllables when they simultaneously saw the lips to pronounce either pa or ka; the auditory perception was pa and ta. The activation of the auditory cortex was different during these two perceptions even though the acoustical features of the stimulus remained the same. The results show that seeing affects information processing in the auditory cortex.
