Spoken communication is accompanied by a wealth of contextual information, including both sensory informa- tion external to the listener (e.g., mouth movements and manual co-speech gestures) and knowledge or expec- tations internal to the listener (e.g., discourse context). Most research on speech perception discards context in favor of studying isolated speech sounds or words. Yet, preliminary studies suggest that the brain makes use of observed contextual information (Skipper, Wassenhove, et al., 2007) and that the cortical networks or pathways involved in language comprehension, are not fixed, but reorganize themselves depending on the type of context available to listeners (Skipper, Goldin-Meadow, et al., 2007).

    The long-term objective of this proposal is to understand the neural mechanisms of language comprehension in real-world settings, in which the brain can use context to aid in communication. This objective is pursued by testing hypotheses derived from a model of communication built on the assumption that context is central to the perception of communicative acts (Skipper et al., 2006, Section B.2). The hypotheses are 1) the brain uses context to predict subsequent speech sounds or words, 2) there are multiple pathways associated with different contexts whose relative weightings fluctuate as a function of informativeness, and 3) these weightings are built up differentially over development as learners acquire perceptual and cognitive abilities related to speech.

    These hypotheses are tested with respect to observed movements and preceding discourse contexts in both children and adults who undergo both EEG and fMRI while viewing and listening to a naturalistic stimulus (Aim 1) or controlled stimuli (Aims 2 and 3). The mentored phase will focus on completion of Aim 1 and pilot work on Aims 2 and 3 with the goal of learning to conduct research with children and EEG. The goal of the independent phase is to establish a lab centered on using real-world stimuli to understand how the brain makes use of context in communication throughout the lifespan. This will lead to an active program of research that can expand to include a broader range of contexts such as emotional and social/cultural expectations and eventually encompass translational issues relating to processing of different contexts in communications disorders.

Participants will watch a dialogue between two individuals engaged in real-world conversation. A novel com- bination of analysis tools (Section C.1) and extensive annotation of the stimulus (Section D.1.a) will be used to discover patterns of activity in brain networks that are tuned to spoken communication, and modulated by con- text. We predict that increased informativeness of one type of context will result in greater tuning and stronger weighting of pathways associated with that source of information. Thus, when mouth movements are informative with respect to phoneme identification, sensory-motor pathways will be tuned and weighted strongly. Increased informativeness of meaningful co-speech gestures or discourse context will increase tuning and weighting of sensory-semantic pathways. We expect children to rely on the sensory-motor pathways less than adults when mouth movements are informative because they are less prone to visual speech influence (Section B.3.d).

Both Aims 2 and 3 use repetition suppression and “relative informativeness” designs. It is hypothesized that, if gestures predict forthcoming speech sounds or words, response suppression will be greater for a word preceded by an informative gesture than an uninformative gesture. When mouth movements are informative, sensory- motor pathways will be weighted more strongly, whereas increased informativeness of meaningful co-speech gestures will increase weighting of sensory-semantic pathways. Relative to adults, when mouth movements are informative children are expected to weight sensory-motor pathways less strongly.

Informative discourse contexts are used to predict subsequent words connoted by that context. It is hypoth- esized that response suppression will be greater for a word preceded by an informative discourse context. Fur- thermore, informative discourse contexts are hypothesized to increase weightings in sensory-semantic pathways. In the presence of the realistic audiovisual stimuli of human communication, less informative discourse contexts should increase reliance on mouth movements, thus weighting sensory-motor pathways more strongly in adults, but less so in children.