- 1 Clinic for Neurology, Universitätsklinikum Schleswig-Holstein, Campus Lübeck, Lübeck, Germany
- 2 Department of Cognitive Neuroscience, Maastricht University, Maastricht, Netherlands
- 3 Department of Neurology, Otto-Von-Guericke University, Magdeburg, Germany
- 4 Center for Behavioral Brain Sciences, Magdeburg, Germany
Pronouns are bound to their antecedents by matching syntactic and semantic information. The aim of this functional magnetic resonance imaging study was to localize syntactic and semantic information retrieval and integration during pronoun resolution. Especially we investigated their possible interaction with verbal working memory manipulated by distance between antecedent and pronoun. We disentangled biological and syntactic gender information using German sentences about persons (biological/syntactic gender) or things (syntactic gender) followed by congruent or incongruent pronouns. Increasing the distance between pronoun and antecedent resulted in a short and a long distance condition. Analysis revealed a language related network including inferior frontal regions bilaterally (integration), left anterior and posterior temporal regions (lexico-semantics and syntactic retrieval) and the anterior cingulate gyrus (conflict resolution) involved in pronoun resolution. Activities within the inferior frontal region were driven by Congruency (incongruent > congruent) and Distance (long > short). Temporal regions were sensitive to Distance and Congruency (but solely within long distant conditions). Furthermore, anterior temporal regions were sensitive to the antecedent type with an increased activity for person pronouns compared to thing pronouns. We suggest that activity modulations within these areas reflect the integration process of an appropriate antecedent which depends on the type of information that has to be retrieved (lexico-syntactic posterior temporal, lexico-semantics anterior temporal). It also depends on the overall syntactic and semantic complexity of long distant sentences. The results are interpreted in the context of the memory–unification-control model for sentence comprehension as proposed by Vosse and Kempen (2000), Hagoort (2005), and Snijders et al. (2009).
Introduction
Reading (and listening) for comprehension requires the access and the integration of syntactic and semantic information. Personal pronouns (for simplicity, from now on pronouns) are well suited to investigate the involved memory processes for retrieval and integration as they are linguistic devices, which refer back to a person or a thing mentioned earlier in discourse. Pronouns create coherence in written or spoken language. Considering the sentence “The woman is smiling because she is happy.” it is easy to tell that she refers to the woman. Building up co-reference between the pronoun and the antecedent noun may be based on syntactic and/or biological/semantic gender information. The woman in our example – a human living antecedent – is characterized by female biological gender information. In many languages including German this biological/semantic gender information is accompanied by feminine syntactic gender information [diefeminineFraufeminine/female (the woman)… sie(she)]1. In contrast to English, things are also characterized by (arbitrary) syntactic gender information without semantic reflex [diefeminineJackefeminine (the jacket)… sie(she)] in German, which becomes apparent in the article “die”2. These characteristics of the German language enabled us to systematically manipulate the process of syntactic and biological/semantic gender information integration during pronoun processing (Hammer et al., 2007; see also Hammer et al., 2005). In prior studies using event-related potentials (Schmitt et al., 2002; Hammer et al., 2005, 2008; Lamers et al., 2006, 2008) and functional magnetic resonance imaging (fMRI; Hammer et al., 2007) we tested whether co-referencing could be based on semantic or syntactic information, or both. In pronoun processing, the antecedent needs to be kept activated in working memory and has to be retrieved upon the encounter of a pronoun. As pronouns do not always appear in the same distance from antecedent, this suggests that working memory demands may be important in pronoun processing and co-referencing (Streb et al., 2004; Hammer et al., 2008; Li and Zouh, 2010). Consider the sentence “The woman is smiling and started to sing because she is happy.” By the insertion of the additional words the pronoun is moved further away from the antecedent and therefore this should tax memory processes to a greater extent than the shorter version of the sentence given above. However, it is unclear whether the higher load is related to a more difficult integration (i.e., unification) or to a more difficult retrieval. Modulation of integration should be visible in left inferior frontal regions, where as modulation of retrieval should be visible in the left posterior middle temporal gyrus (lpMTG; Snijders et al., 2009). The antecedent type and underlying differences in semantic retrieval (for example biological gender) might lead to variation along the lMTG. Here, person antecedents should amplify semantic processing related to the inherent active role of a person (agent) and should lead to a more anterior activation along the lMTG (Martin and Chao, 2001, and for, e.g., face recognition of familiar faces, Gainotti, 2007; Gainotti et al., 2010).The main goal of the present event-related fMRI study was to functionally disentangle integration and retrieval associated with pronoun processing.
Based on Vosse and Kempen (2000), Hagoort (2003, 2005), and Snijders et al. (2009) proposed the memory, unification, and control (MUC) model for language processing, which can be thought of as a neurally specified version of the Vosse and Kempen (2000) model. According to this model each word within the linguistic memory (lexicon) is associated with a structural frame that specifies its possible structural environment. All elements within the frame unify with each other to form a comprehensive sentence. Unification is a dynamic process over time. It is considered as a recursive process working incrementally according to the immediacy principle, i.e., integrating information of any linguistic type when possible. It integrates information by retrieving good candidates from the lexicon and linking them together. These could be syntactic (NP frame), semantic (biological gender), pragmatic, and phonological information. An important assumption in the MUC model is that unification links undergo a gradual decay over time which can be thought of as a decay in verbal working memory. As root nodes (single phrasal node, NP) have a syntactic function, this decay suggests that any type of syntactically based unification can only be carried out within the sentence. It eventually can span across a certain range of NPs only, but not across sentences borders. Neurobiologically, the relevant components of the MUC have been assigned to the left mid temporal cortex (Memory retrieval), left inferior frontal gyrus (IFG; Unification; for a more detailed discussion see Snijders et al., 2009), and the left dorsolateral prefrontal cortex and the anterior cingulate cortex (ACC; Control). Because the MUC model allows syntactic and semantic information to be used in unification and also features a memory decay function with functionally separable brain regions, we chose to use it as a framework for our study. More specific cognitive models outline the details of pronoun processing (Garrod and Sanford, 1994; Graesser et al., 1997; Gordon and Hendrick, 1998) but do not make direct predictions of the influence of differential working memory demands on semantic and syntactic processing during anaphoric resolution or to related brain areas yet.
Previous work of our group demonstrated that pronoun processing involves a highly dynamic spatio-temporal integration of syntactic and biological information depending on the type of the antecedent and whether or not a violation is involved. Hammer et al. (2007) used event-related fMRI to investigate the neural basis of biological and syntactic gender integration during pronoun processing in German sentences about persons or things. Overall, syntactic processing activated areas adjacent to Broca’s area (BA 44), whereas processing of the biological gender information, in addition, involved the supramarginal gyrus (BA 40). A previously reported event-related potential study with an identical paradigm (Hammer et al., 2005) revealed that syntactic and semantic information is integrated 400–700 ms after target onset, visible in both cases as a P600 but with different effect sizes (i.e., an increased P600 for person pronoun as compared to thing pronouns). Overall, this data suggested that pronoun processing undergoes dynamic context-dependent changes in neural circuits. One such context is the type of the antecedent for a particular pronoun (person or thing).Furthermore, using the distance manipulation between a pronoun and its corresponding antecedent in an ERP experiment (Hammer et al., 2008) showed that unification is rather based on syntactic rules when the antecedent is relatively non-salient (thing antecedent: inanimate, no biological gender) and closer to the pronoun [short distance (SD)]. In turn, unification is rather based on semantic grounds when the antecedent is salient (person antecedent: animate, biological gender) and when the distance between antecedent and pronoun is rather enlarged (decay of syntactic root nodes increased with increasing load in working memory).
In the present study, we extended our earlier work and sought (i) to localize brain areas corresponding to semantic and/or syntactic retrieval and integration and (ii) to ask whether these would interact with verbal working memory demands. We used a word-by-word reading task in a standard violation paradigm to avoid effects of explicit syntactic decisions. As in the preceding experiments, our paradigm allows us to investigate syntactic and semantic information processing based on the gender type of the antecedent (Hammer et al., 2005, 2007). In one condition a person was introduced as an antecedent and later referred to by a pronoun, which either agreed in biological/syntactic gender or not (biological/syntactic gender violation). In a second condition, a thing was introduced as an antecedent and the corresponding pronoun either agreed in syntactic gender with the antecedent or not (syntactic gender violation). In addition, we manipulated verbal working memory in pronoun processing for both linguistic conditions by increasing the distance between antecedent and pronoun (for a corresponding ERP study see Hammer et al., 2008). Thus, the distance was either short (short distance = SD condition) or long [long distance = LD condition, see Table 1 for example material]. The distance manipulation effects the syntactic integration and retrieval (LD = more complex syntactic structure for all pronoun types). In addition, it also increases semantic integration and retrieval (more complex semantic retrieval for person compared to thing antecedent). Hence, the manipulation allowed us to disentangle differential processing for semantic and syntactic processing within long distant sentences compared to short distant sentences and thus, a possible interaction between syntactic and semantic gender information processing with verbal working memory. Altogether this resulted in a 2 × 2 × 2 design with Distance (long vs. short), Antecedent (person vs. thing), and Congruency (incongruent vs. congruent) as repeated measures factors.
Table 1. Example materials for the experiment.
Based on earlier findings we expected an activation pattern associated with overall sentence-processing including inferior frontal regions for integration/unification (Hagoort, 2003, 2005; Snijders et al., 2009), here unifying syntactic or syntactic plus semantic gender information. This region has previously been described as sensitive to memory load (Fiebach et al., 2001; Cooke et al., 2002; Newman et al., 2002; Just et al., 2004) and to syntactic processing (Carpenter et al., 1999; Keller et al., 2001; Cooke et al., 2002, 2006; Fiebach et al., 2005; Santi and Grodzinsky, 2007). Based on Snijders et al. (2009), we expected an activation modulation in lpMTG (memory retrieval for lexico-syntactic information, here distance manipulation) and within laMTG (lexico-semantic information, here antecedent type manipulation). Focusing on MTG as target region for semantic retrieval integrates findings by others (Gabrieli et al., 1998; Martin and Chao, 2001; Thierry et al., 2003; Fiebach et al., 2007).
Materials and Methods
Subjects
Ten German native speakers (six females, mean age 25.1 ± 3.4) gave written consent to participate for a small monetary compensation. All had normal vision, were right-handed and neurological healthy. All procedures were undertaken with the understanding and written consent of each subject and were approved by the ethics committee of Magdeburg University.
Measurements
Functional measurement: BOLD dependent fMRI were obtained using a 3-Tesla Siemens Magnetom Trio Vision system (Siemens, Erlangen) equipped with an eight channel phased array head coil. The functional images were acquired with a gradient echo EPI sequence (TR = 2 s, TE = 30 ms, FOV = 220 mm × 220 mm, flip angle = 80°, matrix size = 64 × 64, in-plane resolution 3.4375 mm × 3.4375 mm, slice thickness = 3.5 mm, interslice gap 0.35, 30 slices oriented parallel to the AC–PC-line, specified with a midsagittal scout image). One functional run comprised 550 volumes. Four functional runs were acquired in total. In order to avoid a T1 saturation effect we did not present any material during the first seven volumes and excluded the first four volumes from further analyses.
Anatomical measurement
A high-resolution T1 weighted 3D-MPRAGE image was acquired as anatomical reference (TR = 1800 ms, TE = 3.44 ms, flip angle = 7°, FOV = 256 mm, matrix size = 256 × 256, 192 sagittal slices, in-plane resolution 1 mm × 1 mm, slice thickness = 1 mm).
Material and Design
According to a 2 × 2 × 2 repeated measures design, eight types of sentences were presented to the subjects as summarized in Table 1. We used 64 sentences per condition. Each sentence had two clauses. The first clause in each sentence was the main clause, which described a state of a person or a thing. The person or the thing was the subject of the main clause, and formed the antecedent of the pronoun imbedded in a subsequent clause. Care was taken to match word frequencies for persons (232 ± 568) and things (229 ± 599, t126; two-tailed = 0.02, p = 0.98) using the CELEX-database (Baayen et al., 1995). The second clause was a subordinate clause introduced by the conjunction weil (because). This conjunction was followed by the critical word, a pronoun referring to the person or the thing. All 128 sentences were then copied and the congruent pronoun was replaced by an incongruent pronoun, resulting in 256 sentences. From these SD sentences, long distance (LD) sentences were created by adding four more words between antecedent and pronoun, resulting in 512 different sentences in total. The factors of the 2 × 2 × 2 design were labeled Distance (SD/LD), antecedent type (person/thing P/T), and congruency (congruent/incongruent C/I). To minimize repetition, the sentences were distributed across two different lists, counterbalancing antecedent type, congruency, and distance. Sentences on a list were then pseudo-randomized over four blocks (eight sentences per condition in one run) in such a way that repetitions of antecedents were kept apart as far as possible. We measured five subjects per list. They were pooled again later for analysis.
Procedure
Subjects were asked to read the sentences carefully, as if they were supposed to answer questions concerning the content of the sentences later. They should fixate on the screen and avoid possible movements during scanning. Before entering the scanner, subjects performed a training sequence, in which they read seven sentences similar to the experimental ones. Words were presented white on black background. A sentence trial always started with the first word of a sentence. The beginning of a trial was time-triggered to the sixth main trigger of the scanner following the preceding sentence trial. In between sentences an asterisk was presented as fixation point. Thus, the experiment was a slow event-related design, allowing the BOLD response to settle down to baseline in between trials. We employed a word-by-word presentation in order to avoid eye-movements associated with free-field sentence reading and to control sentence-processing time. Each word was presented for 350 ms with a 250-ms inter-stimulus interval. A word with a period was the terminal word of a sentence.
Each scanning session started with a scout image to obtain position information. Right after that, two functional scans (550 volumes, 64 sentences, 8 sentences per condition) were performed followed by the structural scan allowing subjects to rest. Subsequently, the remaining two functional runs were performed. The entire experiment lasted about 80 min.
Image Analysis
Image analysis was performed using BrainVoyager QX software (Brain Innovation B.V., Maastricht, The Netherlands). Prior to data analysis, all images were corrected for motion [parameters were not added as regressors in the general linear model (GLM)] and slice-scan time order, co-registered with the subjects’ corresponding anatomical (T1-weighted) images, normalized into standard coordinate system (Talairach and Tournoux, 1988), and spatially smoothed using a 8-mm full-width-at-half-maximum Gaussian kernel. Additionally, linear drifts were removed from the signal and data were high-pass filtered to remove slow frequency drifts up to three cycles per time course. Furthermore, surface rendering, and cortex reconstruction were performed.
For multiple regression analysis of the functional data, a random effects GLM with predictors for each experimental condition (SD congruent Person, SD incongruent Person, SD congruent Thing, SD incongruent Thing, LD congruent Person, LD incongruent Person, LD congruent Thing, LD incongruent Thing) was computed. Onset times of regressors (convolved with a two gamma HRF) were determined by the time the critical pronoun appeared on the screen. The rest of the sentences were defined as regressors, too, but were not included in later analysis. Fixation periods served as baseline. We applied a random effects analysis using single-factor repeated measures ANOVA including all pronoun predictors (eight levels corresponding to the different conditions). Thresholding was controlled by false discovery rate (FDR) at 5% and c(V) = 1 (Genovese et al., 2002). In addition, activated clusters were only accepted if more than 50 voxels were significantly activated. All reported activations are based on group statistics. To assess differences between conditions within regions of interest (ROI; as revealed by the RFX–ANOVA) we performed a 2 × 2 × 2 ANOVA (Distance × Antecedent × Congruency). This analysis was followed by planned pair-wise comparisons (see Results).
Results
The overall analysis revealed a left lateralized fronto-temporal network [including IFG bilaterally, superior temporal gyrus (STG), and pMTG; see Figure 1]. Additionally we found a modulation within the ACC. The details of activated regions are summarized in Table 2. The BOLD responses from these regions quantified as percent signal change at 6 s after stimulus onset were subjected to ANOVAs including the factors Distance (long vs. short), Antecedent (Person vs. thing), and Congruency (incongruent vs. congruent) in order to investigate main effects of Distance, Antecedent, and Congruency and possible interactions of verbal working memory with syntactic processing and meaning integration (see Table 3 for statistics and Figure 2 for bar graphs of the percentage BOLD signal). All regions revealed a Distance main effect confirming the involvement of memory processes during pronoun resolution. An antecedent type main effect was found within the STG and the ACC. A Congruency main effect was found for IFG bilaterally, pMTG, and ACC. We found a significant Distance × Antecedent × Congruency interaction within ACC. No further interactions were found.
Figure 1. Cortical statistical map as revealed by the full ANOVA analysis [F(7,63)]. Details of activated regions are given in Table 2. For corresponding signal changes see Figure 2.
Figure 2. Diagrams show the percentage signal change of the BOLD signal in regions of interest at 6 s after pronoun onset. Error bars indicate the SE.
Table 2. Ddetails for regions of interest.
Table 3. Summary of region of interest analysis.
Direct comparisons within the left IFG revealed that the main effect of Distance is based on the increased activity of long distant pronouns as compared to close pronouns referring back to congruent and incongruent Person and congruent Thing antecedents (see Figure 2 and Table 4). Incongruent as compared to congruent pronouns revealed an increased activity (see Figure 2). Statistically this was confirmed by direct comparisons for short distant Person and Thing pronouns and long distant Person pronouns. This demonstrates a crucial role of the left IFG in pronoun processing. Activity within the right IFG showed a similar pattern but less pronounced: there was an overall increase for long distant sentences compared to short distant pronouns and for incongruent compared to congruent pronouns (see Figure 2). Direct pair-wise comparisons of the BOLD response revealed a difference between long and short distant congruent Person pronouns only.
Table 4. Ddirect comparison.
The STG was sensitive to the Distance and Antecedent manipulation. As can be seen in Figure 2, LD sentence revealed increased activity compared to SD sentences. Direct comparisons confirmed this effect for the incongruent Thing sentences and a trend for incongruent Person sentences. More important, pronouns referring to person antecedents showed increased activity compared to thing antecedents. This effect was more pronounced for the short distant condition. The pMTG showed a general increase for long vs. short distant pronoun conditions and a congruency (incongruent > congruent) effect which was more pronounced for the long distant Person conditions (see Figure 2). Thus, both temporal regions were driven by verbal working memory processes (LD > SD). However, the STG was additionally driven by the Antecedent processing (Person > Thing) and the pMTG by congruency (incongruent > congruent).
As listed in Table 3, the analysis for the ACC showed main effects for distance, antecedent type, and congruency and an interaction between these three factors. The ACC showed less activation for long vs. short distant conditions (Figure 2). The decrease in long distant conditions was more pronounced for person compared to thing conditions. The congruency effect can be clearly seen in Figure 2. Pair-wise comparison revealed that the ACC responded to congruency in the SD person pronouns (incongruent more negative than congruent), but not to SD thing pronouns or long distant pronouns.
Discussion
The aim of this study was to investigate which brain areas are involved in different aspects of pronoun processing, specifically semantics, syntax, and verbal working memory and their interactions. Building up co-reference between a pronoun and its antecedent was manipulated based on purely syntactic information (antecedent type thing) or combined syntactic/semantic expectancies (antecedent type person) in a violation paradigm (syntactic and syntactic/semantic congruency). Working memory was varied by manipulating the distance between antecedent and pronoun.
The overall analysis revealed a language related network including frontal and temporal areas which is in accordance with earlier studies (Stromswold et al., 1996; Caplan et al., 1998; Ni et al., 2000; Fiebach et al., 2001; Moro et al., 2001; Newman et al., 2001; Röder et al., 2002; Friederici et al., 2003; Kuperberg et al., 2003; Meltzer et al., 2009; Raettig et al., 2009; Schmidt and Seger, 2009; Snijders et al., 2009). Additionally, the ACC was found to be deactivated, a region which is associated with attentional and verbal control (response conflict) as investigated with the Stroop task (e.g., Bush et al., 2000; Barch et al., 2001; Roelofs and Hagoort, 2002; Botvinick et al., 2004; Roelofs et al., 2006; Carter and van Veen, 2007; Aarts et al., 2008). We will now consider the specific activation patterns of these regions in the light of previous neuroimaging data. We put these data into the perspective of the MUC model as proposed by Vosse and Kempen (2000), Hagoort (2003, 2005), and Snijders et al. (2009).
Frontal Areas
Within the MUC, the left IFG is associated with the Unification component, which is thought to be driven by syntactic, semantic, pragmatic, and phonological properties of linguistic devices. The present experiment led to bilateral but left-preponderant activation of inferior frontal areas. The left IFG is known to be a key region for language processing as activations were found during syntactic tasks (Stromswold et al., 1996; Caplan et al., 1998; Ni et al., 2000; Fiebach et al., 2001; Moro et al., 2001; Newman et al., 2001; Röder et al., 2002; Friederici et al., 2003; Kuperberg et al., 2003; Hammer et al., 2007; Meltzer et al., 2009) and semantic processing (Newman et al., 2001; Kiehl et al., 2002; Kuperberg et al., 2003, 2008; Hagoort et al., 2004; Hammer et al., 2007). Pronouns that were distant to the antecedent revealed an increased activation as compared to close pronouns showing the sensitivity of the IFG to verbal working memory demands (Cooke et al., 2002, 2006; Fiebach et al., 2005; Hagoort, 2005; Bahlmann et al., 2007; Santi and Grodzinsky, 2007). Here, a direct measure of verbal working memory demands was realized by comparing LD to SD sentences (see Figure 2): both congruent LD pronoun types (person and thing) showed increased activity as compared to SD pronouns. This finding is in accordance with earlier studies investigating long syntactic dependencies (Fiebach et al., 2001; Cooke et al., 2002) or semantic working memory (Gabrieli et al., 1998). However, the given data did not show a differentiation between person or thing antecedents indicating that the activations within the IFG are rather driven by Congruency (general gender information processing independent of the antecedent type) and Distance (demand on verbal working memory) indexing the increased complexity of incongruency and longer distance. A virtually identical pattern of activations was found in the right IFG but with reduced strength. The recruitment of right inferior frontal regions was previously shown for pronoun integration (Hammer et al., 2007), deductive reasoning (Goel et al., 2000), processing of semantic vs. syntactic anomalies (Kang et al., 1999), metaphor processing (Bottini et al., 1994; Mashal et al., 2005; Schmidt and Seger, 2009), and topic maintenance compared to logic judgment (for an overview on right hemispheric activation see Caplan and Dapretto, 2001; Bookheimer, 2002).
Our data support the key role of the IFG as also highlighted by the MUC model (Hagoort, 2005; Snijders et al., 2009). In order to form a comprehensive sentence the elements need to be unified. In the present study, next to general unification within the sentence, a link has to be established between the pronoun and the antecedent. The observed main effect of congruency (incongruent > congruent) together with a lack of any antecedent type effect (person = thing conditions) indicates that unification is a flexible process, either based on pure syntactic or on a combination of syntactic and semantic information. In case of incongruency, independent of the type, unification fails within the sentence and finds its reflex in higher bilateral IFG activation. In addition, according to the MUC, unification has to deal with a decay of available information over time. We tested this idea by varying the distance of antecedent and pronoun by the insertion of additional words. In line with the MUC, IFG was sensitive to these manipulations, supporting the idea that Unification load is increased with distance: Here, unification load can be seen as a consequence of the maintenance of pronoun information over time, and also, as a consequence of integrating the additional intervening material. This is fully compatible with the MUC model and with working memory accounts, as it entails the maintenance of information over time and the operations on these information types. In relation to information specific areas and the operation on these information types see Snijders et al. (2009) and the discussion on activation of temporal areas of the given findings below.
Anterior Cingulate Cortex
The ACC was sensitive to the experimental manipulations and showed modulations dependent on Distance, Antecedent, and Congruency (see Table 3). A significantly larger decrease of BOLD was found for LD sentences as compared to SD sentences (see Figure 2), which can be associated with an increased demand on verbal working memory. Overall, this decrease was stronger for Person compared to Thing antecedents, and for incongruent vs. congruent Person antecedent in the SD condition.
The ACC has been assigned to multiple functions, among them sustaining selective attention in conflicting choices (e.g., Botvinick et al., 2001; van Veen et al., 2001; Brown and Braver, 2005, 2008; Carter and van Veen, 2007). In a more general sense, the ACC deals with the flexible allocation of processing resources in concert with the prefrontal cortex (Luks et al., 2002, 2007; Lungu et al., 2007; Mitchell et al., 2007). We therefore see the differential activation of the ACC in the different conditions as a reflex of frontal processing demands.
Within the framework of the MUC model the ACC is associated with verbal control (Hagoort, 2003). Details are not further specified, but one may assume that ACC controls all relevant cognitive functions related to unification (availability of information in working memory, syntactic, and semantic integration). Against this background, the observed ACC distance effect (decrease LD > SD) suggests that this area increases control when challenged by increased verbal working memory demands, i.e., a larger distance between antecedent and pronoun. This larger demand may correspond to the MUC assumption that unification information decays over time. Sustaining this information in working memory may be more difficult the longer the distance or time span or the distance between antecedent and pronoun. Additionally, this effect was more pronounced for the pronouns referring back to a person. In this case co-referencing is based on two types of gender information (biological and syntactic). The fact that only SD person sentences showed a congruency effect (see Figure 2) indicates that control demands are amplified in case more information has to be controlled for (semantic/syntactic gender vs. syntactic gender only).
Temporal Areas
Left temporal areas are associated with the memory retrieval component within the MUC as these are associated with the storage and retrieval of linguistic information, which are defined by their syntactic properties, especially in the plMTG (Hagoort, 2005). Snijders et al. (2009) also highlight the importance of temporal regions for lexical-semantic processing within the MUC, especially in the alMTG. This is in accordance with earlier studies. Superior temporal regions have previously been associated with semantic processing (Kang et al., 1999; Kuperberg et al., 2000, 2003; Ni et al., 2000; Newman et al., 2001). Our previous fMRI study on pronoun processing (Hammer et al., 2007) also showed an involvement of temporal regions. However, the present results go beyond semantics, as we investigated the interaction of syntactic and semantic processing with working memory. We observed an activation of two temporal regions, i.e., the anterior portion of the STG (including the temporal pole) and a more posterior portion of the MTG. Figure 2 shows that both regions resulted in an overall increase of activation for the LD condition as compared to the SD condition. This is in accordance with earlier studies reporting that temporal regions are involved in verbal working memory processes (Gabrieli et al., 1998; Martin and Chao, 2001; Thierry et al., 2003; Fiebach et al., 2007). We suggest that this activation of temporal regions for LD compared to SD conditions indicates that the access to antecedents requires more processing capacity within temporal regions in cases of longer distance between pronoun and antecedent, where an “object representation system” is supposed to be stored (Damasio et al., 1996; Thierry et al., 2003). In LD sentences the antecedent was shown a longer time ago based on the additional words between pronoun and antecedent and thus the comprehension system relies stronger on retrieval of verbal information. This stronger retrieval can be semantic (Martin and Chao, 2001). In a MUC context the two observed areas (alSTG and plMTG) suggest that retrieval is amplified for both semantic and syntactic processing. This is also supported by the congruency effect in pMTG for LD sentences. In case of an incongruent pronoun the comprehension system still tries to integrate the pronoun by a search for a suitable antecedent which, again, requires retrieval. The location (pMTG) suggests that this search is based on syntactic retrieval (Snijders et al., 2009). Interestingly, whereas posterior temporal regions did not show a modulation by the type of antecedent the anterior temporal region did (see Figure 2): sentences with pronouns following a person antecedent revealed an increase in activation compared to thing antecedents. The results need to be interpreted with some caution due to the small number of subjects which might have led us to miss some less prominent effects. However, building up co-reference between a pronoun and a person antecedent is based on biological and syntactic gender information (i.e., two types of information) and in addition might also be driven by agency. In line with the MUC the antecedent effect suggests that this area is involved in semantic retrieval, showing stronger activity related to the retrieval of biological information.
Overall, this differentiation of temporal regions (anterior: driven by distance and antecedent modulation; posterior: driven by distance and congruency modulation) fits into the MUC idea. The anterior temporal region processes semantic retrieval, here especially the person related information. The posterior temporal region processes syntactic congruency independent of the antecedent type. The sensitivity to distance suggests that in general the retrieval process undergoes a higher load in these areas.
Conclusion
The MUC model as presented by Hagoort (2003, 2005) and Snijders et al. (2009) is a working model for unification in cognitive and neural terms. Here, we used it to explain the neural pattern of pronoun resolution in sentence reading. Studies testing the MUC reported that Memory retrieval processes were associated with left temporal regions, Unification with the left IFG and Control with the dorsolateral prefrontal cortex and the ACC. In our study, these areas were indeed active during our experimental manipulations (left IFG, temporal regions, ACC, and additionally the right IFG). Importantly, our data suggests that the Unification interacts with memory (distance) supported by the earlier corresponding ERP findings (Hammer et al., 2008). However, within IFG it does so independent of antecedent type, suggesting the integrated unification of semantic and syntactic information. In contrast, the retrieval within temporal regions is sensitive to the type of antecedent (more anterior, more active for person compared to thing sentences) and to syntactic processing (more posterior, sensitive to congruency in general). Distance effects in these temporal regions suggest that this retrieval process is overall more difficult, or more heavily addressed by higher IFG unification activity.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
The research reported in this paper was supported by a grant of the Dutch Science Foundation (NWO) to BMJ (she also published under her maiden name Schmitt), and by a grant MU1311/9-1 of the German Science Foundation (DFG) to TFM, as a bilateral co-operation project DFG/NWO.
Footnotes
- ^In German there are only few exceptions to this rule where the syntactic gender is not in correspondence with the biological gender [i.e., diminutives (dasneutralFrauchenneutral/female) or dasneutralWeibneutral/female (old word for woman)]. For an ERP study related to diminutives and pronoun processing see Schmitt et al., 2002.)
- ^The masculine gender article (nominative form) is “der,” the neuter gender article “das.”
References
Aarts, E., Roelofs, A., and van Turennout, M. (2008). Anticipatory activity in anterior cingulate cortex can be independent of conflict and error likelihood. J. Neurosci. 28, 4671–4678.
Baayen, R. H., Piepenbrock, R., and Van Rijen, H. (1995). The Celex Lexical Database. Philadelphia: University of Pennsylvania, Linguistic Data Consortium.
Bahlmann, J., Rodriguez-Fornells, A., Rotte, M., and Munte, T. F. (2007). An fMRI study of canonical and noncanonical word order in German. Hum. Brain Mapp. 28, 940–949.
Barch, D. M., Braver, T. S., Akbudak, E., Conturo, T., Ollinger, J., and Snyder, A. (2001). Anterior cingulate cortex and response conflict: effects of response modality and processing domain. Cereb. Cortex 11, 837–848.
Bookheimer, S. (2002). Functional MRI of language: new approaches to understanding the cortical organization of semantic processing. Annu. Rev. Neurosci. 25, 151–188.
Bottini, G., Corcoran, R., Sterzi, R., Paulesu, E., Schenone, P., Scarpa, P., Frackowiak, R. S., and Frith, C. D. (1994). The role of the right hemisphere in the interpretation of figurative aspects of language. A positron emission tomography activation study. Brain 117(Pt 6), 1241–1253.
Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., and Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychol. Rev. 108, 624–652.
Botvinick, M. M., Cohen, J. D., and Carter, C. S. (2004). Conflict monitoring and anterior cingulate cortex: an update. Trends Cogn. Sci. 8, 539–546.
Brown, J. W., and Braver, T. S. (2005). Learned predictions of error likelihood in the anterior cingulate cortex. Science 307, 1118–1121.
Brown, J. W., and Braver, T. S. (2008). A computational model of risk, conflict, and individual difference effects in the anterior cingulate cortex. Brain Res. 1202, 99–108.
Bush, G., Luu, P., and Posner, M. I. (2000). Cognitive and emotional influences in anterior cingulate cortex. Trends Cogn. Sci. 4, 215–222.
Caplan, D., Alpert, N., and Waters, G. (1998). Effects of syntactic structure and propositional number on patterns of regional cerebral blood flow. J. Cogn. Neurosci. 10, 541–552.
Caplan, R., and Dapretto, M. (2001). Making sense during conversation: an fMRI study. Neuroreport 12, 3625–3632.
Carpenter, P. A., Just, M. A., Keller, A., Eddy, W. F., and Thulborn, K. R. (1999). Time course of fMRI- activation in language and spatial networks during sentence comprehension. Neuroimage 10, 216–224.
Carter, C. S., and van Veen, V. (2007). Anterior cingulate cortex and conflict detection: an update of theory and data. Cogn. Affect. Behav. Neurosci. 7, 367–379.
Cooke, A., Grossman, M., DeVita, C., Gonzalez-Atavales, J., Moore, P., Chen, W., Gee, J., and Detre, J. (2006). Large-scale neural network for sentence processing. Brain Lang. 96, 14–36.
Cooke, A., Zurif, E. B., DeVita, C., Alsop, D., Koenig, P., Detre, J., Gee, J., Pinango, M., Balogh, J., and Grossman, M. (2002). Neural basis for sentence comprehension: grammatical and short-term memory components. Hum. Brain Mapp. 15, 80–94.
Damasio, H., Grabowski, T. J., Tranel, D., Hichwa, R. D., and Damasio, A. R. (1996). A neural basis for lexical retrieval. Nature 380, 499–505.
Fiebach, C. J., Friederici, A. D., Smith, E. E., and Swinney, D. (2007). Lateral inferotemporal cortex maintains conceptual-semantic representations in verbal working memory. J. Cogn. Neurosci. 19, 2035–2049.
Fiebach, C. J., Schlesewsky, M., and Friederici, A. D. (2001). Syntactic working memory and the establishment of filler-gap dependencies: insights from ERPs and fMRI. J. Psycholinguist. Res. 30, 321–338.
Fiebach, C. J., Schlesewsky, M., Lohmann, G., von Cramon, D. Y., and Friederici, A. D. (2005). Revisiting the role of Broca’s area in sentence processing: syntactic integration versus syntactic working memory. Hum. Brain Mapp. 24, 79–91.
Friederici, A. D., Ruschemeyer, S. A., Hahne, A., and Fiebach, C. J. (2003). The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. Cereb. Cortex 13, 170–177.
Gabrieli, J. D., Poldrack, R. A., and Desmond, J. E. (1998). The role of left prefrontal cortex in language and memory. Proc. Natl. Acad. Sci. U.S.A. 95, 906–913.
Gainotti, G. (2007). Different patterns of famous people recognition disorders in patients with right and left anterior temporal lesions: a systematic review. Neuropsychologia 45, 1591–1607.
Gainotti, G., Ferraccioli, M., and Marra, C. (2010). The relation between person identity nodes, familiarity judgment and biographical information. Evidence from two patients with right and left anterior temporal atrophy. Brain Res. 1307, 103–114.
Garrod, S. C., and Sanford, A. J. (1994). “Resolving sentences in a discourse context,” in Handbook of Psycholinguistics, ed. M. A. Gernsbacher (New York: Academic Press), 675–698.
Genovese, C. R., Lazar, N. A., and Nichols, T. (2002). Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15, 870–878.
Goel, V., Buchel, C., Frith, C., and Dolan, R. J. (2000). Dissociation of mechanisms underlying syllogistic reasoning. Neuroimage 12, 504–514.
Gordon, P. C., and Hendrick, R. (1998). The representation and processing of coreference in discourse. Cogn. Sci. 22, 389–424.
Graesser, A. C., Millis, K. K., and Zwaan, R. A. (1997). Discourse comprehension. Annu. Rev. Psychol. 48, 163–189.
Hagoort, P. (2003). How the brain solves the binding problem for language: a neurocomputational model of syntactic processing. Neuroimage 20(Suppl. 1), S18–S29.
Hagoort, P., Hald, L., Bastiaansen, M., and Petersson, K. M. (2004). Integration of word meaning and world knowledge in language comprehension. Science 304, 438–441.
Hammer, A., Goebel, R., Schwarzbach, J.,Munte, T. F., and Jansma, B. M. (2007). When sex meets syntactic gender on a neural basis during pronoun processing. Brain Res. 1146, 185–198.
Hammer, A., Jansma, B. M., Lamers, M., and Munte, T. F. (2005). Pronominal reference in sentences about persons or things: an electrophysiological approach. J. Cogn. Neurosci. 17, 227–239.
Hammer, A., Jansma, B. M., Lamers, M., and Munte, T. F. (2008). Interplay of meaning, syntax and working memory during pronoun resolution investigated by ERPs. Brain Res. 1230, 177–191.
Just, M. A., Newman, S. D., Keller, T. A., McEleney, A., and Carpenter, P. A. (2004). Imagery in sentence comprehension: an fMRI study. Neuroimage 21, 112–124.
Kang, A. M., Constable, R. T., Gore, J. C., and Avrutin, S. (1999). An event-related fMRI study of implicit phrase-level syntactic and semantic processing. Neuroimage 10, 555–561.
Keller, T. A., Carpenter, P. A., and Just, M. A. (2001). The neural bases of sentence comprehension: a fMRI examination of syntactic and lexical processing. Cereb. Cortex 11, 223–237.
Kiehl, K. A., Laurens, K. R., and Liddle, P. F. (2002). Reading anomalous sentences: an event-related fMRI study of semantic processing. Neuroimage 17, 842–850.
Kuperberg, G. R., Holcomb, P. J., Sitnikova, T., Greve, D., Dale, A. M., and Caplan, D. (2003). Distinct patterns of neural modulation during the processing of conceptual and syntactic anomalies. J. Cogn. Neurosci. 15, 272–293.
Kuperberg, G. R., McGuire, P. K., Bullmore, E. T., Brammer, M. J., RabeHesketh, S., Wright, I. C., Lythgoe, D. J., Williams, S. C., and David, A. S. (2000). Common and distinct neural substrates for pragmatic, semantic, and syntactic processing of spoken sentences: an fMRI study. J. Cogn. Neurosci. 12, 321–341.
Kuperberg, G. R., Sitnikova, T., and Lakshmanan, B. M. (2008). Neuroanatomical distinctions within the semantic system during sentence comprehension: evidence from functional magnetic resonance imaging. Neuroimage 40, 367–388.
Lamers, M. J., Jansma, B. M., Hammer, A., and Munte, T. F. (2006). Neural correlates of semantic and syntactic processes in the comprehension of case marked pronouns: evidence from German and Dutch. BMC Neurosci. 7, 23. doi: 10.1186/1471-2202-7-23
Lamers, M. J., Jansma, B. M., Hammer, A., and Munte, T. F. (2008). Differences in the processing of anaphoric reference between closely related languages: neurophysiological evidence. BMC Neurosci. 9, 55. doi: 10.1186/1471-2202-9-55
Li, X., and Zouh, X. (2010). Who is ziji? ERP responses to the Chinese reflexive pronoun during sentence comprehension. Brain Res. 1331, 96–104.
Luks, T. L., Simpson, G. V., Dale, C. L., and Hough, M. G. (2007). Preparatory allocation of attention and adjustments in conflict processing. Neuroimage 35, 949–958.
Luks, T. L., Simpson, G. V., Feiwell, R. J., and Miller, W. L. (2002). Evidence for anterior cingulate cortex involvement in monitoring preparatory attentional set. Neuroimage 17, 792–802.
Lungu, O. V., Liu, T., Waechter, T., Willingham, D. T., and Ashe, J. (2007). Strategic modulation of cognitive control. J. Cogn. Neurosci. 19, 1302–1315.
Martin, A., and Chao, L. L. (2001). Semantic memory and the brain: structure and processes. Curr. Opin. Neurobiol. 11, 194–201.
Mashal, N., Faust, M., and Hendler, T. (2005). The role of the right hemisphere in processing nonsalient metaphorical meanings: application of principal components analysis to fMRI data. Neuropsychologia 43, 2084–2100.
Meltzer, J. A., McArdle, J. J., Schafer, R. J., and Braun, A. R. (2009). Neural aspects of sentence comprehension: syntactic complexity, reversibility, and reanalysis. Cereb. Cortex. 20, 1853–1864.
Mitchell, J. P., Heatherton, T. F., Kelley, W. M., Wyland, C. L., Wegner, D. M., and Neil Macrae, C. (2007). Separating sustained from transient aspects of cognitive control during thought suppression. Psychol. Sci. 18, 292–297.
Moro, A., Tettamanti, M., Perani, D., Donati, C., Cappa, S. F., and Fazio, F. (2001). Syntax and the brain: disentangling grammar by selective anomalies. Neuroimage 13, 110–118.
Newman, A. J., Pancheva, R., Ozawa, K., Neville, H. J., and Ullman, M. T. (2001). An event-related fMRI study of syntactic and semantic violations. J. Psycholinguist. Res. 30, 339–364.
Newman, S. D., Just, M. A., and Carpenter, P. A. (2002). The synchronization of the human cortical working memory network. Neuroimage 15, 810–822.
Ni, W., Constable, R. T., Mencl, W. E., Pugh, K. R., Fulbright, R. K., Shaywitz, S. E., Shaywitz, B. A., Gore, J. C., and Shankweiler, D. (2000). An event-related neuroimaging study distinguishing form and content in sentence processing. J. Cogn. Neurosci. 12, 120–133.
Raettig, T., Frisch, S., Friederici, A. D., and Kotz, S. A. (2009). Neural correlates of morphosyntactic and verb-argument structure processing: an EfMRI study. Cortex 46, 613–620.
Röder, B., Stock, O., Neville, H., Bien, S., and Rosler, F. (2002). Brain activation modulated by the comprehension of normal and pseudo-word sentences of different processing demands: a functional magnetic resonance imaging study. Neuroimage 15, 1003–1014.
Roelofs, A., and Hagoort, P. (2002). Control of language use: cognitive modeling of the hemodynamics of Stroop task performance. Brain Res. Cogn. Brain Res. 15, 85–97.
Roelofs, A., van Turennout, M., and Coles, M. G. (2006). Anterior cingulate cortex activity can be independent of response conflict in Stroop-like tasks. Proc. Natl. Acad. Sci. U.S.A. 103, 13884–13889.
Santi, A., and Grodzinsky, Y. (2007). Working memory and syntax interact in Broca’s area. Neuroimage 37, 8–17.
Schmidt, G. L., and Seger, C. A. (2009). Neural correlates of metaphor processing: the roles of figurativeness, familiarity and difficulty. Brain Cogn. 71, 375–386.
Schmitt, B. M., Lamers, M., and Munte, T. F. (2002). Electrophysiological estimates of biological and syntactic gender violation during pronoun processing. Brain Res. Cogn. Brain Res. 14, 333–346.
Snijders, T. M., Vosse, T., Kempen, G., Van Berkum, J. J., Petersson, K. M., and Hagoort, P. (2009). Retrieval and unification of syntactic structure in sentence comprehension: an FMRI study using word-category ambiguity. Cereb. Cortex 19, 1493–1503.
Streb, J., Hennighausen, E., and Rosler, F. (2004). Different anaphoric expressions are investigated by event-related brain potentials. J. Psycholinguist. Res. 33, 175–201.
Stromswold, K., Caplan, D., Alpert, N., and Rauch, S. (1996). Localization of syntactic comprehension by positron emission tomography. Brain Lang. 52, 452–473.
Talairach, J., and Tournoux, P. (1988). Co-planar Stereotaxic Atlas of the Human Brain, 1st Edn. New York: Thieme Medical Publishing Company.
Thierry, G., Ibarrola, D., Demonet, J. F., and Cardebat, D. (2003). Demand on verbal working memory delays haemodynamic response in the inferior prefrontal cortex. Hum. Brain Mapp. 19, 37–46.
van Veen, V., Cohen, J. D., Botvinick, M. M., Stenger, V. A., and Carter, C. S. (2001). Anterior cingulate cortex, conflict monitoring, and levels of processing. Neuroimage 14, 1302–1308.
Keywords: language, pronoun, syntax, semantics, memory, unification, MUC, fMRI
Citation: Hammer A, Jansma BM, Tempelmann C and Münte TF (2011) Neural mechanisms of anaphoric reference revealed by fMRI. Front. Psychology 2:32. doi: 10.3389/fpsyg.2011.00032
Received: 12 March 2010;
Accepted: 14 February 2011;
Published online: 24 February 2011.
Edited by:
Peter Hagoort, Radboud University Nijmegen, NetherlandsReviewed by:
Simon Garrod, University of Glasgow, UKPeter Hagoort, Radboud University Nijmegen, Netherlands
Karl-Magnus Petersson, Max Planck Institute for Psycholinguistics, Netherlands
Copyright: © 2011 Hammer, Jansma, Tempelmann and Münte. This is an open-access article subject to an exclusive license agreement between the authors and Frontiers Media SA, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are credited.
*Correspondence: Thomas F. Münte, Klinik für Neurologie, Universitätsklinikum Schleswig-Holstein, Campus Lübeck, Ratzeburger Allee 160, 23538 Lübeck, Germany. e-mail: thomas.muente@neuro.uni-luebeck.de