“Doctor” or “darling”? Decoding the communication partner from ECoG of the anterior temporal lobe during non-experimental, real-life social interaction
- 1Epilepsy Center, University Medical Center Freiburg, Freiburg, Germany
- 2Faculty of Biology, University of Freiburg, Freiburg, Germany
- 3Bernstein Center Freiburg, University of Freiburg, Freiburg, Germany
- 4Hermann Paul School of Linguistics, University of Freiburg, Freiburg, Germany
- 5Faculty of Philology, University of Freiburg, Freiburg, Germany
Human brain processes underlying real-life social interaction in everyday situations have been difficult to study and have, until now, remained largely unknown. Here, we investigated whether electrocorticography (ECoG) recorded for pre-neurosurgical diagnostics during the daily hospital life of epilepsy patients could provide a way to elucidate the neural correlates of non-experimental social interaction. We identified time periods in which patients were involved in conversations with either their respective life partners (Condition 1; C1) or attending physicians (Condition 2; C2). These two conditions can be expected to differentially involve subfunctions of social interaction which have been associated with activity in the anterior temporal lobe (ATL), including the temporal pole (TP). Therefore, we specifically focused on ECoG recordings from this brain region and investigated spectral power modulations in the alpha (8–12 Hz) and theta (3–5 Hz) frequency ranges, which have been previously assumed to play an important role in the processing of social interaction. We hypothesized that brain activity in this region might be sensitive to differences in the two interaction situations and tested whether these differences can be detected by single-trial decoding. Condition-specific effects in both theta and alpha bands were observed: the left and right TP exclusively showed increased power in C1 compared to C2, whereas more posterior parts of the ATL exhibited similar (C1 > C2) and also contrary (C2 > C1) effects. Single-trial decoding accuracies for classification of these effects were highly above chance. Our findings demonstrate that it is possible to study the neural correlates of human social interaction in non-experimental conditions. Decoding the identity of the communication partner and adjusting the speech output accordingly may be useful in the emerging field of brain-machine interfacing for restoration of expressive speech.
Introduction
There is a long-standing interest in investigating the neural processing of naturalistic sensory stimuli and natural behavior (Aertsen et al., 1981; Montague et al., 2002; Babiloni et al., 2006). An important motivation behind such studies is previous single-neuron research showing that the neural activity in natural, ecologically more valid conditions has different statistical properties than that of artificial stimuli: sparse coding (Vinje and Gallant, 2000; Felsen and Dan, 2005; Yen et al., 2007; Haider et al., 2010), as well as precise (Dan et al., 1996; Mechler et al., 1998; Yao et al., 2007; Haider et al., 2010) and reliable (Haider et al., 2010; Herikstad et al., 2011) spike timing allow conveying information more efficiently. Neural processing in complex, real-life conditions thus cannot be reduced to a superposition of responses to a small set of simple (artificial) stimuli, but likely relies on more complex, non-linear processes [see Hasson et al. (2010) for a review].
Several previous studies on the processing of naturalistic sensory stimuli used natural sounds to explore auditory processing in animals (Suga, 1978; Smolders et al., 1979; Aertsen et al., 1981). In humans, this kind of experiments were adopted (Nelken, 2004) and extended to human-specific stimuli, such as recordings of natural stories (Fletcher et al., 1995; Brennan et al., 2010; Lerner et al., 2011) and movies (Zacks et al., 2001; Bartels and Zeki, 2004; Mukamel et al., 2005; Golland et al., 2007; Privman et al., 2007).
Another line of studies employed non-experimental settings to elucidate the neural basis of unrestrained hand and arm movements in monkeys (Evarts, 1965; Mavoori et al., 2005; Aflalo and Graziano, 2006; Jackson et al., 2006, 2007) and spontaneous, uninstructed language in humans (Towle et al., 2008). Investigations in experimentally unrestricted conditions allow capturing the complexity and functional diversity of real-life behavior more extensively than by standard laboratory procedures (Gibson, 1950) and may prevent a possible contamination of findings caused by the experimental environment as such (Bartlett, 1995), e.g., by the influence of emotional reaction of subjects to the experimenter (Ray, 2002).
Previous studies have also used conditions approximated to real life to study the densely interwoven perception and production processes underlying social interaction in humans. For instance, social interaction has been studied using fMRI experiments in virtual-reality social encounters between subjects and virtual characters (Wilms et al., 2010; Ethofer et al., 2011; Pfeiffer et al., 2011). Also, techniques have been developed to simultaneously record the brain activities of two or more interacting individuals with the help of EEG (Babiloni et al., 2006), fMRI (Montague et al., 2002), and MEG (Baess et al., 2012). In this way, various kinds of interactive behaviors can be investigated, e.g., in spontaneous communication of subjects while playing games (Montague et al., 2002; Babiloni et al., 2006), imitating others' movements (Dumas et al., 2010), or collectively making music (Lindenberger et al., 2009).
Following this trend towards increasingly naturalistic approaches, it would be highly interesting to study brain activity underlying real-life human social interaction outside experiments. This may enable investigators to not only rule out the unwanted effects induced by experimental settings, but, even more so, to investigate the specific kinds of social interaction situations that cannot, or only with great difficulty, be studied experimentally.
Such investigations of the neural basis of social interaction in non-experimental, real-life environments are, however, currently lacking (Hari and Kujala, 2009). Major reasons for the absence of such studies are methodological limitations of most recording techniques in humans: traditional imaging methods [e.g., positron emission tomography (PET) or functional magnetic resonance imaging (fMRI)] require a stationary apparatus, with the subjects placed in a fixed position, and therefore these techniques cannot be employed in measurements of dynamic, unrestricted real-life behavior. Non-invasive electroencephalography (EEG) is also not well suited for this purpose due to its limited spatial resolution and its high susceptibility to artifacts, such as those induced by speaking or other movements (Figure 1).
Figure 1. Example of artifacts related to head movement in simultaneous non-invasive, scalp-recorded EEG (upper 4 traces) and ECoG recorded using subdurally implanted electrodes (lower 6 traces). The height of the black scale bar in the lower right corner of the plot corresponds to 100 μV.
In the present study, we employed, for the first time, human electrocorticography (ECoG) to study neural processes related to real-life social interaction. Owing to the combination of superior temporal resolution and much higher resistance to artifacts compared with non-invasive recordings (see Figure 1 and Ball et al., 2009a), ECoG proved a valuable technique for investigating human motor (Crone et al., 1998a,b) and language (Crone et al., 2001a,b; Sinai et al., 2005) functions, and became a promising candidate signal for clinical brain-machine interface (BMI) applications (Leuthardt et al., 2006; Pistohl et al., 2008, 2012; Ball et al., 2009b), including approaches for restoration of speech production (Blakely et al., 2008; Leuthardt et al., 2011; Pei et al., 2011). In the present study, we performed post hoc analyses of ECoG data continuously recorded for pre-neurosurgical diagnostics over several days or weeks during the daily hospital life of epilepsy patients. Throughout the analyzed time periods, patients were conscious, fully alert, and exhibited a wide spectrum of social behaviors, including active interaction with clinical personnel, family, friends, and other patients.
Previous research on social interaction in the fields of linguistics, social psychology, and health care has extensively studied communication between doctors and patients (Roter and Hall, 1989; Ong et al., 1995; Ha and Longnecker, 2010; Nowak, 2011). By contrast, interaction between intimate partners has been within the focus of psychosociological and linguistic research (Sillars and Scott, 1983; Gottman and Notarius, 2000; Pennebaker et al., 2003). Here, we aimed to elucidate, for the first time, the differential neural processes underlying these interactive situations in real-life communication. To do so, we compared conversations during which patients were either talking to their life partners (Condition 1, C1) or to their attending physicians (Condition 2, C2). The two conditions can be assumed to differ in various aspects of social interaction. For instance, patients are more intimate and emotionally attached to their life partners, and share more life experiences with them than with their physicians. Conversely, conversations with physicians are typically more emotionally contained and based on factual communication (Good and Good, 1982).
Our analysis specifically focused on the temporal poles (TP) and the adjacent area of the anterior temporal lobe (ATL) because these areas are associated with several processes crucially involved in social interaction, including autobiographical memory (Spreng et al., 2009), theory of mind (ToM) (Spreng et al., 2009), comprehension of stories (Mar, 2011), and face processing (Olson et al., 2007).
We investigated spectral power modulations in the TP and in the ATL related to social interaction in the alpha (8–12 Hz) and theta (3–5 Hz) ECoG frequency components. Cortical alpha-rhythm changes have been previously associated with dynamic social interaction including eye contact and inter-personal distance (Gale et al., 1975), perception of others' movements (Tognoli et al., 2007), and social coordination (Tognoli et al., 2007; Naeem et al., 2012). Both increases (Gale et al., 1975; Tognoli et al., 2007) and decreases (Boksem et al., 2009) in alpha frequencies have been reported to reflect social cognitive processing. To our knowledge, however, no study has investigated alpha-rhythm modulations in the ATL during social interaction, and it is currently unclear whether alpha power can be employed as a neural marker for social cognition in this brain region. Theta-band changes have been observed in memory-related processes including episodic recollection (Gruber and Müller, 2006), autobiographical memory (Steinvorth et al., 2010), and recognition of familiar faces (Başar et al., 2006, 2007). We therefore expected theta-band power in our target brain regions to undergo modulations by memory-related processing during social interaction.
To estimate the potential usefulness of neural differences during communication with different dialog partners for BMI applications, we also performed a single-trial classification analysis. BMI-based restoration of expressive speech is a topic of growing interest (Pei et al., 2012). So far, BMI studies mainly aimed at decoding such communication-relevant aspects as phonemes (Blakely et al., 2008; Guenther et al., 2009; Brumberg et al., 2011; Pei et al., 2011), words (Kellis et al., 2010), and semantic entities (Wang et al., 2011). Complementary to these approaches, our study makes a first step toward decoding of such high-level information as the identity of the speaker which may help accurate shaping of the language output.
Materials and Methods
Subjects
Three patients in pre-neurosurgical diagnostics of medically-intractable epilepsy using ECoG were included in this study upon their written informed consent. The study was approved by the Ethics Committee of the University Medical Center Freiburg. Two patients (S1, S3) were right-handed and one (S2) was ambidextrous, all had normal hearing and no history of affective disorders (for more details, see Table 1). Electrode sites analyzed in the present study were outside the seizure onset zone as determined by medical diagnostics. Cortical seizure onset zones in S1 and S2 were in the right posterior superior temporal gyrus and in left parietal areas, respectively, as depicted in Figure 2. In S3, the seizure onset zone was in the left hippocampus and was therefore not visible on the cortical surface.
Table 1. Patient details.
Figure 2. Location of all implanted grid and stripe electrodes in the three included subjects (S1–3). (A) is the lateral view of the right hemisphere of S1 and (B) and (C) show the left hemisphere of S2 and S3, respectively. (D,E,F) Display the corresponding bottom sides of the brain. Blue color indicates all contacts located in the ATL, contacts in yellow revealed language functions according to the results of electrostimulation, and contacts in magenta were located in the seizure onset zone. MNI coordinates of the electrodes are projected to an SPM standard brain. For this reason, some contacts which are actually located in the ATL, as indicated by the blue color, may look as if they were situated in the frontal lobe.
Neural Recordings
All subjects had subdurally implanted platinum or stainless-steel electrodes (Ad-Tech, Racine, Wisconsin, USA) 4 mm in diameter, covered in sheets of silicone and arranged in regular grids and stripes with a 10-mm center-to-center inter-electrode distance. ECoG was recorded using a clinical EEG-System (ITMed, Germany) at a sampling rate of 1024 Hz, a high-pass filter with a cutoff frequency of 0.032 Hz, and a low-pass anti-aliasing filter at 379 Hz. Digital video recordings (25 Hz frame rate) synchronized to ECoG were acquired for all subjects.
Conversation Periods
Based on ongoing digital video recordings, we identified time periods in which the patients were involved in conversations with their respective life partners (Condition 1; C1) or their attending physicians (Condition 2; C2), see Table 2. The selected epochs contained recordings from time periods during which the patients were having a natural, uninstructed conversation. For all subjects, the length of time periods of speech perception and speech production were roughly balanced between C1 and C2. The position of the conversation partners in the room was not restricted by prior instruction. The patients were sitting or lying in bed with wired connections of electrodes to non-portable amplifiers. During the selected conversation periods, patients were neither eating nor extensively moving their body. The epochs selected by this procedure thus do not necessarily correspond to entire conversations. In the course of conversations, all patients were fully alert, conscious, and able to talk, move, and gesticulate.
Table 2. Electrode implantation and details of analysis.
Preprocessing of Neural Data
For each individual subject, ECoG recordings from all channels were re-referenced to a common average reference of all implanted ECoG electrodes that were located outside the seizure onset zone. For the calculation of time-resolved power spectra, we applied a short-time Fourier transform using successive, non-overlapping, 1-s windows of the recorded ECoG signals, moved in steps of 1 s, resulting in a frequency resolution of 1 Hz.
The hypotheses of the present study refer to modulations in the theta and alpha bands. Therefore, we focused our analyses on these particular frequency ranges. Theta and alpha were defined as the range of 3–5 Hz and 8–12 Hz, respectively. We additionally analyzed the high gamma band in 70–150 Hz, as high gamma is a frequency range that has been extensively studied in previous ECoG research (Crone et al., 1998b, 2001a; Schalk et al., 2007; Ball et al., 2009b). For every channel, the median spectral power for the theta, alpha, and high gamma bands was calculated for each 1-s constituent of the C1 and C2 epochs. For statistical comparison, all power values in the C1 partner condition were tested against power values in the C2 physician condition using the non-parametric Wilcoxon rank-sum test, suited for unequal sample sizes (Sheskin, 2007). Cutting down the sample size of a larger group would decrease the statistical power and is thus not advisable (Rosner and Glynn, 2009). We corrected the resulting p-values for multiple comparisons over the number of conditions, channels, and frequency bands (theta, alpha, and gamma) using the false-discovery-rate (FDR) method (Benjamini and Yekutieli, 2001) with a threshold of p < 0.001. Figure 3 shows an overview of the computational procedures employed in the present study.
Figure 3. A systematic overview of the methods applied in the present study to compare neural responses in the TP and in the ATL of three subjects during social interaction with two different dialog partners. In addition to the rank-sum statistics, single-trial decoding analyses were carried out based on the 1-s epochs.
We found that 11 electrodes in S1 showed broad-banded spectral differences across the entire frequency range from 0 to 150 Hz in the two conditions. These channels were not included in further steps of analysis, since such broad-banded responses might be induced by artifacts (e.g., from myographic activity due to head movements) which generally show a broadly and homogeneously distributed frequency spectrum (Kovach et al., 2011). Alternatively, the observed broad-banded changes may arise from unspecific changes of the neural firing rates (Bédard et al., 2006; Miller et al., 2009), representing a different type of response compared to the more narrow-banded spectral power differences investigated in the present study. Such narrow-banded effects (e.g., in the theta or the alpha band) may result from oscillatory mechanisms originating from synchronized neural network activity and may support different dimensions of neural integration, the functional significance of the particular oscillations depending on the brain system involved (Buzsáki and Draguhn, 2004).
To quantify the effect size of the spectral differences between C1 and C2, we calculated in all ATL-electrodes the area under the receiver operating characteristic curve (AUROC) for the theta and alpha-bands separately, using the MES toolbox by Hentschke and Stüttgen (2011).
Single-trial decoding analyses were conducted using a regularized linear discriminant analysis as described in Pistohl et al. (2012). Decoding was performed in each subject separately, based on median (1) theta, (2) alpha, and (3) theta and alpha power values from all available 1-s epochs of C1 and C2. Since theta and alpha signal components may carry complementary information, we used theta and alpha features together in (3). Decoding accuracies were obtained for decoding from all electrodes in the ATL together, as well as for all individual ATL electrodes separately. For the individual contacts, resulting p-values were Bonferroni-corrected for multiple testing across the number of analyzed electrodes.
Electrode Positions
Post-operative T1-weighed MPRAGE data sets were acquired for every subject at a 1-mm isotropic resolution using a 1.5-T Vision MRI scanner (Siemens, Erlangen, Germany). The MR images were normalized to a standard brain in MNI (Montreal Neurological Institute) space using SPM5 (Friston et al., 1994). Electrode void artifacts visible in the MR images were identified and marked manually using Matlab programs developed in our laboratory for MRI visualization. Then, the corresponding MNI coordinates of electrode positions were extracted, and individual 3D locations of the contacts were visualized on a standard brain surface. ATL recording sites used for analyses were selected based on the spatial extension of the ATL as illustrated in Figure 3. The TP was defined according to Brodmann's description of area 38 (Brodmann, 1909) as done in Olson et al. (2007).
Results
C1 and C2 conversation periods were selected according to the criteria described in the “Materials and Methods” section. In subjects S1, S2, and S3, 2, 27, and 7 epochs of conversations with the life partner were available in our monitoring videos with a total duration of 4.4, 104.2, and 14.7 min, respectively. Conversations with the physician could be observed in 4, 6, and 3 epochs for S1, S2, and S3 with a total duration of 3.2, 6.4, and 2.5 min, respectively.
The dialog periods contained intermittent speaking and passive listening, overlapping and non-overlapping talk with different prosodic features of natural discourse, and multiple other aspects of natural oral communication, including conversation fillers, pauses, mimics, and gestures. C1 conversations with life partners covered various topics such as health state, family situation, gossip, news, public events, as well as general reflections about the self and life. In C2 conversations with attending physicians during daily medical rounds, common subjects of discussion were mainly the clinical situation, bodily complaints, progress of the diagnostic process, and small talk, for instance, about an ongoing soccer game and a book. Patients employed the German formal address pronoun “Sie” while talking to the attending physicians, while using the informal “Du” to address their life partners. During the conversation epochs analyzed, the spatial distance between the patients and their dialog partners was on average increased in the C2 condition as opposed to C1.
Of the 61 electrode sites in the ATL included in the whole analyses, 45 electrodes from 2 patients (30 in S2 and 15 in S3) were located in the left, and 16 from S1 in the right ATL (see Figure 2 and Table 2). In total, 25 electrodes were located in the TP, and the majority of all other electrodes were in the temporo-basal part of the ATL. The second most frequent topographical location was the superior temporal gyrus, followed by the inferior temporal gyrus and the middle temporal gyrus (see Table 3 and Table A1).
Table 3. Statistically significant effects (p < 0.001, FDR) in the theta (θ), alpha (α), and gamma (γ) frequency bands, MNI coordinates, and anatomical locations of ATL electrodes in S2.
Statistical tests (p < 0.001, FDR-corrected, see “Materials and Methods” and Figure 4) revealed significant differences across the two conditions in both tested frequency bands as shown in Figure 4, Table 3, and Table A1. The spectral power was significantly enhanced in C1 compared to C2 in both theta and alpha frequency ranges in the bilateral TP (15 and 17 electrodes, respectively) and in other parts of the ATL, both on the basal and lateral surface (red markers in Figure 4). In addition, some channels in more posterior parts of the ATL showed reduced activity in C2 compared to C1 (blue markers in Figure 4). Overall, 16 electrodes from the left and 9 electrodes from the right ATL showed significantly stronger spectral responses in C1 than in C2 in the theta range, whereas 28 and 8 electrodes from the left and right ATL, respectively, exhibited enhanced alpha power. For the TP alone, 11 electrodes from the left and 6 from the right hemisphere showed effects in the theta band, and 9 and 6 electrodes from the left and right hemisphere, respectively, showed effects in the alpha band. Conversely, less spectral power in C1 than in C2 could be observed in 10 electrodes of the left ATL for the theta band and in 4 electrodes for the alpha band. In the right hemisphere, there were no electrodes with increased power in C2 compared to C1. All electrodes with less power in C2 than in C1 were located more posterior in the ATL, and none of them was located in the TP. These effects were found in both S2 and S3.
Figure 4. Projection of ECoG electrode positions on an SPM standard brain. Dots, squares, and triangles depict ECoG electrodes from S1, S2, and S3, respectively. Red: enhanced activity in the theta (top row) and alpha (bottom row) bands during conversations of the three patients with their life partners (C1) in comparison with conversations with their physicians (C2). Blue: electrodes with significantly enhanced activity in C2 > C1. Green and yellow dotted lines show the anatomical definition of the ATL and the TP. (A) and (B) display effects in the theta frequency band in the right and left lateral ATL. (C) Shows theta effects in the inferior ATL. (D–F) Display the corresponding effects for the alpha frequency band. (G) Shows an example of one electrode in the ATL with differences in theta and alpha range power in the two conditions.
Twenty-six electrodes (22 and 4 for increased C1 and C2, respectively) exhibited effects in the same direction in both theta and alpha bands; 16 electrodes (13 and 3 for increased C1 and C2, respectively) showed isolated effects in the theta or in the alpha band; 3 electrodes revealed reverse theta- and alpha-band changes. From all brain areas with electrode coverage, including large parts of the temporal lobes and parts of the frontal and parietal lobes (see Figure 2 for orientation of electrodes from all subjects), pronounced theta and alpha amplitude differences in C1 compared to C2 were focused on the ATL. The theta and alpha effects in our study were thus both spatially focalized to the ATL (i.e., they did not occur in a spatially diffuse way over all electrodes) and frequency-band-specific.
As mentioned above, although the hypotheses of the present study concerned the alpha and theta bands, we additionally analyzed high gamma activity. In all subjects, most ATL electrodes with significant changes in the high gamma range showed increased power in C2 compared to C1 (43 electrodes), whereas only 5 electrodes exhibited the opposite effect. Electrodes with stronger gamma band power in C1 compared to C2 simultaneously showed significant effects (either increases or decreases) in the lower frequency bands. Thirteen and 23 electrodes with significantly stronger gamma band power in C2 than in C1 at the same time showed decreased activity in theta and alpha ranges, respectively, and increased power was observed in 9 and 4 electrodes in these frequency bands (see Table 3 and Table A1). Yet, effects in the gamma band also occurred in isolation, with no significant differences in the lower frequency bands between the two conditions detectable at the selected significant level.
AUROC values of the ATL electrodes in the theta band ranged across the two conditions between 0.33 and 0.79 in S1, between 0.33 and 0.62 in S2, and between 0.26 and 0.5 in S3. The respective values for the alpha band were between 0.23 and 0.7 in S1, 0.35 and 0.51 in S2, and 0.18 and 0.55 in S3.
We performed single-trial classification of 1-s epochs from C1 vs. C2 for all ATL electrodes together. Decoding from all ATL electrodes based on combined theta and alpha-band power yielded values of 0.67, 0.75, and 0.82 for S1, S2, and S3, respectively. More detailed information, including decoding accuracies for the individual theta and alpha frequency bands, is presented in Table 4. In an analysis based on single electrodes, classification was also above chance significantly (p < 0.05, Bonferroni-corrected) in 44% of all ATL electrodes in S1, 46% in S2, and 20% in S3 when decoding was performed based on a combination of theta and alpha frequency bands. Decoding accuracies reached values up to 0.6692, 0.6197, and 0.6788 in S1, S2, and S3, respectively.
Table 4. Results of single-trial decoding of C1 vs. C2 1-s epochs from all ATL electrodes together.
Discussion
Human brain processes underlying real-life social interaction in everyday situations have been difficult to study (Hari and Kujala, 2009) and hence remained, until now, a white spot in the literature. In the present study, we moved one step beyond the existing approaches to studying social interaction in near-to-natural experimental conditions by investigating brain activity underlying real-life interaction in ECoG-implanted epilepsy patients under diagnostic monitoring. Epilepsy patients undergoing presurgical diagnostics are in a very specific social situation. Usually, they share rooms with other patients, have a large fluctuation of clinical staff and visitors entering and leaving the room, and are constantly being monitored by video cameras required for this kind of diagnostic procedure. For these reasons, we refrained from calling this situation “natural” and rather employed the term “real-life” to account for the specific circumstances of our patients.
Based on ongoing digital video recordings synchronized to ECoG from 3 patients, we identified time periods in which the patients were involved in conversations with their respective life partners (C1) or with their attending physicians (C2), and compared neural activity in these two conditions as reflected by spectral power in the alpha and theta frequency bands. Both frequency bands showed increased power in C1 compared to C2 in many electrodes located bilaterally in the TP and the entire ATL region. Alpha and theta effects occurred in different combinations, e.g., only in alpha, only in theta, or in both frequency ranges simultaneously. Some contacts in more posterior parts of the left ATL showed opposite effects with significantly increased power in C2 compared to C1. There, modulations of alpha and theta responses sometimes even went in opposite directions at one and the same electrode (Figure 4). These posterior areas might support a different set of cognitive functions which may be recruited more strongly during conversations of the patient with the attending physician. Alternatively, the effects might be linked to inhibitory functional connectivity within an extended cortical network, where increased activity in one node may suppress activity in another, when their coupling is inhibitory.
Conversations between patients and their life partners differ from those with their attending physicians. This becomes apparent from the length and frequency of the interaction periods: indeed, all patients in our study spent much more time communicating with their partners than with the physicians, whom they mostly met during medical rounds for discussing health issues. The TP and the entire ATL region have been associated with the processing of different aspects of social cognition (Olson et al., 2007) and are thus suitable candidate areas for investigating modulations of neural activity related to social interaction. With its widespread connections to other cortical and subcortical areas of the brain (Morán et al., 1987; Kondo et al., 2003), the ATL is a suitable association area for high-level operations to coordinate multiple functions involved in social cognition (Olson et al., 2007). As this part of the brain is topographically remote from primary auditory and visual areas, processing of low-level features is not likely to have affected our results.
An important role in autobiographical memory processing has been attributed to the TP, i.e., recollecting personal events from the life of an individual (Spreng et al., 2009). Autobiographical memories are integral to natural conversation and provide a basis for self-disclosure, entertainment, joint planning and problem solving (Dritschel, 1991). Different social situations involve varying amounts of autobiographical memory, depending on the social distance of the dialog partner and other factors (Dritschel, 1991). Thus, differences in the recruitment of autobiographical memory between C1 and C2 in our study may have played an important role in the strong effects in the TP we observed for C1.
Clearly, social interaction via spoken language also has a linguistic dimension. The ATL region has been associated with language-related processing, including comprehension of narrative speech (Mar, 2011), syntactic complexity of natural stories (Brennan et al., 2010), semantic content (Visser et al., 2010), and narrative context (Xu et al., 2005). As these features may have likely differed between C1 and C2, a possible modulation of the spectral power of the ATL electrodes by such linguistic features is conceivable. A detailed linguistic analysis of the conversation data was, however, beyond the scope of the present study and is a topic for further research.
Another subfunction of social interaction which may have contributed to the observed differential oscillatory modulations in the ATL is the inference of mental states of the dialog partners, a mental function known as ToM. ToM has been associated with processing in the ATL (Spreng et al., 2009; Mar, 2011). Patients can be expected to have more elaborate and consolidated internal models of their life partners than of their physicians that may facilitate the prediction of mental states of the partner (Wolpert et al., 2003). Recognition of various features that are essential to successful interaction may be required to understand another person. An important role has been attributed to the TP and the ATL in recognizing familiar faces (Nakamura et al., 2000; Sugiura et al., 2011), names (Sugiura et al., 2009), and voices (Nakamura et al., 2001) of people. Beyond these specific effects, processing of familiarity in the ATL may be domain-unspecific (Nakamura et al., 2000). Indeed, overarching effects have been shown for personal acquaintances and famous people (Sugiura et al., 2009), familiar faces and scenes (Nakamura et al., 2000), and tools and animals (Whatmough et al., 2002). Evidence for such generality, however, remains contradictory (e.g., Barense et al., 2011).
Since the present study was conducted in epilepsy patients and under non-experimental conditions, it has certain limitations which will be addressed in the following. Although the seizure onset zone in all of our patients was located outside the ATL region (see Figure 2), it cannot be entirely ruled out that our observations may have been influenced by epileptiform activity. Also, we cannot exclude the possibility of epilepsy-related reorganization in our subjects. Therefore, validation of the present findings will be desirable in a sample of epilepsy patients with different seizure origins, as well as in ECoG recordings from subjects with other neural pathologies, such as tumor patients, and confirmation of our results with non-invasive methods in healthy controls will be important.
As electrode placements were defined solely by clinical demands, the three ATL-implanted subjects included into the present study had different electrode coverage. S2 and S3 had electrodes in the left hemisphere and S1 in the right, and, unlike S1 and S2, S3 had no basal electrodes (see Figure 2). These topographic differences have possibly affected our findings. However, differences in the theta and alpha frequencies were consistently observed across conditions and subjects, and could be observed bilaterally both on the basal and lateral surface of the ATL. Since the amount of ATL-implanted subjects available to the present study was limited, we could not systematically address differences across the hemispheres and basal vs. lateral temporal cortex. These interesting topics need to be addressed in future studies based on a larger group of patients.
Another challenge to non-experimental investigation is that we had to rely on the available amount of video-ECoG data. All three patients had longer conversations with their partners than with the attending physicians, who they only talked to during the relatively brief medical rounds. As a consequence, the number of C1-epochs (partner) surpassed that of C2-epochs (attending physician, see Table 2), and this fact had to be considered in the choice of the statistical procedure which had to be suited for group comparisons with unequal sizes (Sheskin, 2007). Furthermore, due to our non-experimental approach, it was not necessarily human interaction alone that may have affected our results. For instance, non-specific effects due to increased arousal/stress levels in the patients while conversing with their physicians might have contributed to the differences of spectral power across the two conditions. In our study, however, the strongest amplitude differences in the lower frequency bands, especially in the alpha range, were clearly focused on the ATL region, which speaks against an explanation of our findings by a spatially global arousal-related modulation of neural activity. A previous study investigating the effect of naturalistic stressors on alpha-range EEG reports modulations in this frequency range to predominate in frontal areas (Lewis et al., 2007) and not in the ATL region, speaking in favor of the view that the spectral power modulations we observed in the ATL cannot be reduced to non-specific arousal-/stress-related effects.
Apart from the regions of interest in the present study, other human brain areas have been associated with the processing of social cognition such as the medial prefrontal cortex, the anterior cingulate cortex, the inferior frontal gyrus, the temporo-parietal junction, and the amygdala (Frith and Frith, 2007). Further studies might reveal novel insights into neural activation in these and other parts of neural networks for social processing with respect to different communicational situations. Investigations may be also extended to other frequency bands. Although the hypotheses of the present study concerned the alpha and theta bands, we additionally analyzed high gamma activity, and typically found increased spectral power in C2 relative to C1. These changes occurred without any strict relation to the changes in lower frequency bands, possibly indicating a different functional contribution of the high gamma band in the investigated brain regions during social interaction. This frequency band has, among other functions, been linked to increased selective attention (Ray et al., 2008), and thus the greater high gamma in C2 might be related to greater attentional demands during conversations with the attending physician. Enhanced power in the high-gamma band in combination with decreased power in the lower frequencies has been previously proposed to indicate increased information processing (Pfurtscheller and Lopes da Silva, 1999). Thus, our observation of stronger power in gamma together with weaker power in the lower frequencies in C2 compared to C1 may also arise from the higher cognitive load during conversations with the attending physician than with the partner.
The two types of social situations involved different degrees of formality: patients addressed their attending physicians in a more official style than they addressed their life partners. Usually, the choice of non-linguistic and linguistic behaviors depend on whom a person is talking to. Such meta-information may be useful for BMI applications aimed at restoration of expressive speech. In natural discourse, decoding whether a BMI user is talking to a stranger, a friend, or an intimate partner may provide helpful information for selecting the style and register to generate the appropriate speech output. Thus, whereas more official, standard language is preferable while speaking to less familiar people and authorities, more colloquial expressions and non-standard language varieties may be favored in conversations with closer people. Context sensitivity may enable the BMI to switch between social situations and select the corresponding mode of speaking. Context sensitivity would thus be a principle to rule out confusion of possibly competitive (e.g., phonetically similar) terms and prevent inaccurate output. For instance, reliable decoding of the C1 and C2 conditions as investigated in the present study from cortical activity in the ATL could prevent a BMI user from startling the beloved person by calling them “doctor” and complimenting the attending physician “darling.”
An interesting step in the present study was hence to investigate whether the identity of the different conversation partners could be decoded from the ECoG signals in the ATL. Based on the theta and alpha frequency components, such decoding was indeed possible in all patients and significance was highly above chance (Table 4). Here, we classified only two communication partners, and future research will be needed to establish whether and to what extent signals from the ATL can be used to extract information about more and other speakers from ongoing activity. Improved classification may be achieved by using alternative brain regions, signal components, and decoding algorithms. Higher spatial resolution using such recording methodology as micro-ECoG (Blakely et al., 2008; Gierthmuehlen et al., 2011; Viventi et al., 2011) will very likely increase the amount of decodable information. We anticipate that decoding of speaker-related information with such optimized techniques may be a valuable contribution to BMI-based restoration of speech in paralyzed patients.
Outlook
As discussed above, various subfunctions involved in social interaction are likely to have contributed to the observed modulations of neural activity in the present study. Disentangling individual functional aspects that are integral to social interaction will be crucial to address in future research. Many tools are available to characterize different features of real-life behavior at various levels of description. For example, the amount of autobiographical memory units present in natural discourse can be assessed with the system by Dritschel (1991). Many other quantitative systems are available that can be applied to examine human real-life behavior. Thus, the Facial Action Coding System by Ekman and Friesen (1978), available as an automatic tool (Hamm et al., 2011; Maaten and Hendriks, 2011), can be used to infer emotions from facial muscle movements. Approaches from conversation analysis (Sacks et al., 1974), such as the Discussion Coding System, have been utilized to analyze various interpersonal and functional aspects of social interaction (Schermuly et al., 2010). Linguistic methods of discourse analysis can be also applied in neuroscientific research (Brennan et al., 2010), and other aspects of social interaction such as gestures, spatial distance, and body language might be worth investigating. Similar to hyperscanning approaches that employ non-invasive techniques to record simultaneous brain activity from two or more people, even “hyper-ECoG,” or “hyper-ECoG-EEG” studies are conceivable as a way to obtain brain activity measurements from several subjects simultaneously, one or more of them being invasively recorded by means of ECoG.
As discussed in the previous paragraph, a rich spectrum of tools is available that can be applied to refine and extend the real-life ECoG approaches to investigate social interaction. A major purpose of future studies in this direction would be to achieve a better understanding of communication success and failure. Generally, there is much public and scientific interest as to how communicative success during social interaction may affect relationships, e.g., in communication between couples with respect to marital satisfaction (Boland and Follingstad, 1987). Also, several studies showed that specifically for patient-physician interactions, successful communication is causal to patient satisfaction and health status outcome (Stewart, 1984; Jozien, 1991; Staiger et al., 2005). In the present study, we demonstrate that the neural basis of interaction with different communication partners can be traced using ECoG recorded in epilepsy patients. A next step would be to analyze ECoG recordings in epilepsy patients with respect to the success of communication that can be, e.g., quantified using Bales' Interaction Process Analysis (Bales, 1950). This approach might not only reveal the neural signatures of communication, but also provide information that could be used as feedback to improve interaction strategies.
Extraoperative ECoG is a promising candidate signal to study social interaction that may provide new insights into human social cognition. Importantly, such data can be obtained without additional burden to patients and with no need for conducting experiments. A wide range of interaction phenomena and their underlying brain processes can be addressed by means of post hoc analyses. This opportunity to investigate brain activity in non-experimental settings may also inspire further experimental studies. Such a combined approach may be particularly helpful to elucidate the neural basis of human social interaction.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
This work was supported by the German Academic Exchange Service (DAAD) and the German Federal Ministry of Education and Research (BMBF grants 01GQ0420 to BCCN Freiburg, 01GQ0830 to BFNT Freiburg/Tübingen).
References
Aertsen, A. M., Olders, J. H., and Johannesma, P. I. (1981). Spectro-temporal receptive fields of auditory neurons in the grassfrog. III. Analysis of the stimulus-event relation for natural stimuli. Biol. Cybern. 39, 195–209.
Aflalo, T. N., and Graziano, M. S. A. (2006). Partial tuning of motor cortex neurons to final posture in a free-moving paradigm. Proc. Natl. Acad. Sci. U.S.A. 103, 2909–2914.
Babiloni, F., Cincotti, F., Mattia, D., Mattiocco, M., De Vico Fallani, F., Tocci, A., Bianchi, L., Marciani, M. G., and Astolfi, L. (2006). Hypermethods for EEG hyperscanning. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 3666–3669.
Baess, P., Zhdanov, A., Hirvenkari, L., Mäkelä, J. P., Jousmäki, V., and Hari, R. (2012). MEG dual scanning: a procedure to study real-time auditory interaction between two persons. Front. Hum. Neurosci. 6:83. doi: 10.3389/fnhum.2012.00083
Bales, R. F. (1950). Interaction Process Analysis: A Method for the Study of Small Groups. Cambridge, MA: Addison-Wesley.
Ball, T., Kern, M., Mutschler, I., Aertsen, A., and Schulze-Bonhage, A. (2009a). Signal quality of simultaneously recorded invasive and non-invasive EEG. Neuroimage 46, 708–716.
Ball, T., Schulze-Bonhage, A., Aertsen, A., and Mehring, C. (2009b). Differential representation of arm movement direction in relation to cortical anatomy and function. J. Neural Eng. 6, 016006.
Barense, M. D., Henson, R. N. A., and Graham, K. S. (2011). Perception and conception: temporal lobe activity during complex discriminations of familiar and novel faces and objects. J. Cogn. Neurosci. 23, 3052–3067.
Bartels, A., and Zeki, S. (2004). Functional brain mapping during free viewing of natural scenes. Hum. Brain Mapp. 21, 75–85.
Bartlett, S. F. C. (1995). Remembering: A Study in Experimental and Social Psychology. Cambridge: Cambridge University Press.
Başar, E., Güntekin, B., and Oniz, A. (2006). Principles of oscillatory brain dynamics and a treatise of recognition of faces and facial expressions. Prog. Brain Res. 159, 43–62.
Başar, E., Ozgören, M., Oniz, A., Schmiedt, C., and Başar-Eroğlu, C. (2007). Brain oscillations differentiate the picture of one's own grandmother. Int. J. Psychophysiol. 64, 81–90.
Benjamini, Y., and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29, 1165–1188.
Blakely, T., Miller, K. J., Rao, R. P. N., Holmes, M. D., and Ojemann, J. G. (2008). Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2008, 4964–4967.
Boksem, M. A. S., Smolders, R., and De Cremer, D. (2009). Social power and approach-related neural activity. Soc. Cogn. Affect. Neurosci. 7, 516–520.
Boland, J. P., and Follingstad, D. R. (1987). The relationship between communication and marital satisfaction: a review. J. Sex Marital Ther. 13, 286–313.
Brennan, J., Nir, Y., Hasson, U., Malach, R., Heeger, D. J., and Pylkkänen, L. (2010). Syntactic structure building in the anterior temporal lobe during natural story listening. Brain Lang. 120, 163–173.
Brodmann, K. (1909). Vergleichende Lokalisationslehre der Grosshir-nrinde, in Ihren Prinzipien Dar-gestellt auf Grund des Zellenbaues. Leipzig: Johann Ambrosius Barth Verlag.
Brumberg, J. S., Wright, E. J., Andreasen, D. S., Guenther, F. H., and Kennedy, P. R. (2011). Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech-motor cortex. Front. Neurosci. 5:65. doi: 10.3389/fnins.2011.00065
Buzsáki, G., and Draguhn, A. (2004). Neuronal oscillations in cortical networks. Science 304, 1926–1929.
Bédard, C., Kröger, H., and Destexhe, A. (2006). Does the 1/f frequency scaling of brain signals reflect self-organized critical states? Phys. Rev. Lett. 97, 118102.
Crone, N. E., Boatman, D., Gordon, B., and Hao, L. (2001a). Induced electrocorticographic gamma activity during auditory perception. Brazier Award-winning article, 2001. Clin. Neurophysiol. 112, 565–582.
Crone, N. E., Hao, L., Hart, J. Jr., Boatman, D., Lesser, R. P., Irizarry, R., and Gordon, B. (2001b). Electrocorticographic gamma activity during word production in spoken and sign language. Neurology 57, 2045–2053.
Crone, N. E., Miglioretti, D. L., Gordon, B., and Lesser, R. P. (1998a). Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band. Brain 121(Pt 12), 2301–2315.
Crone, N. E., Miglioretti, D. L., Gordon, B., Sieracki, J. M., Wilson, M. T., Uematsu, S., and Lesser, R. P. (1998b). Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. I. Alpha and beta event-related desynchronization. Brain 121(Pt 12), 2271–2299.
Dan, Y., Atick, J. J., and Reid, R. C. (1996). Efficient coding of natural scenes in the lateral geniculate nucleus: experimental test of a computational theory. J. Neurosci. 16, 3351–3362.
Dritschel, B. (1991). Autobiographical memory in natural discourse-a methodological note. Appl. Cogn. Psychol. 5, 319–330.
Dumas, G., Nadel, J., Soussignan, R., Martinerie, J., and Garnero, L. (2010). Inter-brain synchronization during social interaction. PLoS ONE 5:e12166. doi: 10.1371/journal.pone.0012166
Ekman, P., and Friesen, W. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto, CA: Consulting Psychologists Press.
Ethofer, T., Gschwind, M., and Vuilleumier, P. (2011). Processing social aspects of human gaze: a combined fMRI-DTI study. Neuroimage 55, 411–419.
Evarts, E. V. (1965). Relation of discharge frequency to conduction velocity in pyramidal tract neurons. J. Neurophysiol. 28, 216–228.
Fletcher, P. C., Happé, F., Frith, U., Baker, S. C., Dolan, R. J., Frackowiak, R. S., and Frith, C. D. (1995). Other minds in the brain: a functional imaging study of “theory of mind” in story comprehension. Cognition 57, 109–128.
Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J.-P., Frith, C. D., and Frackowiak, R. S. J. (1994). Statistical parametric maps in functional imaging: a general linear approach. Hum. Brain Mapp. 2, 189–210.
Gale, A., Spratt, G., Chapman, A. J., and Smallbone, A. (1975). EEG correlates of eye contact and interpersonal distance. Biol. Psychol. 3, 237–245.
Gierthmuehlen, M., Ball, T., Henle, C., Wang, X., Rickert, J., Raab, M., Freiman, T., Stieglitz, T., and Kaminsky, J. (2011). Evaluation of μ ECoG electrode arrays in the minipig: experimental procedure and neurosurgical approach. J. Neurosci. Methods 202, 77–86.
Golland, Y., Bentin, S., Gelbard, H., Benjamini, Y., Heller, R., Nir, Y., Hasson, U., and Malach, R. (2007). Extrinsic and intrinsic systems in the posterior cortex of the human brain revealed during natural sensory stimulation. Cereb. Cortex 17, 766–777.
Good, M.-J. D., and Good, B. (1982). “Patient Requests in Primary Care Clinics,” in Clinically Applied Anthropology, eds N. J. Chrisman and T. Maretzki (Dordrecht, Boston, London: D. Reidel Publishing Co.), 275–295.
Gottman, J. M., and Notarius, C. I. (2000). Decade review: observing marital interaction. J. Marriage Fam. 62, 927–947.
Gruber, T., and Müller, M. M. (2006). Oscillatory brain activity in the human EEG during indirect and direct memory tasks. Brain Res. 1097, 194–204.
Guenther, F. H., Brumberg, J. S., Wright, E. J., Nieto-Castanon, A., Tourville, J. A., Panko, M., Law, R., Siebert, S. A., Bartels, J. L., Andreasen, D. S., Ehirim, P., Mao, H., and Kennedy, P. R. (2009). A wireless brain-machine interface for real-time speech synthesis. PLoS ONE 4:e8218. doi: 10.1371/journal.pone.0008218
Haider, B., Krause, M. R., Duque, A., Yu, Y., Touryan, J., Mazer, J. A., and McCormick, D. A. (2010). Synaptic and network mechanisms of sparse and reliable visual cortical activity during nonclassical receptive field stimulation. Neuron 65, 107–121.
Hamm, J., Kohler, C. G., Gur, R. C., and Verma, R. (2011). Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. J. Neurosci. Methods 200, 237–256.
Hari, R., and Kujala, M. V. (2009). Brain basis of human social interaction: from concepts to brain imaging. Psychol. Rev. 89, 453–479.
Hasson, U., Malach, R., and Heeger, D. J. (2010). Reliability of cortical activity during natural stimulation. Trends Cogn. Sci. (Regul. Ed.) 14, 40–48.
Hentschke, H., and Stüttgen, M. C. (2011). Computation of measures of effect size for neuroscience data sets. Eur. J. Neurosci. 34, 1887–1894.
Herikstad, R., Baker, J., Lachaux, J.-P., Gray, C. M., and Yen, S.-C. (2011). Natural movies evoke spike trains with low spike time variability in cat primary visual cortex. J. Neurosci. 31, 15844–15860.
Jackson, A., Mavoori, J., and Fetz, E. E. (2007). Correlations between the same motor cortex cells and arm muscles during a trained task, free behavior, and natural sleep in the macaque monkey. J. Neurophysiol. 97, 360–374.
Jackson, A., Moritz, C. T., Mavoori, J., Lucas, T. H., and Fetz, E. E. (2006). The neurochip BCI: towards a neural prosthesis for upper limb function. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 187–190.
Jozien, B. (1991). Doctor-patient communication and the quality of care. Soc. Sci. Med. 32, 1301–1310.
Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., and Greger, B. (2010). Classification of spoken words using surface local field potentials. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2010, 3827–3830.
Kondo, H., Saleem, K. S., and Price, J. L. (2003). Differential connections of the temporal pole with the orbital and medial prefrontal networks in macaque monkeys. J. Comp. Neurol. 465, 499–523.
Kovach, C. K., Tsuchiya, N., Kawasaki, H., Oya, H., Howard, M. A. 3rd, and Adolphs, R. (2011). Manifestation of ocular-muscle EMG contamination in human intracranial recordings. Neuroimage 54, 213–233.
Lerner, Y., Honey, C. J., Silbert, L. J., and Hasson, U. (2011). Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31, 2906–2915.
Leuthardt, E. C., Gaona, C., Sharma, M., Szrama, N., Roland, J., Freudenberg, Z., Solis, J., Breshears, J., and Schalk, G. (2011). Using the electrocorticographic speech network to control a brain-computer interface in humans. J. Neural Eng. 8, 036004.
Leuthardt, E. C., Miller, K. J., Schalk, G., Rao, R. P. N., and Ojemann, J. G. (2006). Electrocorticography-based brain computer interface–the Seattle experience. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 194–198.
Lewis, R. S., Weekes, N. Y., and Wang, T. H. (2007). The effect of a naturalistic stressor on frontal EEG asymmetry, stress, and health. Biol. Psychol. 75, 239–247.
Lindenberger, U., Li, S.-C., Gruber, W., and Müller, V. (2009). Brains swinging in concert: cortical phase synchronization while playing guitar. BMC Neurosci. 10, 22.
Maaten, L., and Hendriks, E. (2011). Action unit classification using active appearance models and conditional random fields. Cogn. Process. doi: 10.1007/s10339-011-0419-7. [Epub ahead of print].
Mar, R. A. (2011). The neural bases of social cognition and story comprehension. Annu. Rev. Psychol. 62, 103–134.
Mavoori, J., Jackson, A., Diorio, C., and Fetz, E. (2005). An autonomous implantable computer for neural recording and stimulation in unrestrained primates. J. Neurosci. Methods 148, 71–77.
Mechler, F., Victor, J. D., Purpura, K. P., and Shapley, R. (1998). Robust temporal coding of contrast by V1 neurons for transient but not for steady-state stimuli. J. Neurosci. 18, 6583–6598.
Miller, K. J., Sorensen, L. B., Ojemann, J. G., and den Nijs, M. (2009). Power-law scaling in the brain surface electric potential. PLoS Comput. Biol. 5:e1000609. doi: 10.1371/journal.pcbi.1000609
Montague, P. R., Berns, G. S., Cohen, J. D., McClure, S. M., Pagnoni, G., Dhamala, M., Wiest, M. C., Karpov, I., King, R. D., Apple, N., and Fisher, R. E. (2002). Hyperscanning: simultaneous fMRI during linked social interactions. Neuroimage 16, 1159–1164.
Morán, M. A., Mufson, E. J., and Mesulam, M. M. (1987). Neural inputs into the temporopolar cortex of the rhesus monkey. J. Comp. Neurol. 256, 88–103.
Mukamel, R., Gelbard, H., Arieli, A., Hasson, U., Fried, I., and Malach, R. (2005). Coupling between neuronal firing, field potentials, and FMRI in human auditory cortex. Science 309, 951–954.
Naeem, M., Prasad, G., Watson, D. R., and Kelso, J. A. S. (2012). Electrophysiological signatures of intentional social coordination in the 10-12Hz range. Neuroimage 59, 1795–1803.
Nakamura, K., Kawashima, R., Sato, N., Nakamura, A., Sugiura, M., Kato, T., Hatano, K., Ito, K., Fukuda, H., Schormann, T., and Zilles, K. (2000). Functional delineation of the human occipito-temporal areas related to face and scene processing. A PET study. Brain 123(Pt 9), 1903–1912.
Nakamura, K., Kawashima, R., Sugiura, M., Kato, T., Nakamura, A., Hatano, K., Nagumo, S., Kubota, K., Fukuda, H., Ito, K., and Kojima, S. (2001). Neural substrates for recognition of familiar voices: a PET study. Neuropsychologia 39, 1047–1054.
Nelken, I. (2004). Processing of complex stimuli and natural scenes in the auditory cortex. Curr. Opin. Neurobiol. 14, 474–480.
Nowak, P. (2011). Synthesis of qualitative linguistic research-a pilot review integrating and generalizing findings on doctor-patient interaction. Patient Educ. Couns. 82, 429–441.
Olson, I. R., Plotzker, A., and Ezzyat, Y. (2007). The Enigmatic temporal pole: a review of findings on social and emotional processing. Brain 130, 1718–1731.
Ong, L. M., de Haes, J. C., Hoos, A. M., and Lammes, F. B. (1995). Doctor-patient communication: a review of the literature. Soc. Sci. Med. 40, 903–918.
Pei, X., Barbour, D. L., Leuthardt, E. C., and Schalk, G. (2011). Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J. Neural Eng. 8, 046028.
Pei, X., Hill, J., and Schalk, G. (2012). Silent communication: toward using brain signals. IEEE Pulse 3, 43–46.
Pennebaker, J. W., Mehl, M. R., and Niederhoffer, K. G. (2003). Psychological aspects of natural language. use: our words, our selves. Annu. Rev. Psychol. 54, 547–577.
Pfeiffer, U. J., Timmermans, B., Bente, G., Vogeley, K., and Schilbach, L. (2011). A non-verbal turing test: differentiating mind from machine in gaze-based social interaction. PLoS ONE 6:e27591. doi: 10.1371/journal.pone.0027591
Pfurtscheller, G., and Lopes da Silva, F. H. (1999). Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin. Neurophysiol. 110, 1842–1857.
Pistohl, T., Ball, T., Schulze-Bonhage, A., Aertsen, A., and Mehring, C. (2008). Prediction of arm movement trajectories from ECoG-recordings in humans. J. Neurosci. Methods 167, 105–114.
Pistohl, T., Schulze-Bonhage, A., Aertsen, A., Mehring, C., and Ball, T. (2012). Decoding natural grasp types from human ECoG. Neuroimage 59, 248–260.
Privman, E., Nir, Y., Kramer, U., Kipervasser, S., Andelman, F., Neufeld, M. Y., Mukamel, R., Yeshurun, Y., Fried, I., and Malach, R. (2007). Enhanced category tuning revealed by intracranial electroencephalograms in high-order human visual areas. J. Neurosci. 27, 6234–6242.
Ray, S., Niebur, E., Hsiao, S. S., Sinai, A., and Crone, N. E. (2008). High-frequency gamma activity (80-150Hz) is increased in human cortex during selective attention. Clin. Neurophysiol. 119, 116–133.
Ray, W. J. (2002). Methods Toward a Science of Behavior and Experience, 7th Edn. Belmont, CA: Wadsworth Publishing.
Rosner, B., and Glynn, R. J. (2009). Power and sample size estimation for the Wilcoxon rank sum test with application to comparisons of C statistics from alternative prediction models. Biometrics 65, 188–197.
Roter, D. L., and Hall, J. A. (1989). Studies of doctor-patient interaction. Annu. Rev. Public Health 10, 163–180.
Sacks, H., Schegloff, E. A., and Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language 50, 696–735.
Schalk, G., Kubánek, J., Miller, K. J., Anderson, N. R., Leuthardt, E. C., Ojemann, J. G., Limbrick, D., Moran, D., Gerhardt, L. A., and Wolpaw, J. R. (2007). Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J. Neural Eng. 4, 264–275.
Schermuly, C., Schröder, T., Nachtwei, J., and Scholl, W. (2010). Das Instrument zur Kodierung von Diskussionen (IKD)1-Ein Verfahren zur zeitökonomischen und validen Kodierung von Interaktionen in Organisationen. Z. Arb. Organ. 54, 149–170.
Sheskin, D. J. (2007). Handbook of Parametric and Nonparametric Statistical Procedures, 4th Edn. Boca Raton, FL: Chapman and Hall.
Sillars, A., and Scott, M. (1983). Interpersonal perception between intimates-an integrative review. Hum. Commun. Res. 10, 153–176.
Sinai, A., Bowers, C. W., Crainiceanu, C. M., Boatman, D., Gordon, B., Lesser, R. P., Lenz, F. A., and Crone, N. E. (2005). Electrocorticographic high gamma activity versus electrical cortical stimulation mapping of naming. Brain 128, 1556–1570.
Smolders, J. W., Aertsen, A. M., and Johannesma, P. I. (1979). Neural representation of the acoustic biotope. A comparison of the response of auditory neurons to tonal and natural stimuli in the cat. Biol. Cybern. 35, 11–20.
Spreng, R. N., Mar, R. A., and Kim, A. S. (2009). The common neural basis of autobiographical memory, prospection, navigation, theory of mind, and the default mode: a quantitative meta-analysis. J. Cog. Neurosci. 21, 489–510.
Staiger, T. O., Jarvik, J. G., Deyo, R. A., Martin, B., and Braddock, C. H. (2005). Brief Report: patient-physician agreement as a predictor of outcomes in patients with back pain. J. Gen. Intern. Med. 20, 935–937.
Steinvorth, S., Wang, C., Ulbert, I., Schomer, D., and Halgren, E. (2010). Human entorhinal gamma and theta oscillations selective for remote autobiographical memory. Hippocampus 20, 166–173.
Stewart, M. A. (1984). What is a successful doctor-patient interview? A study of interactions and outcomes. Soc. Sci. Med. 19, 167–175.
Suga, N. (1978). Specialization of the auditory system for reception and processing of species-specific sounds. Fed. Proc. 37, 2342–2354.
Sugiura, M., Mano, Y., Sasaki, A., and Sadato, N. (2011). Beyond the memory mechanism: person-selective and nonselective processes in recognition of personally familiar faces. J. Cogn. Neurosci. 23, 699–715.
Sugiura, M., Sassa, Y., Watanabe, J., Akitsuki, Y., Maeda, Y., Matsue, Y., and Kawashima, R. (2009). Anatomical segregation of representations of personally familiar and famous people in the temporal and parietal cortices. J. Cogn. Neurosci. 21, 1855–1868.
Tognoli, E., Lagarde, J., DeGuzman, G. C., and Kelso, J. A. S. (2007). The phi complex as a neuromarker of human social coordination. Proc. Natl. Acad. Sci. U.S.A. 104, 8190–8195.
Towle, V. L., Yoon, H.-A., Castelle, M., Edgar, J. C., Biassou, N. M., Frim, D. M., Spire, J.-P., and Kohrman, M. H. (2008). ECoG gamma activity during a language task: differentiating expressive and receptive speech areas. Brain 131, 2013–2027.
Vinje, W. E., and Gallant, J. L. (2000). Sparse coding and decorrelation in primary visual cortex during natural vision. Science 287, 1273–1276.
Visser, M., Jefferies, E., and Lambon Ralph, M. A. (2010). Semantic processing in the anterior temporal lobes: a meta-analysis of the functional neuroimaging literature. J. Cogn. Neurosci. 22, 1083–1094.
Viventi, J., Kim, D.-H., Vigeland, L., Frechette, E. S., Blanco, J. A., Kim, Y.-S., Avrin, A. E., Tiruvadi, V. R., Hwang, S.-W., Vanleer, A. C., Wulsin, D. F., Davis, K., Gelber, C. E., Palmer, L., Van der Spiegel, J., Wu, J., Xiao, J., Huang, Y., Contreras, D., Rogers, J. A., and Litt, B. (2011). Flexible, foldable, actively multiplexed, high-density electrode array for mapping brain activity in vivo. Nat. Neurosci. 14, 1599–1605.
Wang, W., Degenhart, A. D., Sudre, G. P., Pomerleau, D. A., and Tyler-Kabara, E. C. (2011). Decoding semantic information from human electrocorticographic (ECoG) signals. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 6294–6298.
Whatmough, C., Chertkow, H., Murtha, S., and Hanratty, K. (2002). Dissociable brain regions process object meaning and object structure during picture naming. Neuropsychologia 40, 174–186.
Wilms, M., Schilbach, L., Pfeiffer, U., Bente, G., Fink, G. R., and Vogeley, K. (2010). It's in your eyes–using gaze-contingent stimuli to create truly interactive paradigms for social cognitive and affective neuroscience. Soc. Cogn. Affect. Neurosci. 5, 98–107.
Wolpert, D. M., Doya, K., and Kawato, M. (2003). A unifying computational framework for motor control and social interaction. Philos. Trans. R. Soc. Lond. B Biol. Sci. 358, 593–602.
Xu, J., Kemeny, S., Park, G., Frattali, C., and Braun, A. (2005). Language in context: emergent features of word, sentence, and narrative comprehension. Neuroimage 25, 1002–1015.
Yao, H., Shi, L., Han, F., Gao, H., and Dan, Y. (2007). Rapid learning in cortical coding of visual scenes. Nat. Neurosci. 10, 772–778.
Yen, S.-C., Baker, J., and Gray, C. M. (2007). Heterogeneity in the responses of adjacent neurons to natural stimuli in cat striate cortex. J. Neurophysiol. 97, 1326–1341.
Zacks, J. M., Braver, T. S., Sheridan, M. A., Donaldson, D. I., Snyder, A. Z., Ollinger, J. M., Buckner, R. L., and Raichle, M. E. (2001). Human brain activity time-locked to perceptual event boundaries. Nat. Neurosci. 4, 651–655.
Appendix
Table A1. Statistically significant effects (p < 0.001, FDR) in the theta (θ), alpha (α), and gamma (γ) frequency band, MNI coordinates, and anatomical locations of ATL electrodes in S1 and S3.
Keywords: natural behavior, temporal pole, theta, alpha, language, speech, BMI, BCI
Citation: Derix J, Iljina O, Schulze-Bonhage A, Aertsen A and Ball T (2012) “Doctor” or “darling”? Decoding the communication partner from ECoG of the anterior temporal lobe during non-experimental, real-life social interaction. Front. Hum. Neurosci. 6:251. doi: 10.3389/fnhum.2012.00251
Received: 03 February 2012; Accepted: 16 August 2012;
Published online: 05 September 2012.
Edited by:
Kai Vogeley, University Hospital Cologne, GermanyReviewed by:
Eva K. Ritzl, The Johns Hopkins Hospital, USAMohamad Koubeissi, George Washington University, USA
Copyright © 2012 Derix, Iljina, Schulze-Bonhage, Aertsen and Ball. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.
*Correspondence: Tonio Ball and Johanna Derix, Epilepsy Center, University Medical Center Freiburg, Engelbergerstr. 21, 79106 Freiburg, Germany. e-mail: tonio.ball@uniklinik-freiburg.de; johanna.derix@uniklinik-freiburg.de