Worth a glance: using eye movements to investigate the cognitive neuroscience of memory
- 1 Department of Psychology, University of Wisconsin, Milwaukee, WI, USA
- 2 Department of Psychiatry, University of Vermont, Burlington, VT, USA
- 3 Department of Psychology, University of Vermont, Burlington, VT, USA
- 4 Department of Pediatrics, University of Vermont, Burlington, VT, USA
- 5 Department of Neurology, University of Iowa, Iowa City, IA, USA
- 6 Rotman Research Institute, Baycrest, Toronto, ON, Canada
- 7 Department of Psychology, University of Toronto, Toronto, ON, Canada
- 8 Beckman Institute and Department of Psychology, University of Illinois, Urbana-Champaign, IL, USA
- 9 Department of Psychiatry, Division of Geriatric Psychiatry, University of Toronto, Toronto, ON, Canada
Results of several investigations indicate that eye movements can reveal memory for elements of previous experience. These effects of memory on eye movement behavior can emerge very rapidly, changing the efficiency and even the nature of visual processing without appealing to verbal reports and without requiring conscious recollection. This aspect of eye movement based memory investigations is particularly useful when eye movement methods are used with special populations (e.g., young children, elderly individuals, and patients with severe amnesia), and also permits use of comparable paradigms in animals and humans, helping to bridge different memory literatures and permitting cross-species generalizations. Unique characteristics of eye movement methods have produced findings that challenge long-held views about the nature of memory, its organization in the brain, and its failures in special populations. Recently, eye movement methods have been successfully combined with neuroimaging techniques such as fMRI, single-unit recording, and magnetoencephalography, permitting more sophisticated investigations of memory. Ultimately, combined use of eye-tracking with neuropsychological and neuroimaging methods promises to provide a more comprehensive account of brain–behavior relationships and adheres to the “converging evidence” approach to cognitive neuroscience.
In this review we advocate for the use of eye movement monitoring as a powerful tool that can advance the field of cognitive neuroscience. Because eye movement based investigations of attention and language have been described comprehensively in several excellent reviews (cf. Van der Stigchel et al., 2006; Rayner, 2009; Theeuwes et al., 2009), emphasis is placed on investigations of memory; however, the general approach is expected to have merit for other domains of investigation as well. In the sections that follow we will show that viewers’ eye movements can reveal memory for elements of previous experience without appealing to verbal reports and without requiring conscious recollection. We also point to research which has shown that the effects of memory on eye movement behavior can emerge very rapidly, changing the efficiency and even the nature of visual processing. These effects have been characterized as obligatory (and perhaps automatic) and are consistently reported independent of task instructions or viewers’ intentions. Taking advantage of these properties, several important questions have been addressed and certain long-held views about the nature of memory, its organization in the brain, and its failures in certain special populations have been challenged. In general, a converging methods approach is highlighted, with special emphasis placed on the utility of eye movement monitoring when it is used in conjunction with other cognitive neuroscience methods.
Measurement of memory with eye movements is powerful because the nature of visual processing requires that we sample one region of the world at a time by directing the high-acuity portion of our retinas onto successively sampled regions. This movement of the eyes across the visual world is not random, but rather is dictated by two factors. The first concerns stimulus characteristics – the physical properties (e.g., luminance, hue) of the elements embedded in visual arrays (e.g., Buswell, 1935; Mackworth and Morandi, 1967; Antes, 1974). The second concerns our previous experience (i.e., episodic memory) and the knowledge we bring to a particular viewing situation (i.e., semantic memory). We look more at objects that are the targets of volitional search, but also dwell on objects that are discrepant with expectations, based, for example, on general knowledge about the context in which they are embedded (e.g., an octopus in a barnyard scene; Loftus and Mackworth, 1978), or on previous viewing history in a laboratory setting (see Revealing Distinct Types of Mnemonic Information).
When control of basic physical properties has been achieved it becomes possible to discern effects of memory on eye movement behavior. An early example of this strategy comes from a report by Yarbus (1967). In this work, eye movements were recorded as a viewer examined a picture by the Russian artist Repin depicting the unexpected return of a man, who is shown entering a room filled with various people and objects. Eye movements elicited by this scene under free-viewing conditions were largely to prominent visual elements – the man and objects in the room. The same viewer was then asked to examine the scene again while providing answers to several different questions (e.g., how old are the people in the scene, what might they have been doing before the arrival of the unexpected visitor). Different patterns of viewing were observed across these instructional conditions, presumably reflecting (pre-experimental) knowledge the viewer brought to bear about where in the scene relevant information would be located.
In what follows, we illustrate just how powerful eye movement methods can be for addressing important questions about memory. In short, by using eye movements to assess memory, rather than merely relying on verbal reports or introspective judgments, we gain the ability to test memory under circumstances in which behavioral reports may not (or cannot) be reliably obtained. This pays particular dividends when eye movement methods are used with special populations (e.g., young children, elderly individuals, and patients with severe amnesia), and also permits use of comparable paradigms in animals and humans, helping to bridge different memory literatures and permitting cross-species generalizations.
To best illustrate the use of eye movements to advance the cognitive neuroscience of memory, we will provide an extended example of how eye movement studies offered a way to resolve a major disagreement in the field about the nature of amnesia and the role of the hippocampus in memory (see Revealing a Critical Role for the Hippocampus in Memory Without Awareness). In this work (Althoff, 1998; Ryan et al., 2000; Hannula et al., 2007), eye movement methods provided critical data about the role of hippocampus in relational memory, as distinct from the role it may play in explicit memory or conscious recollection. Accordingly, we will discuss and attempt to resolve a number of controversies about memory, awareness, behavioral and eye movement measures, and the hippocampus. Promising results from some recent efforts to apply these methods to other patients and special populations will also be presented.
Finally, and as alluded to above, we will discuss the particular strengths of eye movement measures as part of a larger converging methods approach to the study of memory, outlining some promising early steps in relating behavioral, eye movement, and neuroimaging measures (Herdman and Ryan, 2007; Ryan et al., 2007; Hannula and Ranganath, 2009; Riggs et al., 2009; see Integration of Eye Movement Monitoring With Neuroimaging). Including eye movement monitoring in neuropsychological and neuroimaging investigations provides a comprehensive account of the brain-behavior relationship and adheres to the “converging evidence” approach to cognitive neuroscience.
Revealing Distinct Types of Mnemonic Information
Before the field of cognitive neuroscience could use eye movement methods to link the function of specific neural regions to particular aspects of memory, it was necessary to produce evidence showing that memory does indeed influence eye movement behavior. This section provides a very brief, and necessarily selective, discussion of studies aimed at documenting eye movement based memory effects.
An early line of work on eye movements and memory was guided by the “scanpath hypothesis” (Noton and Stark, 1971), which postulated that recognition occurs when the same scanning pattern enacted during initial viewings is recapitulated during subsequent viewing and recognition of a stimulus. Consistent with this hypothesis, striking similarity in scanning patterns between initial inspection and subsequent recognition was indeed observed by some investigators for a variety of materials and tasks (Walker-Smith et al., 1977; Fisher et al., 1978; Parker, 1978). However, these investigators interpreted their results as suggesting that during recognition (e.g., for a face), a participant would move to features more salient to him or her after first registering the presence of the object. They proposed a filtering system that would interpret incoming data from the periphery, identify the most salient feature, and then direct a saccade towards it (Didday and Arbib, 1975; see also Carpenter and Just, 1977). Starting with this work and continuing through ever more sophisticated studies, multiple measures were developed to characterize the types of changes that might occur in eye movement behavior that reflect – and, thus, that reveal – the operation of mnemonic processes (see Boxes 1 and 2).
Box 1. Eye movement measures of memory.
Eye movement data can be compiled and analyzed in several different ways. Sampling of visual materials can be characterized either at the level of an entire experimental display (i.e., overall viewing), or at the level of regions, objects, or stimuli within that display (i.e., directed viewing). For example, eye movements to a display containing several faces could be summarized as the total number of fixations made to the display. Alternatively, fixations to the display could be categorized according to their targets (e.g., novel and studied faces), and summarized separately. Measures of directed viewing divide a display into more than one region of interest (ROI) in order to permit evaluation of effects related to independent variables of interest. Directed viewing measures may also be combined with temporal indices in order to gauge changes in a particular measure over time (i.e., time-course analyses), or may be time-locked to a particular event, such as an overt behavioral response (i.e., response-locked analyses). Various commonly used characterizations of overall and directed viewing are defined below.
Measures of overall viewing
• Number of fixations1: the number of discrete pauses of the eyes for a display.
• Fixation duration1: the length of time in which the eye pauses on a display, typically between 200–300 ms long. Median or mean fixation duration to a display can be calculated.
• Saccade amplitude: the distance traversed between successive fixations, reported in degrees per second.
• Number of regions fixated2: the number of discrete regions sampled within a display.
• Number of transitions between regions: the number of transitions made by the eyes between discrete regions.
• First return fixation: the number of fixations made before returning to a previously sampled region.
• First-order entropy3: the predictability of the transitions between the locations of a given fixation and the preceding fixation.
• Second-order entropy3: the predictability of the transitions to a given fixation location based on the location of the two immediately preceding fixations.
• Chi-square, Asymmetric lambda2: other measures used to quantify the randomness of an eye movement transition table.
Measures of directed viewing• Proportion of fixations: the proportion of total fixations that are directed to an experimenter-defined ROI.
• Proportion of time2: the proportion of total viewing time that is directed to an experimenter-defined ROI.
• Number of transitions into/out of a critical region: the number of gaze transitions into/out of an experimenter-defined ROI.
• Duration of the first gaze: total viewing time to an experimenter-defined ROI on the first gaze that is directed into that ROI.
• Number of fixations in the first gaze: number of fixations to an experimenter-defined ROI during the first gaze that is directed into that ROI.
Considerations for Analysis
While the above outlines general definitions for the predominant measures derived from eye movement monitoring, there are differences in how such measures are calculated. For instance, the definition of a fixation may vary between research groups and/or eyetracking platforms. Successive recording samples of eye position may be considered as a single fixation if changes in gaze position across samples are less than 1° of visual angle and, when combined, have a minimum duration of 100 ms (e.g., Hannula et al., 2007). Alternatively (e.g., Ryan et al., 2007), a fixation can be defined as the absence of any saccade (e.g., the velocity of two successive eye movement samples exceeds 22° per second over a distance of 0.1°), or blink (e.g., pupil is missing for three or more samples) activity. In turn, definitions for saccades and blinks may vary across labs/platforms.
Regarding the number of regions fixated, in our work (cf. Althoff and Cohen, 1999) this number is defined based on the pattern by which each participant’s fixations are clustered; that is, fixations that fall within an a priori specified distance from one another are considered to be within the same region, while fixations outside of this distance are considered to belong to a separate region. This approach takes into account individual differences in viewing behavior as well as differences in particular stimulus characteristics, such as the size or prominence of distinct features on an object. An alternative approach uses a fixed grid with equally-sized sections for all of the stimuli and all of the participants indiscriminately (e.g., Smith and Squire, 2008). This latter approach has an advantage of standardizing the number of possible regions that could be fixated across stimuli and participants, but this approach may lack precision and/or sensitivity. For example, if an object attracts disproportionate viewing, and occupies two or more sections of the grid, then the measure of sampling behavior may be artificially inflated; by contrast, if two smaller objects occupy the same section, and each is distinctly fixated by the viewer, the number of unique regions fixated is under-sampled.
At least two different approaches have also been used to calculate the proportion of time that is directed to a ROI. In our work, the proportion is considered in reference to a baseline in which only the amount of viewing time that is directed to the stimulus (e.g., a scene) is counted, excluding any time spent looking outside of the display, blinking, or making saccadic movements of the eyes (e.g., Ryan et al., 2000; Hannula et al., 2007). The alternative approach (e.g., Smith and Squire, 2008) involves calculating the proportion of viewing time using the entire trial period as the baseline, regardless of how much time was spent not actually viewing the stimulus. The number of saccades, as well as the number of blinks, varies from image to image, both within and across participants; therefore, the actual time that is spent inspecting a stimulus is never constant. Thus, the use of total trial duration in the denominator will either artificially inflate or reduce the proportion of viewing time measure differentially across trials, participants, and experimental conditions. As such, we do not advocate this latter approach as it introduces noise into the data.
Box 2. Markovian measures.
To determine whether there is particular top-down influence of cognition on eye movement patterns, several investigators turned to the examination of scanpaths. To describe scanpaths, contingency tables are created; these tables examine the probability of moving from one ROI to another within a display and can be probed for their structure. The idea that eye movements followed a first-order Markovian process (i.e., a stochastic process in which any transition between states depends only the current state) had been hypothesized as early as 1966 by Molnar (as described in Molnar and Ratsikas, 1987). The simplest examination of this structure is a chi-square test which examines the probability of seeing a particular pattern in the contingency table over and above what would be expected by chance. Stark and Ellis (1981) applied a chi-square goodness of fit calculation to determine the difference between observed and expected matrices M(1) where the expected matrix was based on the stratified random situation (i.e., the situation where a particular saccade could be predicted entirely by the marginals of the contingency table). The results from this experiment yielded the expected finding that there were significant differences between the observed and expected first-order dependencies as measured by chi-square. To calculate the change in the direction of dependency in those data, he further calculated the conditional information Hc in both the observed contingency table M(1) and the expected contingency table M(1) (see Stark and Ellis, 1981; Althoff and Cohen, 1999 or Althoff, 1998 for calculation details). This provided a measure of statistical dependency in the matrix (Brillouin, 1962); the larger the Hc, the less statistical dependency there was in the matrix, and the more random the eye movements were. One could, therefore, simply take this information as a measure of the amount of randomness in the signal, with a high degree of entropy (Hc) indicating a high degree of randomness. Further refining these methods, Ellis and Stark (1986) modeled the behavior of pilots in viewing a cockpit display of traffic information by examining whether sampling of the regions of the a display was “random”, “stratified random” (i.e., dependent only on the information on the marginals of a contingency table) or “statistically dependent” (i.e., with dependencies in the data that linked movements from one region of the display to another). In our work, we have termed these variables H1 for the first-order transitions and H2 for the second-order transitions.
Since different participants have different path lengths (i.e., number of fixations), Tole et al. (1982) normalized these entropy measures for the number of fixations to show a decrease in the entropy of a scanning pattern when pilots were under an increased workload. In other words, as pilots became stressed, their scanning patterns became less random and more structured, revealing the top-down influence of cognition and task demands on eye movement behavior. These types of entropy measures have been used by our group (termed S1 and S2) and others to demonstrate that viewing of an unfamiliar face or scene is typically highly constrained (i.e., has low entropy) with viewers tending to follow an idiosyncratic pattern while scanning novel stimuli (Althoff and Cohen, 1999) which becomes less so as faces or scenes have been viewed repeatedly.
Robust effects of semantic memory on the way in which we evaluate or extract information from a visual stimulus have been clearly demonstrated (cf. Henderson, 2003). For example, general world knowledge about the context in which objects are typically found (e.g., a tractor is usually found in a barnyard scene) and their relative positions within the environment (e.g., a toaster is typically on a counter) has been found to facilitate the speed with which the eyes detect a visual target (e.g., Loftus and Mackworth, 1978; Henderson et al., 1999; Brockmole and Henderson, 2008; Hollingworth, 2009; see Figure 1). Target detection is also facilitated by repeated exposure to a specific scene context in the lab (Brockmole et al., 2006), and by brief exposure to, or preview of, that scene (Castelhano and Henderson, 2007; Hollingworth, 2009).
Figure 1. Illustration of the experimental methods and results from Loftus and Mackworth (1978). Participants viewed scenes in which an object was either consistent (i.e., a tractor; top), or inconsistent (i.e., octopus; bottom) with the semantic context of the scene (i.e., barnyard). Objects that were inconsistent with the scene context were viewed with longer fixation durations than non-informative objects, and this increase in viewing occurred within the first few fixations, revealing early and obligatory influences of prior knowledge on eye movements.
In our work, we have examined how eye movements reveal memory for pre-experimentally familiar materials and specific prior experiences in a series of studies conducted with images of famous and non-famous faces (Althoff and Cohen, 1999), familiar and unfamiliar buildings (Althoff et al., 1998), and novel scenes (Ryan et al., 2000). In these and subsequent studies, eye movement behavior revealed memory for previous occurrence of these various types of items. Pre-experimentally familiar items were viewed with fewer fixations and with lower constraint on the location of successive fixations than were novel items. Furthermore, with repeated exposure to pre-experimentally unfamiliar items, the amount of sampling decreased systematically (see Figure 2; Heisz and Ryan, submittedAlthoff, 1998; Althoff et al., 1998; Heisz and Shore, 2008; ). This repetition effect was observed whether participants performed a recognition task or an emotion labeling task (i.e., regardless of task demands; Althoff et al., 1998).
Figure 2. Illustration of the experimental methods and results from Althoff (1998). (A) Participants viewed non-famous faces either 0 (novel), 1, 3 or 5 times throughout the experiment while their eye movements were monitored. (B) With increasing exposure, participants sampled fewer regions of the studied faces, compared to novel (non-studied) faces that were introduced during the same phase of the experiment. These findings illustrate the influence of memory for the previously viewed faces on subsequent processing.
Subsequent experiments, which used pre-experimentally unfamiliar materials, demonstrated that eye movements could also reveal relational memory for the spatial positions of elements within scenes (e.g., Ryan et al., 2000; Ryan and Cohen, 2004a,b), arbitrary pairings of faces and scenes (Hannula et al., 2007; Hannula and Ranganath, 2009), and temporal sequences (Ryan and Villate, 2009). In the work with scenes (Ryan et al., 2000), participants viewed several pictures, each accompanied by a question regarding the relative relationships among scene elements (e.g., is there a cat to the left of the boy?). During a final, critical block, eye movements were monitored as participants viewed scenes that were either novel (i.e., not studied), repeated (i.e., unchanged from previous exposures), or manipulated (i.e., with a change in spatial relationships among elements). Critically, a manipulated scene for one participant was repeated for another, permitting comparison of viewing directed to the exact same regions across participants for whom the only difference was in viewing history. Results showed more fixations to, and transitions into and out of, the critical region(s) when scenes were manipulated versus repeated. This relational manipulation effect was documented in four separate experiments, and was evident whether participants were instructed to identify changes or were merely instructed to view the scenes (i.e., free-viewing). Furthermore, it did not depend upon use of orienting questions meant to direct a viewer’s attention and gaze to regions of scenes that might ultimately be manipulated, as it was documented even when these questions were not used (see Ryan et al., 2000 for details; see also Ryan and Cohen, 2004a). Particularly compelling were findings of greater viewing directed to “now-empty” manipulated regions as compared to the exact same regions when they were “always empty” (see Figure 3), revealing the effects of relational memory on current processing.
Figure 3. Illustration of the experimental methods and results from Ryan et al. (2000). (A) Scenes presented during the study phase and (B) during the test phase, as seen by two different participants. In this case, the scenes in the test phase are exactly the same, and differ only with respect to the previous viewing histories of the two participants. For one participant, the scene is repeated (i.e., unchanged) from the study phase, whereas for the other participant, the scene has been manipulated (i.e., the girls near the bridge have been removed from the scene). Eye movements, superimposed on each scene, illustrate the eye movement based relational memory effect. More fixations (i.e., white cross hairs) are directed to, and more saccades (i.e., red lines) are made into and out of the critical region for manipulated scenes. (C) Graph illustrating the proportion of total viewing time directed to the critical regions of novel, repeated and manipulated scenes for college-age participants, neurologically intact controls, and amnesic patients. College-age participants look disproportionately at manipulated regions with or without concomitant awareness of the change. Neurologically intact controls also show this disproportionate viewing effect, which is completely absent from the viewing behavior of amnesic patients.
Relational memory for arbitrary scene-face pairings has also been revealed in viewers’ eye movement behavior (Hannula et al., 2007; Hannula and Ranganath, 2009). In this work, participants studied several scene-face pairs and were tested with 3-face displays superimposed on previously studied scenes. Participants showed disproportionate viewing of the face that matched (i.e., had previously been paired with) the scene, even among equally familiar faces (i.e., equated in terms of previous viewing history) and in the absence of any reliable spatial cues to guide choices (i.e., the matching face could be in any of three spatial locations, none of which matched the original presentation location; see Figure 4).
Figure 4. IIllustration of the basic paradigm and results from Hannula et al. (2007) and Hannula and Ranganath (2009). (A) Examples of scene-face pairs presented during the study trials, along with a single, associated test display in which the face on the left was the match (i.e., was the associate of the scene). Each test trial began with the presentation of a scene cue meant to prompt retrieval of the associated face. (B) Eye movements from a representative participant superimposed on the test display shown above. Fixations are indicated by white circles and the size of the circle was proportionate to the amount of viewing time directed to the fixated region. Transitions from one fixation to the next are indicated by red lines. (C) Proportion of total viewing time directed to correctly identified matching faces vs. faces that were merely selected from displays that did not contain a match broken down into 250 ms time bins following the onset of the three-face test display. Neurologically intact control participants showed disproportionate viewing of the matching face just 500–750 ms after the faces were presented; no evidence of relational memory was evident in the eye movement behavior of amnesic patients. (D) Bilateral regions of the hippocampus for which BOLD signal was greater for incorrect trials during presentation of the scene cue when subsequent viewing of the match was high vs. when subsequent viewing of the match was low in college-age participants. Trial-averaged time courses extracted from the left and right hippocampal regions, respectively, show differences in BOLD signal between incorrect high and incorrect low viewing trials during presentation of the scene cue. This result illustrates hippocampal recruitment, even when explicit memory has failed.
Finally, memory for temporal relations was shown in eye movement behavior in an experiment in which participants were presented with three objects each shown one at a time in different spatial locations during the study phase and then presented simultaneously after a short delay (Ryan and Villate, 2009). While all three objects were shifted in their absolute position, the relative positions of objects in the display with respect to one another were either intact, or were manipulated by displacing one object with respect to the others. Despite simultaneous presentation during the test phase, participants tended to inspect the objects in the order that matched the originally experienced temporal sequence; this tendency decreased when spatial relationships were manipulated.
Taken together, these results indicate that memory for different aspects of experience (e.g., individual items, spatial and non-spatial relationships, temporal order) guide eye movement behavior. In the sections to follow, we show that the effects of memory on eye movement behavior can occur very rapidly and obligatorily, even in the absence of conscious awareness, which affords us a powerful tool with which to examine memory in a variety of special populations and in conjunction with various other cognitive neuroscience methods.
Revealing Memory Prior to Conscious Awareness
The observation that effects of memory on eye movement behavior develop quickly was initially reported by Parker (1978) who showed that manipulated regions of previously viewed scenes were fixated earlier in scanning than unchanged regions of the same scenes. This rapid acquisition of remembered content with the eyes is accompanied by extended fixation duration (e.g., Ryan and Cohen, 2004a; Ryan et al., 2007) suggesting that eye movements might be used to gather evidence that can inform subsequent behavioral responses. Consistent with this idea, recent work has shown that participants look disproportionately at fragmented target objects embedded in distractor-filled displays as many as 25 fixations before explicit object identification; systematic and nearly exclusive evaluation of the target region was evident four fixations prior to naming (Holm et al., 2008).
Extending the above observations, time-course and response-locked measures (see Box 1) commonly used in event-related potential investigations have recently been adapted to examine when, with respect to stimulus onset and response execution, eye movement based memory effects emerge (Hannula et al., 2007; Ryan et al., 2007; Hannula and Ranganath, 2009). Using these measures, relational memory effects were documented in the face-scene experiment described earlier just 500–750 ms after the onset of the 3-face display (see Figure 4), and as much as 1000 ms prior to explicit behavioral responses. The time-course of this eye movement based relational memory effect was impervious to manipulations of task instructions, emerging within 500–750 ms when participants were explicitly instructed to identify the matching face, and emerging within the exact same time frame even when viewing of the match was counterproductive to the task at hand. That these rapid disproportionate viewing effects were uninfluenced by task demands and develop so far in advance of behavioral responding provides strong evidence for the obligatory nature of memory on eye movement behavior, and suggests that such effects may precede and contribute to conscious recollection of the previously learned association (see Moscovitch, 2008; Hannula and Ranganath, 2009). Collectively, these results indicate that eye movements provide a “temporally precise measure that indexed the evolution of … memory expression from perception to action” (Kumaran and Wagner, 2009, p. 563), and hint at the possibility that eye movements might reflect remembered content even when explicit (conscious) recognition has failed.
Revealing Memory in the Absence of Awareness
A suggestion that eye movements may be decoupled from conscious awareness comes from the attention literature. It has been shown, for example, that task-irrelevant abrupt onsets (e.g., visual stimuli that suddenly appear in a display) capture viewing even when there is volitional effort to avoid them, and even when there is no conscious awareness for the action itself (Kramer et al., 2000; Belopolsky et al., 2008). Converging evidence from several investigations indicates that eye movements can also be influenced by higher-level cognitive processes, like memory, without concomitant awareness for prior learning episodes.
As described above, when the relationships among elements in a scene have changed, eye movements are drawn disproportionately to the manipulated region (e.g., Ryan et al., 2000). Results from several experiments have confirmed that these changes (Ryan and Cohen, 2004a; Beck et al., 2007) and others like them (Hayhoe et al., 1997; Hollingworth et al., 2001, 2008; Hollingworth and Henderson, 2002; Henderson and Hollingworth, 2003) are reflected in modulations of eye movements, even when they go unreported by participants. For example, gaze duration to an object embedded in a scene is significantly longer when it is replaced with a different exemplar, even when participants fail to explicitly detect the change (e.g., Hollingworth et al., 2001). Along similar lines, recent work using the visual paired comparison task, in which a novel object and a studied object are presented simultaneously as eye movements are recorded, has shown that viewing is modified based on the type of associate a learned object was paired with during a study exposure. Novelty preferences in patterns of eye movements were disrupted when learned objects had been paired with positively or negatively valenced scenes, but were intact when learned objects had been paired with a neutral gray image. Differences across these conditions were not evident in explicit recognition responses, suggesting that different memory processes support eye movement based expressions of memory and explicit behavioral responses in this task (Snyder et al., 2008). Collectively, these results suggest that eye movement based memory effects can be expressed even when conscious recollection has failed or is non-diagnostic, which, as we will see in Section “Revealing a Critical Role for the Hippocampus in Memory Without Awareness”, proves important in supporting conclusions from eye movement investigations of the nature of amnesia and the functional role of the hippocampus.
The purpose of these memory-guided eye movements is worth considering, especially since evidence suggests that they may precede or be unaccompanied by conscious awareness. In general, eye movements are used to extract information from the environment; they permit us to continually compare prior experience with current perceptual input in order to detect novelty/change and to guide our subsequent behavior (e.g., make decisions, navigate; Ryan and Cohen, 2004a). Irrespective of whether conscious awareness plays a role, the intersection of current perception with remembered content ensures that perceptual processing proceeds efficiently and rapidly, e.g., by allowing attention to be biased towards particular regions of interest as a function of past experience without the need for supervisory control (Chun and Nakayama, 2000; Maljkovic and Nakayama, 2000).
As time-course analyses have suggested (e.g., Hannula et al., 2007), the rapid influence of memory on eye movements may permit eventual conscious appreciation for items and/or relations that had been previously viewed. That is, conscious awareness may be derived from changes in our eye movements, rather than eye movement based memory effects resulting from conscious awareness for prior learning episodes (Ryan and Villate, 2009). Under most circumstances, eye movements may ultimately be correlated with conscious recognition (e.g., later in processing), but converging evidence from several investigations (see above) indicates that awareness is not a requirement for the expression of memory in eye movement behavior. Precise factors that determine whether eye movements will be correlated with, and give rise to, overt expressions of memory or, alternatively, will be expressed in the absence of explicit awareness, have yet to be identified. However, existing evidence does confirm that eye movements provide a sensitive measure with which to investigate the manner in which multiple memory systems create, access and update mnemonic representations ( Ryan et al., 2000; Brockmole et al., 2002, 2003; Brockmole and Irwin, 2005; Ryan and Villate, 2009).
Revealing a Critical Role for the Hippocampus in Memory Without Awareness
Nowhere are the effects of multiple memory systems more evident than in the study of special populations, especially those involving patients with amnesia following damage to the hippocampus and related medial temporal lobe (MTL) structures. We will develop this example in some detail, as it makes a strong case for the promise of eye movements as part of a converging methods approach. Not only does this example provide data critical for resolving debates about the nature of the memory deficit in hippocampal amnesia and the fundamental role of the hippocampus in memory; it also affords an important illustration of the power of eye movement data when combined with other cognitive neuroscience methods.
The nature of the impairment in amnesia and the functional role of the hippocampal system in memory have been the subject of intense study ever since the report of profound and pervasive memory impairment following bilateral removal of the hippocampus in patient H.M. (Scoville and Milner, 1957). A major advance came from findings of preserved learning abilities in amnesia – while some aspects of learning and memory were profoundly impaired, others were left fully intact (e.g., Milner, 1962; Milner et al., 1968; Cohen and Squire, 1980; Graf and Schacter, 1985). Such findings, and many that followed, led to the emergence of theories about multiple memory systems of the brain (Cohen, 1984; Tulving, 1985; Schacter, 1987; Squire, 1992; Cohen and Eichenbaum, 1993; Gabrieli, 1998; Eichenbaum and Cohen, 2001).
One multiple memory systems view that received considerable support distinguished explicit from implicit memory, emphasizing the role of the hippocampus in conscious recollection or conscious awareness of prior experiences (Graf and Schacter, 1985; Schacter, 1987; Squire, 1992). According to this view conscious awareness for prior learning episodes is a fundamental property of the memory representations that are formed by the hippocampus, with hippocampal representations necessarily available to conscious introspection (c.f., Manns and Squire, 2001; Smith and Squire, 2008). Preserved vs. impaired performances in hippocampal amnesia would therefore revolve around the necessity of conscious access to memories for successful performance.
An alternative view (Cohen and Eichenbaum, 1993; Eichenbaum and Cohen, 2001) has emphasized the role of the hippocampus in relational memory binding (i.e., the formation of representations that consist of relationships among constituent elements of scenes or events). On this view, although the ability for conscious recollection of prior episodes requires relational memory for the items that comprise those prior episodes and/or the contexts in which they were encountered, preserved vs. impaired performance in hippocampal amnesia instead revolves around the demand for relational memory (i.e., leaving memory for individual items intact). By this account, relational representations may be expressed without concomitant conscious awareness.
The challenge in adjudicating between these theories is to vary relational memory demands independently of conscious access demands. That is, because tests involving conscious recollection as the measure of performance also involve relational memory (Ryan and Cohen, 2003), it was necessary to step outside of standard recognition memory testing paradigms. Based on our findings that eye movements provide a means for indirect testing and expression of both item memory and relational memory, we applied eye movement studies to answer this fundamental question about memory and amnesia. Findings revealed that patients with hippocampal amnesia had selective relational memory impairments, sparing item memory, independent of issues of conscious access.
Despite amnesia, patients exhibited normal eye movement based memory effects for individual items, showing decreased sampling in the viewing of repeated, as compared to novel, faces (Althoff et al., 1993; Althoff, 1998) and scenes (Ryan et al., 2000). In the case of faces, one patient with severe amnesia presented with famous and non-famous faces showed intact repetition effects despite impaired recognition of the viewed faces (Althoff et al., 1993). Similarly, seven amnesic patients tested in a different experiment with sets of non-famous faces viewed either 0, 1, 3, or 5 times across a study session showed significant changes in viewing (on measures of constraint and the number of regions sampled) with increased repetitions. These changes in viewing were evident despite recognition rates that remained at chance and did not improve with repeated exposure to the faces (Althoff, 1998).
By contrast, eye movement measures of relational memory were found to be impaired in hippocampal amnesia (Ryan et al., 2000; Hannula et al., 2007). In the Ryan et al. (2000) study of novel, repeated, and relationally manipulated scenes, neurologically intact controls looked disproportionately at regions of scenes that had undergone a change in spatial relationships, even when they were unaware of the nature of the changes that had occurred; such eye movement behavior was absent in amnesic patients (see Figure 4). However, as noted above, the same patients showed normal repetition effects for the scenes, revealing normal memory for items. In the Hannula et al. (2007) study of arbitrary face-scene pairings, neurologically intact control subjects presented with test displays composed of three equally familiar faces superimposed on a familiar scene showed early preferential viewing of the one face that had actually been paired with the scene during the study phase; amnesic patients failed to show the normal effect of relational memory in their eye movement behavior. In a follow-up investigation that combined eye movement monitoring with fMRI (Hannula and Ranganath, 2009), hippocampal activity during the scene cue was shown to predict preferential viewing of the matching face, even when participants failed to explicitly identify the associated face (see Figure 4).
Taken together, the use of eye movement paradigms for indirect testing of item and relational memory provided the field with critical evidence in favor of a relational memory account over an explicit memory account with respect to the impairment in hippocampal amnesia, and the functional role of hippocampus in normal memory. Memory for single items was intact while memory for relations among the items was impaired, despite using methods by which both types of memory could be observed without requiring conscious recollection. This example also illustrates the great promise of converging cognitive neuroscience methods that include eye movement measures. Using the same paradigm with combined eye-tracking and fMRI methods, Hannula and Ranganath (2009) linked the deficit seen in hippocampal amnesic patients to hippocampal activity in normal individuals, and linked hippocampal activity to the eye movement based relational memory effect independent of explicit remembering.
Some of these findings, and the corresponding conclusions that were drawn with regard to the hippocampus and memory, have been challenged. Based on their failure to replicate some of the results reported by Ryan et al. (2000), Smith and Squire (2008) concluded that eye movement based memory effects were only expressed with accompanying conscious awareness for prior learning episodes, and that both item and relational memory effects were critically dependent on hippocampal integrity.
Consistent with previous work (e.g., Althoff and Cohen, 1999; Ryan et al., 2000), Smith and Squire (2008) reported that neurologically intact participants sampled fewer regions and made fewer fixations to repeated (vs. novel) images. However, these effects of repetition on eye movement behavior were not statistically reliable when amnesic patients were tested, a finding that contradicts past work (Althoff et al., 1993; Althoff, 1998; Ryan et al., 2000). There are important methodological differences between the experiments, including issues concerning design and measurement that bear importantly on the ability to elicit and detect the eye movement effects of interest. Such concerns emphasize how critical it is that careful consideration be given to the manner in which eye movement measures are defined and calculated (see Box 1).
Methodological issues aside, claims that eye movement based memory effects require conscious recollection (Smith and Squire, 2008) are at odds not only with studies from our group (Althoff et al., 1993; Althoff, 1998; Ryan et al., 2000), but also with several other eye movement investigations including those by Hayhoe et al. (1997), Hollingworth and colleagues (Hollingworth et al., 2001, 2008; Hollingworth and Henderson, 2002; Henderson and Hollingworth, 2003), and Beck et al. (2007; see Revealing Memory in the Absence of Awareness). When these results are considered together with more recent work (e.g., Hannula and Ranganath, 2009), the findings provide a powerful demonstration that the crucial mnemonic contribution of the hippocampus is relational and distinct from any potential role in conscious awareness. To date, there have been no demonstrations of intact eye movement based relational memory effects in the face of hippocampal damage. Moreover, our conclusion about the central role of relational memory in any account of hippocampal function receives overwhelming support from other lines of cognitive neuroscience research (for review see Davachi, 2006; Konkel and Cohen, 2009).
Revealing Spared and Impaired Memory in Other Special Populations
Cognitive tasks developed for use with college-age participants may not always lend themselves easily to translation for work with special populations. Traditional behavioral measures can be confounded by issues of task comprehension, complex decision making requirements, and behavioral response mapping difficulties in these individuals (e.g., Luck and Gold, 2008). Accordingly, sensitive, indirect methods, especially those that have the potential to be translated for use with animals or for use with neuroimaging methods, aid in the development of new investigations of cognition with special populations. A notable strength of eye movement methods is that they can be used to assess memory with or without concomitant collection of behavioral responses. As such, eye movement methods are particularly useful as a memory assessment tool for animals and infants (see Boxes 3 and 4, and Figure 5), and for investigations of the memory impairments associated with normal aging and psychiatric disorders (e.g., schizophrenia). Better characterization of memory performance may ultimately permit remediation when performance is impaired (see Box 5), and may contribute to identification of neural substrates that can be targeted for pharmaceutical intervention (see Carter and Barch, 2007).
Figure 5 Illustration of early methods used to examine eye movement behavior in infants. This infant “looking chamber“ was used by Fantz (1963) to examine the length of gaze directed to visual targets. Fixation duration was recorded by the experimenter who monitored the infant’s gaze through a small hole in the ceiling of the chamber. Current methods are akin to those used with adult participants (cf. Richmond and Nelson, 2009)
Box 3. Eye movement monitoring of non-human primates.
Non-human primates provide a means to pose scientific questions that cannot be addressed ethically in human participants, and both the lesion method and single-cell recording have been fruitfully combined with eye movement methods in this population. In some cases eye movement data have settled longstanding questions within the memory literature, and a notable example is the performance on certain memory tasks by animals assumed to model anterograde amnesia.
The delayed-non-match-to-sample (DMNS) task was a mainstay of animal research into memory systems for decades, but the results from lesion studies of the task were difficult to interpret. For example, lesions of the hippocampus did not impair DNMS performance, while lesions of the perirhinal cortex did, even at fairly short delays (see Suzuki and Amaral, 2004). This suggested that the perirhinal cortex must support knowledge about single objects across delays, but that functionality could reasonably have been assigned to late visual areas. By combining lesions with eye-tracking, Buffalo et al. (1999) demonstrated a dissociation between the contributions of late visual areas and perirhinal cortex in a preferential viewing task. Lesions of late visual areas produced at-chance viewing, while lesions of perirhinal cortex produced impaired-but-above-chance performance at most delays. This showed that the DNMS and visual preferential viewing tasks were differentially sensitive to MTL damage.
The study of non-human primates also allows for observation of the normal operation of MTL systems using single-unit recording in conjunction with eye movement monitoring. Jutras and Buffalo (2010) recently investigated the firing of hippocampal neurons in just this fashion. Using a set of thousands of images novel to the monkeys, they employed the visual paired comparison task (also termed the visual preferential looking task) and observed single units in hippocampus that were reliably modulated by the novelty or familiarity of a stimulus. Further, firing rates of hippocampal neurons sensitive to novelty were reliably correlated with an eye movement index of memory strength, indicating that the hippocampus plays a direct and crucial role in recognition memory.
The finding that hippocampal cell firing rates correspond to eye movement based measures of memory prompts another question, i.e., whether saccadic eye movements directly affect brain systems crucial for memory. Given that the visual experience of many organisms is frequently and unconsciously interrupted by abrupt changes in our point of gaze, it is reasonable to hypothesize that the memory systems that evolved alongside these visual systems might make use of saccades as boundaries between memoranda. Single-unit recording studies indicate that the activity of MTL structures is influenced by eye movements in intriguing ways: transmission of electrical impulses within the MTL is uniquely enhanced in the 100 ms following a saccade, and saccades elicit activity in those same structures (Ringo et al., 1994; Sobotka et al., 1997, 2002). Critically, these effects are extraretinal, all having been observed under both light and dark conditions. These findings indicate that individual eye movements are constantly modulating the activity of the same MTL structures known to be necessary for normal declarative (relational) memory.
Box 4. Eye movement monitoring of infants.
Characterizing the development of human memory systems is necessary for a thorough understanding of how memory operates in adults. Representing the beginning of the development spectrum, infants are a challenging population to study owing to their lack of language, short attention span, and poor motor control. Eye movements are one index of infant behavior that can be compared to adults, and with an appropriate testing apparatus, short tasks, and many participants, eye movement data can be gathered from even very young infants. Importantly, early investigations established that infants can discriminate relatively complex visual stimuli, paving the way for sophisticated investigations of infant memory that followed (see Figure 5). Fantz (1964) repeatedly exposed the same stimulus to infants between 1 and 6 months old, observing that the stimulus was fixated less and less frequently with increasing repetition. Extending from this, Fagan (1970) simultaneously presented two stimuli with different viewing histories, one novel and one familiar, and found that infants as young as 2–6 months showed preferential viewing for novel items. Further evidence for memory in infants emerged fortuitously; Fagan re-used stimuli on three consecutive days and on the second and third days of testing, there was no preferential viewing of stimuli seen the day before. This demonstrated that even before 6 months of age infants could form memories of visual stimuli that persisted for days. Preferential viewing indicates that humans have at least basic mnemonic functions soon after birth. However, researchers have often argued that there are aspects of adult memory that may not be present early in life. For instance, findings from behavioral studies have suggested that the flexible, relational, and hippocampally dependent memories that typify declarative memory may not be available to young children, perhaps due to a medial temporal lobe system that has not been fully developed (e.g., Sluzenski et al., 2006). However, a fascinating replication of an experiment first used to investigate relational memory in healthy and amnesic adults calls these ideas into question. Hannula et al. (2007) used a passive viewing paradigm to test for knowledge of previously established face-scene relations expressed in eye movements (see Figure 4). Richmond and Nelson (2009) adapted the same experiment to the testing of infants, and observed patterns of viewing that demonstrated memory for face-scene pairings in infants with the same time-course reported in work with adults. This startling result indicates that relational memories are being formed as early as 9 months of age, a major departure from existing hypotheses.
Box 5. Eye movements as a diagnostic tool and key to remediation following brain damage or dysfunction.
Recognition of facial identity and facial expression of emotion are fundamental to human interaction, but these capacities can be disrupted by damage to or dysfunction of specific brain regions. Measurement of eye movements can both enhance understanding of these disruptions and suggest strategies for remediation and rehabilitation.
Prosopagnosia refers to a severe deficit in recognizing familiar faces either subsequent to brain damage (acquired prosopagnosia, AP) or present since birth (congenital prosopagnosia, CP). Eye movement monitoring has revealed that individuals with prosopagnosia, in particular CP, show abnormal scanning patterns of faces: fewer fixations to central features like the eyes, nose, and mouth; enhanced viewing of the mouth over the eyes; and more fixations to peripheral features including hair and hairline (Le et al., 2003; Schwarzer et al., 2007; de Xivry et al., 2008; Schmalzl et al., 2008; Stephan and Caine, 2009; but see Bate et al., 2008). If abnormal scanning patterns underlie CP, then it stands to reason that rehabilitative techniques that target eye movements may be beneficial for patients with prosopagnosia. In independent efforts (De Gutis et al. 2007; Schmalzl et al., 2008), two patients with well-characterized CP received specific instructions regarding attention to internal features of faces. Both patients benefitted from this remedial technique, and were better able to recognize faces afterward. Further, one of these investigations (De Gutis et al., 2007) used neuroimaging techniques to determine whether this new ability was reflected in neural activity. In that patient, an N170 ERP response to faces was evident only after training, and increased functional connectivity was observed between face-selective regions of the brain. These neural changes presumably mirrored and supported the new abilities that the patient developed following conscious changes to their eye movement behavior.
Problems with recognition of facial expression of emotion can also benefit from the study of eye movements. Research conducted with an individual who has nearly complete bilateral amygdala damage due to Urbach–Weithe disease has provided new insight into the mechanism that underlies her inability to identify/recognize fearful faces (Adolphs et al., 2005). In contrast to intact recognition of other facial expressions, S.M. showed severely impaired recognition of faces that bear fearful expressions (Adolphs et al. 1994). Perhaps surprisingly, S.M. could recognize fearful prosody (Adolphs and Tranel, 1999) and could describe fearful situations (Adolphs et al. 1995). Eye movement monitoring has been used to examine this patient’s visual recognition problem in detail; in contrast to controls, S.M. spent very little time looking spontaneously at the eye region of faces under free-viewing conditions (Adolphs et al., 2005). Having identified this abnormal viewing pattern, investigators instructed S.M. specifically to look at the eyes of the faces she was shown (i.e., instructed viewing). Under these circumstances her ability to identify fearful facial expressions improved and equaled that of healthy comparisons (see associated Figure).
Jointly, these promising results from eye movement monitoring of two patient populations suggest that remediation of deficient recognition can sometimes be accomplished with a simple, conscious behavioral modification.
The promise of indirect eye movement methods for providing insight into the nature of reported memory deficits associated with normal aging (see Light, 1996 for a review) has been realized in recent work (Heisz and Ryan, submittedFirestone et al., 2007; Ryan et al., 2007; ). Older individuals expect to perform poorly if they know their memory is being tested, thereby unwittingly undermining their performance (e.g., Rahhal et al., 2001; Chasteen et al., 2005). As eye movement methods do not require participants to explicitly comment on the contents of their memory (i.e., in free-viewing paradigms), they can be used to eliminate confounding effects of anticipated failure. Here, we highlight one such investigation (Ryan et al., 2007), which shows the power of eye movements to test the predictions of more than one theoretical account simultaneously, very much like the approach taken in investigations of amnesia.
Several competing theories have been proposed to explain memory decline associated with normal aging (see Light, 1996; Balota et al., 2000 for reviews). Two influential theories attribute age-related memory deficits to impaired inhibition of irrelevant information (e.g., Zacks et al., 2000) and impaired binding of relational representations (Naveh-Benjamin, 2000). In an eye movement experiment designed to test predictions consistent with these theoretical perspectives, we used an experimental manipulation in which deficits in inhibition should benefit relational memory binding: objects that were to be ignored within scenes ultimately underwent a change in their spatial relations (Ryan et al., 2007). Older adults displayed a concurrent deficit in both inhibition and binding. Relative to younger adults, they were more likely to fixate abrupt onsets (i.e., the objects that were subsequently manipulated). Despite this lack of inhibition, and in contrast to younger adults, older adults did not look disproportionately at manipulated objects. Additional work from our group has shown that age-related deficits in binding can be ameliorated when older adults can appeal to information that is already in memory (Heisz and Ryan, submittedFirestone et al., 2007; ). Together, these findings suggest an age-related deficit in relational binding, akin to amnesic patients, which is consistent with the finding that aging is associated with disproportionate atrophy in the hippocampus (e.g., Driscoll et al., 2003; Erickson et al., 2010).
Recent work suggests that eye movements may also have some diagnostic value in the early identification of mild cognitive impairment (MCI; Crutcher et al., 2009) and may provide some insight into the nature of the deficit in Alzheimer’s disease (Daffner et al., 1992, 1999). For instance, Crutcher et al. (2009) have shown that, unlike matched controls, patients with MCI fail to look preferentially at a novel picture over one presented 2 min earlier in the context of the visual paired comparison task. Instead, patients distributed viewing almost equally between the two pictures. Because novelty preferences were intact when the delay between the familiarization phase and the test phase was shorter (i.e., 2 s), between-group differences in patterns of viewing were unlikely to have been a consequence of perceptual, motivational, or attentional deficits. The implication is that eye movement based memory assessment may be of use in the identification and characterization of dementia (Crutcher et al., 2009), but because this research is just getting underway, future investigations are needed to confirm the potential utility of eye movement methods in this arena.
Studies conducted with psychiatric populations (e.g., schizophrenia patients) may also benefit from the use of indirect eye movement methods. Schizophrenia is a psychiatric disease that affects performance on several cognitive and information-processing tasks, but disproportionate impairments have been observed on tests of episodic memory (e.g., McKenna et al., 1990; Saykin et al., 1991), and memory is a strong predictor of functional outcome in these individuals (Green et al., 2000). Recent results suggest that patients with schizophrenia may have relatively spared memory for items and disproportionately impaired memory for item-context relationships, although others maintain that the deficit is generalized (see Ranganath et al., 2008). Eye movement based memory experiments, like those described in the preceding sections, may help resolve this debate. In contrast to behavioral measures, eye movement data can be acquired after providing participants with relatively simple instructions, and need not invoke behavioral responses with complex decision making requirements; therefore, this technique may be particularly suitable for investigating the nature of the memory impairment in schizophrenia.
Recently, results from two independent investigations have shown that relational memory effects are considerably reduced in the eye movement behavior of schizophrenic patients (Hannula et al., 2010; Williams et al., 2010), while other measures of viewing related to specific task demands (e.g., viewing a face that will be selected) are similar to those of controls. Results like these are tantalizing, and help to validate use of eye movement monitoring as a tool for studying memory in these individuals. They also suggest that eye movement measures may be useful in future attempts to relate impaired cognitive performance to underlying neural mechanisms.
Integration of Eye Movement Monitoring with Neuroimaging
The preceding sections have provided examples of the ability of eye movement methods to address questions about memory that were not necessarily possible with more traditional behavioral methods. However, such methods are particularly powerful for advancing the cognitive neuroscience of memory when combined with other methods and approaches. Eye movement data acquired in a scanning environment is clearly useful for observing participants’ compliance if the task requires fixation on a central location, but more nuanced data can also be used to categorize trials according to some variable of interest (e.g., viewing time directed to particular regions of a display). The use of eye movement monitoring in conjunction with neuroimaging techniques can begin to dissociate different stages of stimulus processing (e.g., early obligatory effects of memory versus later explicit retrieval), and to determine whether these stages are supported by different underlying neural networks.
A compelling example of the integration of fMRI with eye movement methods was discussed briefly above (Hannula and Ranganath, 2009; see Revealing a Critical Role for the Hippocampus in Memory Without Awareness). Adapting our previous paradigm in which eye movement effects of relational memory were observed in healthy participants but not in patients with hippocampal damage (Hannula et al., 2007; see Figure 4), the use of converging methods revealed that hippocampal activity during presentation of the scene cue predicted viewing of the matching face, even when participants failed to identify that face correctly via behavioral response. Further, when participants made a correct behavioral response, activity in lateral prefrontal areas, and functional connectivity between these areas and the hippocampus, was increased, suggesting that while the hippocampus may be the critical area for retrieval of the relational information, additional recruitment of extrahippocampal brain regions may be required for its explicit expression (Hannula and Ranganath, 2009).
The concurrent collection of eye movements and magnetoencephalography (MEG) is also expected to provide insight into mnemonic processes. A non-invasive neuroimaging technique that estimates neuronal activity based on recordings of the magnetic flux outside of the head (Hämäläinen et al., 1993; Hari et al., 2000), MEG allows recording of neural activity with temporal resolution on the order of milliseconds and spatial resolution comparable to that of fMRI (Miller et al., 2007), making it an ideal tool for studying the dynamics of brain function.
Exploration of memory processes with combined MEG and eye movement techniques has just begun, but work has already shown that MEG can localize signals from the hippocampus (Riggs et al., 2009; see also Breier et al., 1998, 1999; 2000; Tesche and Karhu, 2000; Hanlon et al., 2003, 2005; Gonsalves et al., 2005; Moses et al., 2009), and that hippocampal responses are evident as early as 120–130 ms after stimulus onset during a recognition task (Riggs et al., 2009). Critically, recent work has also shown that MEG can be successfully combined with eye movement monitoring (Herdman and Ryan, 2007; see also Hirvenkari et al., 2010). In this work, MEG data were time-locked to eye movements, permitting examination of neural activity immediately preceding and following saccades. The next step in the combined use of eye movement monitoring and MEG is to reveal the neural networks that drive eye movement based memory effects.
Concluding Statements
In this review, we have attempted to illustrate the utility and promise of eye movement methods for advancing cognitive neuroscience investigations, with a focus on investigations of memory. Eye movements have been shown to reveal the influence of different types of memory (e.g., item memory and relational memory) using a variety of materials (e.g., faces, building, scenes, arbitrary pairings and sequences), permitting us to address questions about distinct memory systems. The influence of memory on eye movements can be observed obligatorily, soon after stimulus onset, and may occur long before, or in the absence of, conscious awareness for remembered content. Given that eye movements can be acquired without explicit reports or other overt responses, this method is an ideal tool for indexing memory function in special populations (i.e., non-human primates, infants, and patients with neuropsychological, psychiatric or neurodegenerative conditions). We have exploited this technique to observe changes in memory function that are associated with healthy aging and amnesia, and in so doing have reinforced the critical role of the hippocampal system in memory. Beyond memory, findings from eye movement investigations have influenced, or promise to influence, our understanding of disordered cognition exhibited by patients who have prosopagnosia, schizophrenia, and dementia. Indeed, when eye movement methods are combined with patient populations, there is even promising translational potential. This is exemplified by work conducted by Adolphs et al. (2005; see Box 5), in which observation of abnormal viewing patterns prompted remedial strategies that restored recognition of fearful faces in the laboratory.
Studies that combine eye movements with neuroimaging techniques have the potential to provide unparalleled insights into the brain networks that support various memory abilities and answer questions about the role of conscious awareness in the use of memory. Eye movement methods also hold great promise for relating the activity of particular brain regions and systems to the time at which the various influences of memory emerge. Given that eye movement findings reveal the early and obligatory influences of memory in online processing (Hannula et al., 2007; Ryan et al., 2007; Warren et al., 2010), such findings challenge our traditional notions of “perception” and “memory” and suggest that the very nature of perceptual processing is altered as a result of our prior experiences (Hannula et al., 2007; Ryan et al., 2007, 2008). Furthermore, eye movement methods provide a powerful tool for revealing the influences of memory across cognitive domains (e.g., in language: Rubin et al., 2009), and may reveal the influence of other cognitive processes on memory itself (e.g., emotional valence: Riggs et al., in press).
Cognitive neuroscience as a discipline has benefited greatly from the converging methods approach, using multiple methods to provide comprehensive answers to difficult questions. This review was intended to illustrate the promise of eye movement monitoring as one of the methods that should be considered by cognitive neuroscientists, and to demonstrate the advances that have been made in the cognitive neuroscience of memory as a result of combining eye movement methods with neuropsychological and neuroimaging approaches.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
We wish to thank Douglas McQuiggan for his assistance with the manuscript,Ralph Adolphs for providing us with a high resolution version of the Figure in Box 5, and Tanya Kirouac for improving the quality of the drawing in Figure 1. Funding from NIMH grant MH062500 was provided to Neal J. Cohen. Funding from the Natural Sciences and Engineering Research Council of Canada (NSERC), Canadian Institutes of Health Research (CIHR), Canada Research Chairs Program and Canadian Foundation for Innovation (CRC/CFI) was provided to Jennifer D. Ryan.
Footnotes
- ^also measures of directed viewing
- ^please refer to Considerations for Analysis (this box) for more information about calculating this measure
- ^see Box 2 and Althoff and Cohen (1999)
References
Adolphs, R., Gosselin, F., Buchanan, T. W., Tranel, D., Schyns, P., and Damasio, A. R. (2005). A mechanism for impaired fear recognition after amygdala damage. Nature 433, 68–72 (Figure reprinted by permission from Macmillan Publishers Ltd: Nature).
Adolphs, R., and Tranel, D. (1999). Intact recognition of emotional prosody following amygdale damage. Neuropsychologia 37, 1285–1292.
Adolphs, R., Tranel, D., Damasio, H., and Damasio, A. (1994). Impaired recognition of emotion in facial expressions following bilateral damage to the human amygdala. Nature 372, 669–672.
Adolphs, R., Trauel, D., Damasio, H., and Damasio, A. (1995). Fear and the human amygdala. J. Neurosci. 15, 5879–5892.
Althoff, R. R. (1998). Eye-Movement-Based Memory Assessment: The Use of Eye Movement Monitoring as an Indirect Measure of Memory. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign, Urbana.
Althoff, R. R., and Cohen, N. J. (1999). Eye-movement based memory effect: a reprocessing effect in face perception. J. Exp. Psychol. 25, 997–1010.
Althoff, R. R., Cohen, N. J., McConkie, G., Wasserman, S., Maciukenas, M., Azen, R., and Romine, L. (1998). “Eye-movement based memory assessment,” in Current oculomotor research: Physiological and Psychological Aspects, eds W. Becker, H. Deubel, and T. Mergner (New York: Kluwer Academic/Plenum Press), 293–302.
Althoff, R. R., Maciukenas, M., and Cohen, N. J. (1993). Indirect assessment of memory using eye movement monitoring. Abstr. - Soc. Neurosci. 19, 439.
Balota, D. A., Dolan, P. O., and Duchek, J. (2000). “Memory changes in healthy older adults,” in The Oxford Handbook of Memory, eds E. Tulving and F. I. M. Craik (New York: Oxford University Press), 395–409.
Bate, S., Haslam, C., Tree, J. J., and Hodgson, T. L. (2008). Evidence of an eye-movement based memory effect in congenital prosopagnosia. Cortex 44, 806–819.
Beck, M. R., Peterson, M. S., and Angelone, B. L. (2007). The roles of encoding, retrieval, and awareness in change detection. Mem. Cognit. 35, 610–620.
Belopolsky, A. V., Kramer, A. F., and Theeuwes, J. (2008). The role of awareness in processing of oculomotor capture: evidence from event-related potentials. J. Cogn. Neurosci. 20, 2285–2297.
Breier, J. I., Simos, P. G., Zouridakis, G., and Papanicolaou, A. C. (1998). Relative timing of neuronal activity in distinct temporal lobe areas during a recognition memory task for words. J. Clin. Exp. Neuropsychol. 20, 782–790.
Breier, J. I., Simos, P. G., Zouridakis, G., and Papanicolaou, A. C. (1999). Lateralization of cerebral activation in auditory verbal and non-verbal memory tasks using magnetoencephalography. Brain Topogr. 12, 89–97.
Breier, J. I., Simos, P. G., Zouridakis, G., and Papanicolaou, A. C. (2000). Lateralization of activity associated with language function using magnetoencephalography. J. Clin. Neurophysiol. 17, 503–510.
Brockmole, J. R., Castelhano, M. S., and Henderson, J. M. (2006). Contextual cueing in naturalistic scenes: global and local contexts. J. Exp. Psychol. Learn. Mem. Cogn. 32, 699–706.
Brockmole, J. R., and Henderson, J. M. (2008). Prioritizing new objects for eye fixation in real-world scenes: effects of object-scene consistency. Vis. Cogn. 16, 375–390.
Brockmole, J. R., and Irwin, D. E. (2005). Eye movements and the integration of visual memory and visual perception. Percept. Psychophys. 67, 495–512.
Brockmole, J. R., Irwin, D. E., and Wang, R. F. (2003). The locus of spatial attention during the temporal integration of visual memories and visual percepts. Psychon. Bull. Rev. 10, 510–515.
Brockmole, J. R., Wang, R. F., and Irwin, D. E. (2002). Temporal integration between visual images and visual percepts. J. Exp. Psychol. Hum. Percept. Perform. 28, 315–334.
Buffalo, E. A., Ramus, S. J., Clark, R. E., Teng, E., Squire, L. R., and Zola, S. M. (1999). Dissociation between the effects of damage to perirhinal cortex and area TE. Learn. Mem. 6, 572–599.
Buswell, G. T. (1935). How People Look at Pictures: A study of the Psychology of Perception in Art. Chicago, IL: University of Chicago Press.
Carpenter, P. A., and Just, M. A. (1977). “Reading comprehension as eyes see it,” in Cognitive processes in comprehension, eds M. A. Just and P. A. Carpenter (Hillsdale, NJ: Erlbaum), 109–139.
Carter, C. S., and Barch, D. M. (2007). Cognitive neuroscience-based approaches to measuring and improving treatment effects on cognition in schizophrenia: the CNTRICS initiative. Schizophr. Bull. 33, 1131–1137.
Castelhano, M. S., and Henderson, J. M. (2007). Initial scene representations facilitate eye movement guidance in visual search. J. Exp. Psychol. Hum. Percept. Perform. 33, 753–763.
Chasteen, A. L., Bhattacharyya, S., Horhota, M., Tam, R., and Hasher, L. (2005). How feelings of stereotype threat influence older adults’ memory performance. Exp. Aging Res. 31, 235–260.
Chun, M. M., and Nakayama, K. (2000). On the functional role of implicit visual memory for the adaptive deployment of attention across scenes. Vis Cogn. Special Issue: Change Blindness and Visual Memory, 7, 65–81.
Cohen, N. J. (1984). “Preserved learning capacity in amnesia: evidence for multiple memory systems,” in Neuropsychology of Memory, eds L. R. Squire and N. Butters (New York: Guilford), 83–103.
Cohen, N. J., and Eichenbaum, H. (1993). Memory, Amnesia, and the Hippocampal System. Cambridge, MA: The MIT Press.
Cohen, N. J., and Squire, L. R. (1980). Preserved learning and retention of pattern-analyzing skill in amnesia: Dissociation of knowing how and knowing that. Science 210, 207–210.
Crutcher, M. D., Calhoun-Haney, R., Manzanares, C. M., Lah, J. J., Levey, A. I., and Zola, S. M. (2009). Eye tracking during a visual paired comparison task as a predictor of early dementia. Am. J. Alzheimers Dis. Other Demen. 24, 258–266.
Daffner, K. R., Mesulam., M. M., Cohen, L. G., and Scinto, L. F. M. (1999). Mechanisms underlying diminished novelty-seeking behavior in patients with probably Alzheimer’s disease. Neuropsychiatry Neuropsychol. Behav. Neurol. 12, 58–66.
Daffner, K. R., Scinto, L. F. M., Weintraub, S., Guinessey, J., and Mesulam, M. M. (1992). diminished curiosity in patients with probable Alzheimer’s disease as measured by exploratory eye movements. Neurology 42, 320–328.
Davachi, L. (2006). Item, context and relational episodic encoding in humans. Curr. Opin. Neurobiol. 16, 693–700.
De Gutis, J. M., Bentin, S., Robertson, L. C., and D’Esposito, M. (2007). Functional plasticity in ventral temporal cortex following cognitive rehabilitation of a congenital prosopagnosic. J. Cogn. Neurosci. 19, 1790–1802.
de Xivry, J. O., Ramon, M., Lefèvre, P., and Rossion, B. (2008). Reduced fixation on the upper area of personally familiar faces following acquired prosopagnosia. J. Neuropsychol. 2, 245–268.
Didday, R. L., and Arbib, M. A. (1975). Eye movements and visual perception: A “two visual system” model. Int. J. Man. Mach. Stud. 7, 547–569.
Driscoll, I., Hamilton, D. A., Petropoulos, H., Yeo, R. A., Brooks, W. M., Baumgartner, R. N., and Sutherland, R. J. (2003). The aging hippocampus: Cognitive, biochemical and structural findings. Cereb. Cortex 13, 1344–1351.
Eichenbaum, H., and Cohen, N. J. (2001). From Conditioning to Conscious Recollection. New York: Oxford University Press.
Ellis, S. R., and Stark, L. (1986). Statistical dependency in visual scanning. Hum. Factors 28, 421–438.
Erickson, K. I., Prakash, R. S., Voss, M. W., Chaddock, L., Heo, S., McLaren, M., Pence, B. D., Martin, S. A., Vieira, V. J., McAuley, E., and Kramer, A. F. (2010). Brain-derived neurotrophic factor is associated with age-related decline in hippocampal volume. J. Neurosci. 30, 5368–5375.
Fantz, R. L. (1963). Pattern vision in newborn infants. Science 140, 296–297 (Figure reprinted with permission from AAAS).
Fantz, R. L. (1964). Visual experience in infants: Decreased attention to familiar patterns relative to novel ones. Science 146, 668–670.
Firestone, A., Turk-Browne, N., and Ryan, J. D. (2007). Age-related deficits in face recognition are related to underlying changes in scanning behavior. Neuropsychol. Dev. Cogn. B Aging Neuropsychol. Cogn. 14, 594–607.
Fisher, D. F., Monty, R. A., and Perlmuter, L. C. (1978). Visual recognition memory for binary pictures: another look. J. Exp. Psychol. Hum. Learn. 4, 158–164.
Graf, P., and Schacter, D. L. (1985). Implicit and explicit memory for new associations in normal and amnesic patients. J. Exp. Psychol. Hum. Learn. 11, 501–518.
Gonsalves, B. D., Kahn, I., Curran, T., Norman, K. A., and Wagner, A. D. (2005). Memory strength and repetition suppression: multimodal imaging of medial temporal cortical contributions to recognition. Neuron 47, 751–761.
Green, M. F., Kern, R. S., Braff, D. L., and Mintz, J. (2000). Neurocognitive deficits and functional outcome in schizophrenia: Are we measuring the “right stuff”? Schizophr. Bull. 26, 119–136.
Hämäläinen, M., Hari, R., Ilmoniemi, R. J., Knuutila, J., and Lounasmaa, O. V. (1993). Magnetoencephalography – theory, instrumentation, and applications to noninvasive studies of the working human brain. Rev. Modern Phys. 65, 413–496.
Hanlon, F. M., Weisend, M. P., Huang, M., Lee, R. R., Moses, S. N., Paulson, K. M., Thoma, R. J., Miller, G. A., and Cañive, J. M. (2003). A non-invasive method for observing hippocampal function. Neuroreport 14, 1957–1960.
Hanlon, F. M., Weisend, M. P., Yeo, R. A., Huang, M., Lee, R. R., Thoma, R. J., Moses, S. N., Paulson, K. M., Miller, G. A., and Cañive, J. M. (2005). A specific test of hippocampal deficit in schizophrenia. Behav. Neurosci. 119, 863–875.
Hannula, D. E., and Ranganath, C. (2009). The eyes have it: hippocampal activity predicts expression of memory in eye movements. Neuron 63, 592–599.
Hannula, D. E., Ranganath, C., Ramsay, I. S., Solomon, M., Yoon, J., Niendam, T. A., Carter, C. S., and Ragland, J. D. (2010). Use of eye movement monitoring to examine item and relational memory in schizophrenia. Biol. Psychiatry. doi:10.1016/j. biopsych.2010.06-001. [Epub ahead of print].
Hannula, D. E., Ryan, J. D., Tranel, D., and Cohen, N. J. (2007). Rapid onset relational memory effects are evident in eye movement behavior, but not in hippocampal amnesia. J. Cogn. Neurosci. 19, 1690–1705.
Hari, R., Levänen, S., and Raij, T. (2000). Timing of human cortical functions during cognition: role of MEG. Trends Cogn. Sci. 4, 455–462.
Hayhoe, M. M., Bensinger, D. G., and Ballard, D. H. (1997). Task constraints in visual working memory. Vision Res. 38, 125–137.
Henderson, J. M. (2003). Human gaze control during real-world scene perception. Trends Cogn. Sci. 7, 498–504.
Henderson, J. M., and Hollingworth, A. (2003). Eye movements and visual memory: detecting changes to saccade targets in scenes. Percept. Psychophys. 65, 58–71.
Henderson, J. M., Weeks, P. A., and Hollingworth, A. (1999). The effects of semantic consistency on eye movements during complex scene viewing. J. Exp. Psychol. Hum. Percept. Perform. 25, 210–228.
Herdman, A. T., and Ryan, J. D. (2007). Spatiotemporal brain dynamics underlying saccade execution, suppression and error-related feedback. J. Cogn. Neurosci. 19, 420–432.
Hirvenkari, L. Jousmäki, V., Lamminmäki, S., Saarinen, V., Sams, M. E., and Hari, R. (2010). Gaze-direction-based MEG averaging during audiovisual speech perception. Front. Hum. Neurosci. 4:17. doi: 10.3389/fnhum.2010.00017.
Hollingworth, A. (2009). Two forms of scene memory guide visual search: Memory for scene context and memory for the binding of target object to scene location. Vis. Cogn. 17, 273–291.
Hollingworth, A., and Henderson, J. M. (2002). Accurate visual memory for previously attended objects in natural scenes. J. Exp. Psychol. Hum. Percept. Perform. 28, 113–136.
Hollingworth, A., Richard, A. M., and Luck, S. J. (2008). Understanding the function of visual short-term memory: Transsaccadic memory, object correspondence, and gaze correction. J. Exp. Psychol. Gen. 137, 163–181.
Hollingworth, A., Williams, C. C., and Henderson, J. M. (2001). To see and remember: Visually specific information is retained in memory from previously attended objects in natural scenes. Psychon. Bull. Rev. 8, 761–768.
Holm, L., Eriksson, J., and Andersson, L. (2008). Looking as if you know: systematic object inspection precedes object recognition. J. Vis. 8, 1–7.
Jutras, M. J., and Buffalo, E. A. (2010). Recognition memory signals in the macaque hippocampus. Proc. Natl. Acad. Sci. U.S.A. 107, 401–406.
Konkel, A., and Cohen, N. J. (2009). Relational memory and the hippocampus: representation and methods. Front. Neurosci. 3:23. doi: 10.3389/neuro.01.023.2009.
Kramer, A. F., Hahn, S., Irwin, D. E., and Theeuwes, J. (2000). Age differences in the control of looking behavior: do you know where your eyes have been? Psychol. Sci. 11, 210–217.
Kumaran, D., and Wagner, A. D. (2009). It’s in my eyes, but it doesn’t look that way to me. Neuron 63, 561–563.
Le, S., Raufaste, E., and Demonet, K. (2003). Processing of normal, inverted, and scrambled faces in a patient with prosopagnosia: behavioural and eye tracking data. Cogn. Brain Res. 17, 26–35.
Light, L. L. (1996). “Memory and aging,” in Memory, eds E. L. Bjork and R. A. Bjork (San Diego, CA: Academic Press), 443–490.
Loftus, G. R., and Mackworth, N. H. (1978). Cognitive determinants of fixation location during picture viewing. J. Exp. Psychol. Hum. Percept. Perform. 4, 565–572.
Luck, S. J., and Gold, J. M. (2008). The translation of cognitive paradigms for patient research. Schizophr. Bull. 34, 629–644.
Mackworth, N. H., and Morandi, A. J. (1967). The gaze selects informative details within pictures. Percept. Psychophys. 2, 547–552.
Maljkovic, V., and Nakayama, K. (2000). Priming of popout: III. A short-term implicit memory system beneficial for rapid target selection. Vis. Cogn. 7, 571–595.
Manns, J. R., and Squire, L. R. (2001). Perceptual learning, awareness, and the hippocampus. Hippocampus 11, 776–782.
McKenna, P. J., Tamlyn, D., Lund, C. E., and Mortimer, A. M. (1990). Amnesic syndrome in schizophrenia. Psychol. Med. A Journal of Research in Psychiatry and the Allied Sciences, 20, 967–972.
Miller, G. A., Elbert, T., Sutton, B. P., and Heller, W. (2007). Innovative clinical assessment technologies: Challenges and opportunities in neuroimaging. Psychol. Assess. 19, 58–73.
Milner, B. (1962). “Les troubles de la mémoire accompagnant des lesions hippocampiques bilatérales,” in Physiologie de l’hippocampe, ed. P. Passouant (Paris: Centre de la Recherche Scientifique), 257–262.
Milner, B., Corkin, S., and Teuber, H. L. (1968). Further analysis of the hippocampal amnesic syndrome: 14-year follow-up study of H.M. Neuropsychologia 6, 215–234.
Molnar, F., and Ratsikas, D. (1987). “Some aesthetical aspects of visual exploration,” in Eye Movements: From Physiology to Cognition, eds J. K. O’Regan and A. Levy-Schoen (Amsterdam: North Holland), 363–374.
Moscovitch, M. (2008). The hippocampus as a “stupid” domain-specific module: Implications for theories of recent and remote memory, and of imagination. Can. J. Exp. Psychol. 62, 62–79.
Moses, S. N., Ryan, J. D., Bardouille, T., Kivacevic, N., Hanlon, F. M., and McIntosh, A. R. (2009). Semantic information alters neural activation during transverse patterning performance. Neuroimage 46, 863–873.
Naveh-Benjamin, M. (2000). Adult age differences in memory performance: Tests of an associative deficit hypothesis. J. Exp. Psychol. Learn. Mem. Cogn. 26, 1170–1187.
Noton, D., and Stark. L. (1971). Scanpaths in eye movements during pattern perception. Science 171, 308–311.
Parker, R. E. (1978). Picture processing during recognition. J. Exp. Psychol. Hum. Percept. Perform. 4, 284–293.
Rahhal, T. A., Hasher, L., and Colcombe, S. J. (2001). Instructional manipulations and age differences in memory: now you seem, now you don’t. Psychol. Aging 16, 697–706.
Ranganath, C., Minzenberg, M. J., and Ragland, J. D. (2008). The cognitive neuroscience of memory function and dysfunction in schizophrenia. Biol. Psychiatry 64, 18–25.
Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. Q. J. Exp. Psychol. 62, 1457–1506.
Richmond, J., and Nelson, C. A. (2009). Relational memory during infancy: Evidence from eye tracking. Dev. Sci. 12, 549–556.
Riggs, L., McQuiggan, D., Anderson, A. K., and Ryan, J. D. (in press). Eye movement monitoring reveals differential influences of emotion on memory. Front. Emo. Sci.
Riggs, L., Moses, S. N., Bardouille, T., Herdman, A. T., Ross, B., and Ryan, J. D. (2009). A converging analytic approach to examining medial temporal lobe sources using magnetoencephalography. Neuroimage 45, 627–642.
Ringo, J. L., Sobotka, S., Diltz, M. D., and Bunce, C. M. (1994). Eye movements modulate activity in hippocampal, parahippocampal, and inferotemporal neurons. J. Neurophysiol. 71, 1285–1288.
Rubin, R. D., Brown-Schmidt, S., Duff, M. C., Tranel, D., and Cohen, N. J. (2009). “Common ground representations in hippocampal amnesia,” In Society Neuroscience, 39th Annual Meeting, Chicago, IL.
Ryan, J. D., Althoff, R. R., Whitlow, S., and Cohen, N. J. (2000). Amnesia is a deficit in relational memory. Psychol. Sci. 11, 454–461.
Ryan, J. D., and Cohen, N. J. (2003). Evaluating the neuropsychological dissociation evidence for multiple memory systems. Cogn. Affect. Behav. Neurosci. 3, 168–185.
Ryan, J. D., and Cohen, N. J. (2004a). The nature of change detection and on-line representations of scenes. J. Exp. Psychol. Hum. Percept. Perform. 30, 988–1015.
Ryan, J. D., and Cohen, N. J. (2004b). Processing and short-term retention of relational information in amnesia. Neuropsychologia 42, 497–511.
Ryan, J. D., Hannula, D. E., and Cohen, N. J. (2007). The obligatory effects of memory on eye movements. Memory 15, 508–525.
Ryan, J. D., Herdman, A. T., Riggs, L., and Moses, S. N. (2007). “Eye movement and spatiotemporal brain dynamics underlying processing of relations among objects,” In Cognitive Neuroscience Society Annual Meeting, New York, NY.
Ryan, J. D., Leung, G., Turk-Browne, N. B., and Hasher, L. (2007). Assessment of age-related changes in inhibition and binding using eye movement monitoring. Psychol. Aging 22, 239–250.
Ryan, J. D., Moses, S. N., Ostreicher, M., Bardouille, T., Herdman, A. T., Riggs, L., and Tulving, E. (2008). Seeing sounds and hearing sights: The influence of prior learning on current processing. J. Cogn. Neurosci. 20, 1030–1042.
Ryan, J. D., and Villate, C. (2009). Building visual representations: the binding of relative spatial relations across time. Vis. Cogn. 17, 254–272.
Saykin, A. J., Gur, R. C., Gur, R. E., Mozley, P. D., Mozley, L. H., Resnick, S. M., Kester, D. B., and Stafiniak, P. (1991). Neuropsychological function in schizophrenia: Selective impairment in memory and learning. Arch. Gen. Psychiatry 48, 618–624.
Schacter, D. L. (1987). Implicit memory: history and current status. J. Exp. Psychol. Learn. Mem. Cogn. 13, 501–508.
Schmalzl, L., Palermo, R., Green, M., Brunsdon, R., and Coltheart, M. (2008). Training of familiar face recognition and visual scan paths for faces in a child with congenital prosopagnosia. Cogn. Neuropsychiatry 25, 704–729.
Schwarzer, G., Huber, S., Grueter, M., Grueter, T., Grob, C., Hipfel, M., and Kennerknecht, I. (2007). Gaze behaviour in hereditary prosopagnosia. Psychol. Res. 71, 583–590.
Scoville, W. B., and Milner, B. (1957). Loss of recent memory after bilateral hippocampal lesions. J. Neuropsychiatry Clin. Neurosci. 20, 11–21.
Sluzenski, J., Newcombe, N., and Kovacs, S. (2006). Binding, relational memory, and recall of naturalistic events: a developmental perspective. J. Exp. Psychol. Learn. Mem. Cogn. 32, 89–100.
Smith, C. N., and Squire, L. R. (2008). Experience-dependent eye movements reflect hippocampus-dependent (aware) memory. J. Neurosci. 28, 12833–12825.
Snyder, K. A., Blank, M. P., and Marsolek, C. J. (2008). What form of memory underlies novelty preferences? Psychon. Bull. Rev., 15, 315–321.
Sobotka, S., Nowicka, a., and Ringo, J. L. (1997). Activity linked to externally cued saccades in single units recorded from hippocampal, parahippocampal, and inferotemporal areas of macaques. J. Neurophysiol. 78, 2156–2163.
Sobotka, S., Zuo, W., and Ringo, J. L. (2002). Is the functional connectivity within temporal lobe influenced by saccadic eye movements? J. Neurophysiol. 88, 1675–1684.
Squire L. R. (1992). Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychol. Rev. 99, 195–231.
Stark, L. W., and Ellis, S. R. (1981). “Scanpaths revisited: cognitive models direct active looking,” in Eye Movements: Cognition and Visual Perception, eds D. F. Fisher, R. A. Monty, and J. W. Senders. (Hillsdale, NJ: Lawrence Erlbaum Associates), 193–226.
Stephan, B. C. M., and Caine, D. (2009). Aberrant pattern of scanning in prosopagnosia reflects impaired face processing. Brain Cogn. 69, 262–268.
Suzuki, W. A., and Amaral, D. G. (2004). Functional neuroanatomy of the medial temporal lobe memory system. Cortex 40, 220–222.
Tesche, C. D., and Karhu, J. (2000). Theta oscillations index human hippocampal activation during a working memory task. Proc. Natl. Acad. Sci. U.S.A. 97, 919–924.
Theeuwes, J., Belopolsky, A., and Olivers, C. N. L. (2009). Interactions between working memory, attention and eye movements. Acta Psychologica 132, 106–114.
Tole, J. R., Stephens, A. T., Vivaudou, M., Harris, R. L. and Ephrath, A. R. (1982). “Entropy, instrument scan, and pilot workload. In Proceedings of the International Conference on Cybernetics and Society (New York: IEEE), 588–592.
Van der Stigchel, S., Meeter, M., and Theeuwes, J. (2006). Eye movement trajectories and what they tell us. Neurosci. Biobehav. Rev. 30, 666–679.
Walker-Smith, G. J., Gale, A. G., and Findlay, J. M. (1977). Eye movement strategies involved in face perception. Perception 6, 313–326.
Warren, D. E., Duff, M. C., Tranel, D., and Cohen, N. J. (2010). Medial temporal lobe damage impairs representation of simple stimuli. Front. Hum. Neurosci. 4:35. doi: 10.3389/fnhum.2010.00035.
Keywords: eye movements, fMRI, MEG, memory, hippocampus, amnesia
Citation: Hannula DE, Althoff RR, Warren DE, Riggs L, Cohen NJ and Ryan JD (2010). Worth a glance: Using eye movements to investigate the cognitive neuroscience of memory. Front. Hum. Neurosci. 4:166. doi:10.3389/fnhum.2010.00166
Received: 05 May 2010;
Paper pending published: 13 June 2010;
Accepted: 26 July 2010;
Published online: 08 October 2010.
Edited by:
Michael X. Cohen, University of Amsterdam, NetherlandsReviewed by:
Walter Boot, Florida State University, USAKelly Giovanello, Cognitive Neuroscience of Memory Laboratory, USA
Copyright: © 2010 Hannula, Althoff, Warren, Riggs, Cohen and Ryan. This is an open-access publication subject to an exclusive license agreement between the authors and the Frontiers Research Foundation, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are credited.
*Correspondence: Deborah E. Hannula, Department of Psychology, University of Wisconsin-Milwaukee, 224 Garland Hall, 2441 E. Hartford Ave., Milwaukee, WI 53211, USA. e-mail: hannula@uwm.edu