# PERCEIVING AND ACTING IN THE REAL WORLD: FROM NEURAL ACTIVITY TO BEHAVIOR

EDITED BY: Simona Monaco, Gavin Buckingham, Irene Sperandio and J. Douglas Crawford PUBLISHED IN: Frontiers in Human Neuroscience

#### *Frontiers Copyright Statement*

*© Copyright 2007-2016 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

> *All copyright, and all rights therein, are protected by national and international copyright laws.*

*The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88945-028-2 DOI 10.3389/978-2-88945-028-2

# About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

# Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

# Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

# What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# **PERCEIVING AND ACTING IN THE REAL WORLD: FROM NEURAL ACTIVITY TO BEHAVIOR**

Topic Editors:

**Simona Monaco,** University of Trento, Italy **Gavin Buckingham,** University of Exeter, UK **Irene Sperandio,** University of East Anglia, UK **J. Douglas Crawford,** York University, Canada

We are grateful to Stefania Benetti for providing the image of the brain with fiber tracts for the cover of this e-book. Photo credit: University of Exeter.

One remarkable ability of the human brain is to process large amounts of information about our surroundings to allow us to interact effectively with them. In everyday life, the most common way to interact with objects is by reaching, grasping, lifting and manipulating them. Although these may sound like simple tasks, the perceptual properties of the target object, such as its location, size, shape, and orientation all need to be processed in order to set the movement parameters that allow an accurate reach-to-grasp-to lift movement. Several brain areas work in concert to process this outstanding amount of visual information and drive the execution of a motor plan in just a few hundred milliseconds. How are these processes orchestrated?

In developing this type of comprehensive knowledge about the interactions between objects perception and goal-directed actions, we have a window into the mechanisms underlying the functioning of the visuo-motor system. With this

research topic we aim to further understand the neural mechanisms that mediate our interactions with the world. Therefore, we particularly encourage submission of papers that attempt to relate such findings to real-world situations by investigating behavioural and neural correlates of information processing related to eye-hand coordination and visually-guided actions, including reaching, grasping, and lifting movements. This topic welcomes submissions of original research using any relevant techniques and methods, from behavioural kinematics/kinetics, to neuroimaging and transcranial magnetic stimulation (TMS), as well as neuropsychological studies.

**Citation:** Monaco, S., Buckingham, G., Sperandio, I., Crawford, J. D., eds. (2016). Perceiving and Acting in the Real World: From Neural Activity to Behavior. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-028-2

# Table of Contents



Bahareh Taghizadeh and Alexander Gail

*151 Binocular advantage for prehension movements performed in visually enriched environments requiring visual search*

Roshani Gnanaseelan, Dave A. Gonzalez and Ewa Niechwiej-Szwedo

*162 An allocentric exception confirms an egocentric rule: a comment on Taghizadeh and Gail (2014)*

Paul Dassonville, Benjamin D. Lester and Scott A. Reed

*164 The development of trunk control and its relation to reaching in infancy: a longitudinal study*

Jaya Rachwani, Victor Santamaria, Sandra L. Saavedra and Marjorie H. Woollacott


Marco Davare, Alexandre Zénon, Michel Desmurget and Etienne Olivier

*206 Corrigendum: The development of trunk control and its relation to reaching in infancy: a longitudinal study*

Jaya Rachwani, Victor Santamaria, Sandra L. Saavedra and Marjorie H. Woollacott


F. Garbarini, L. Turella, M. Rabuffetti, A. Cantagallo, A. Piedimonte, E. Fainardi, A. Berti and L. Fadiga

*230 Direct comparisons of hand and mouth kinematics during grasping, feeding and fork-feeding actions*

D. J. Quinlan and J. C. Culham


Mattia Marangon, Agnieszka Kubiak and Gregory Króliczak

# Editorial: Perceiving and Acting in the Real World: From Neural Activity to Behavior

#### Simona Monaco<sup>1</sup> \*, Gavin Buckingham2, 3, Irene Sperandio<sup>4</sup> and J. Doug Crawford<sup>5</sup>

*<sup>1</sup> Center for Mind/Brain Sciences, University of Trento, Italy, <sup>2</sup> School of Life Sciences, Heriot-Watt University, Edinburgh, UK, <sup>3</sup> Sport and Health Sciences, University of Exeter, Exeter, UK, <sup>4</sup> School of Psychology, University of East Anglia, Norwich, UK, <sup>5</sup> Centre for Vision Research, York University, Toronto, ON, Canada*

#### Keywords: perception, vision, touch, action, grasping, reaching, saccades

**The Editorial on the Research Topic**

## **Perceiving and Acting in the Real World: From Neural Activity to Behavior**

The interaction between perception and action represents one of the pillars of human evolutionary success. Our interactions with the surrounding world involve a variety of behaviors, almost always including movements of the eyes and hands. Such actions rely on neural mechanisms that must process an enormous amount of information in order to generate appropriate motor commands. Yet, compared to the great advancements in the field of perception for cognition, the neural underpinnings of how we control our movements, as well as the interactions between perception and motor control, remain elusive. With this research topic we provide a framework for: (1) the perception of real objects and shapes using visual and haptic information, (2) the reference frames for action and perception, and (3) how perceived target properties are translated into goal-directed actions and object manipulation. The studies in this special issue employ a variety of methodologies that include behavioral kinematics, neuroimaging, transcranial magnetic stimulation, and patient cases. Here we provide a brief summary and commentary on the articles included in this research topic.

# Edited and reviewed by:

*Srikantan S. Nagarajan, University of California, San Francisco, USA*

#### \*Correspondence:

*Simona Monaco simona.monaco@gmail.com*

Received: *19 February 2016* Accepted: *08 April 2016* Published: *27 April 2016*

#### Citation:

*Monaco S, Buckingham G, Sperandio I and Crawford JD (2016) Editorial: Perceiving and Acting in the Real World: From Neural Activity to Behavior. Front. Hum. Neurosci. 10:179. doi: 10.3389/fnhum.2016.00179*

# 3D VISION FOR PERCEPTION AND ACTION

Snow et al. (2011) have previously shown that the neural mechanisms involved in the visual processing of 3D real objects differ from those involved in processing 2D images of the same objects. Here, Snow et al. provide behavioral evidence that real-world objects are more memorable than photographs of the same objects. This difference might be related to higher-level attributes that are intrinsic to real but not images of objects, such as affordances for actions, prior associations of a real object with our experience in the world and/or differences in binocular depth cues.

However, binocular vision is not always necessary for movement control. When we perform actions, such as grasping an object or hitting a ball, we normally have binocular vision of the goal and the surrounding scene. Despite the potential advantages of binocular vision, monocular viewing provides sufficient information to engage in online control to correct initial errors in movement planning (Brenner et al.; Gnanaseelan et al.).

# REFERENCE FRAMES FOR VISUAL LOCALIZATION AND AIMING MOVEMENTS

The accurate localization of a target in the surrounding environment is essential for skilled aiming movements. There are two general types of reference frames that can be used in order to localize objects in the extra-personal space: allocentric and egocentric. While allocentric frames of reference allow us to encode the location of a target relative to contextual cues in the outside world, egocentric frames of reference allow us to encode the location of a target relative to one's self. Reliance on these frames of reference has been shown to depend on the demands of the spatial task (Taghizadeh and Gail) and on the nature of the task itself (Fiehler et al.). Indeed, while perceptual tasks—generally associated with the ventral visual stream and object-centered coding—are affected by visual illusions, tasks that involve an action—generally associated with the dorsal visual stream and egocentric coding—are not (Dassonville et al.). However, Filimon et al.suggests that all spatial comparisons, even those between different allocentric cues, are ultimately processed in the brain with respect to the self, and therefore involve egocentric frames of reference. If so, this would require a reconsideration of schemes that separate ventral vs. dorsal stream vision on the basis of allocentric vs. egocentric processing.

# AIMING MOVEMENTS OF THE EYES AND HAND

Behavioral dissociations between the control of reach direction and amplitude have been recognized for decades (Soechting and Flanders, 1992). However, a corresponding dissociation between the neural mechanisms for reach direction and depth—past early visual cortex—remains controversial. Here, Davare et al. provide evidence for a double-dissociation between direction and amplitude coding for reaching movements within the frontoparietal reaching circuit in humans. While aIPS is involved in processing the direction of movements, dPM processes the amplitude of movements.

Saccades are often tested in paradigms where their past history is disregarded. However, Jones et al. show that the direction of prior saccades affects the direction of current saccades. Finally, oculomotor physiologists are familiar with studies where visual information is remapped in eye-centered coordinates to compensate for saccades. However, when the saccade target is located on the body, a more complex series of egocentric reference frame transformations is required to account for the position and motion of that body part. Buchholz et al. demonstrate that tactile remapping for saccades induces alpha and beta oscillations that prepare the brain for the upcoming eyemovement based on eye- and body-centered frames of reference.

# GRASP CONTROL AND KINEMATICS

Grasping movements have been extensively studied over the past two decades, and human neuroimaging as well as cell recording in macaques have allowed unveiling the neural mechanisms underlying actions (for a review see Turella and Lingnau). The two main components of grasping movements consist of reaching the target location and pre-shaping the fingers according to the shape, size and orientation of the target object. Begliomini et al. show that although the involvement of dorsal visual stream areas in reaching and grasping depends on the temporal progression of the movement, similar areas are sensitive to both types of movements, suggesting that the neural underpinnings of reaching and grasping may overlap in both in spatial and temporal terms.

The intimate relationship between the visual system and grasp control is further illustrated by its dependence on field of view. For instance, the motor control of an action is facilitated when the object to be acted upon is in the same visual hemifield of our hand. In particular, right-handed participants scale their grip aperture more accurately to objects placed on their right visual field when grasping with the right hand. Similarly, participants scale their grip aperture more accurately to objects placed on the left visual hemifield when grasping with the left hand (Le and Niemeier).

Finally, grasp is also influenced by the effector and intended use of the object. In particular, Quinlan et al. report that the kinematics of biting a piece of food, which in essence consists of grasping the piece of food with the mouth, differ from those of grasping with the hand. In particular, participants oversize the mouth to a lesser extent when biting than the hand when grasping the same-sized piece of food. The use of a tool, such as a fork, also affects the movement kinematics by slowing down hand movements while leaving the grip component unchanged.

# HAPTIC CONTRIBUTION TO PERCEPTION AND ACTION

Although vision is the sense that we most use in order to perceive objects in our environment, haptic feedback is also crucial for manual exploration and grasping movements. For example, Whitwell et al. show that the removal of haptic feedback at the end of a grasping movement causes higher reliance on vision and cognitive supervision, resulting in grasps that appear to be more like pantomimed movements.

When vision is degraded or unavailable, we often use touch in order to explore and recognize objects in our environment. The network of areas involved in haptic exploration includes much of the cerebral cortex, ranging from occipital areas to temporal and fronto-parietal cortices. In particular, Marangon et al. find that haptic exploration engages ventral visual stream areas (including the lateral occipital area, LO) known to be involved in visual recognition of objects. Marangon et al. further show that LO is involved in exploring and grasping shapes regardless of their complexity, whereas the anterior intraparietal sulcus (aIPS), a dorsal stream area, is more involved in performing grasping movements toward complex vs. simple shapes that have been haptically explored. It has been previously suggested that grasps of increased complexity toward visually explored objects, like tools, require the recruitment of ventral stream areas (van Polanen and Davare, 2015). Therefore, it is possible that the

extent to which ventral and dorsal stream areas are recruited during actions toward complex shapes depends on the sensory modality used to explore the shape, with higher involvement of ventral stream areas when the object has been seen and higher involvement of dorsal stream areas when the object has been haptically explored.

# MULTI-SENSORY CALIBRATION AND MOTOR LEARNING

Since vision is such a dominant sense, it affects the motor and proprioceptive systems even when only limited visual information is available. Indeed, Barkley et al. provide evidence that brief exposure to altered vision of one's arm position in the environment induces motor adaptation and proprioceptive recalibration, resulting in the matching of proprioception with the misaligned visual feedback. This study highlights the powerful influence of vision on both multi-sensory calibration and the dynamics of motor learning.

Successful interactions with the external world includes the ability to accurately predict the forces necessary to lift and manipulate objects in our environment. However, such actions are often subject to perturbations that can be caused by either internal or external factors. For example, the act of walking can be rendered more effortful by an internal factor, such as tiredness, or an external factor, such as walking against the direction of the wind. To make sure that learning happens in the appropriate context, it is important that perturbations are correctly attributed to the right source. In their paper, Fercho et al. contribute to this topic by showing that the rate of perturbation that is experienced by participants while lifting an object plays a critical role in how the motor system solves the credit assignment problem and consequently, motor adaptation effects for subsequent lifting actions.

# SENSORIMOTOR COORDINATION

Bimanual coordination is required for many daily activities, ranging from simple tasks such as peeling an orange, to more complex learned tasks like playing the piano. In these examples, the hands concurrently perform different movements, each with differing temporal and spatial demands. Garbarini et al. explored the neural correlates of congruent and noncongruent bimanual coordination in patients with motor neglect. Congruent movements consisted of performing the same drawings (lines or circles) with both hands, while non-congruent movements required participants to perform different movements with the two hands (line drawings with the right hand and circle drawings with the left hand). The lack of interference between the motor programs of two hands during non-congruent bimanual movements (an effect observed in individuals with motor neglect) is associated with decreased activation in pre-supplementary motor area (pre-SMA) as compared to congruent bimanual movements. Control participants (with and without brain damage) showed the opposite pattern, with higher activation in pre-SMA for non-congruent vs. congruent bimanual coordination. These results suggest that the lack of inhibition exerted by pre-SMA might be at the basis of the behavioral impairment during non-congruent bimanual coordination in patients with motor neglect. This might be related to the role of pre-SMA in in processing the flow of information between the two hemispheres in order to control for interference between the motor programs of the two hands during bimanual coordination.

Parkinson's disease (PD) is associated with basal ganglia dysfunction and a number of symptoms, including deficits in muscle co-activation (i.e., synergies). In particular, van der Stouwe et al. demonstrate that decreased performance in composite arm movements in PD patients is associated with decreased activity in the striatum and in the fronto-parietal network. This highlights the need to understand interactions between sub-cortical and cortical disease processes such as PD.

Finally, arm movements are not controlled in isolation from the rest of the body. Indeed, the trunk provides the postural basis for reach. Effects of controlling the upper and lower regions of the trunk during reaching provides insight into the mechanisms by which trunk control impacts reaching in infants. Trunk control is acquired in a segmental sequence across development of upright sitting and it is tightly correlated with reaching performance (Rachwani et al.)

# ACTION OBSERVATION

Actions allow us to interact not only with objects in our environment but also with other people. During social interactions, we observe other people's movements which in turn activate our own motor system through a process known as "motor resonance." Motor resonance is differently affected by hand dominance (Sartori et al.), suggesting that the motor system is fine-tuned not only to our own actions but also to other people's actions. In addition, Balser et al. provide evidence that, during action observation, brain activity in the fronto-parietal network correlates with performance in sport experts when anticipating the effects of actions performed by others in their preferred discipline.

# CONCLUSIONS

This research topic outlines a number of recent advances in our understanding of the neural mechanisms and the associated behavior for perception and action. In this review, we have emphasized the intimate relationship between perceptual motor systems, not only in the obvious sense that sensation can be used to guide action, but in the many ways that perception and action interact, up to and including the perception of actions in others. Further, the many examples cited above illustrate the clear link between this topic and applications for real world behavior; not only for clinical populations and elite athletes, but in nearly every aspect of our waking lives. For this, we are grateful to all of the authors and reviewers that contributed to the composition of this special topic issue.

# AUTHOR CONTRIBUTIONS

SM, JC contributed to the design of the work and drafted the editorial. GB, IS revised the draft for important intellectual content and contributed with the interpretation of the work. SM, GB, IS, JC approved the final version to be published and agreed to be accountable for all aspects of the editorial in ensuring that questions related to the accuracy or integrity

# REFERENCES


of any part of the work are appropriately investigated and resolved.

# FUNDING

Canada Research Chairs, Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada to JC.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Monaco, Buckingham, Sperandio and Crawford. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Precise timing when hitting falling balls

# *Eli Brenner\*, Ben Driesen and Jeroen B. J. Smeets*

*Faculty of Human Movement Sciences, MOVE Research Institute, VU University Amsterdam, Netherlands*

#### *Edited by:*

*Gavin Buckingham, Heriot-Watt University, UK*

#### *Reviewed by:*

*Jean-Jacques Orban De Xivry, Université Catholique De Louvain, Belgium Adrian Mark Haith, Johns Hopkins University School of Medicine, USA*

*\*Correspondence:*

*Eli Brenner, Faculty of Human Movement Sciences, Vrije Universiteit, Van der Boechorststraat 9, 1081 BT Amsterdam, Netherlands e-mail: e.brenner@fbw.vu.nl*

People are extremely good at hitting falling balls with a baseball bat. Despite the ball's constant acceleration, they have been reported to time hits with a standard deviation of only about 7 ms. To examine how people achieve such precision, we compared performance when there were no added restrictions, with performance when looking with one eye, when vision was blurred, and when various parts of the ball's trajectory were hidden from view. We also examined how the size of the ball and varying the height from which it was dropped influenced temporal precision. Temporal precision did not become worse when vision was blurred, when the ball was smaller, or when balls falling from different heights were randomly interleaved. The disadvantage of closing one eye did not exceed expectations from removing one of two independent estimates. Precision was higher for slower balls, but only if the ball being slower meant that one saw it longer before the hit. It was particularly important to see the ball while swinging the bat. Together, these findings suggest that people time their hits so precisely by using the changing elevation throughout the swing to adjust the bat's movement to that of the ball.

**Keywords: interception, timing, hitting, gravity, motor control, precision, baseball, vision**

# **INTRODUCTION**

People are extremely good at intercepting a falling ball with a bat (McLeod et al., 1985; Brenner et al., 2012). They can time their attempts to hit a ball with a precision of about 7 ms (we use the standard deviation as our measure of temporal precision throughout this article). This is much better than the temporal precision in indicating which of two targets stopped moving first (27 ms at best; Figure 3C of Tadin et al., 2010) or changed length first (35 ms at best; Figure 5 of Baruch et al., 2013). It is also much better than the precision in indicating whether the interval between the first and the second of three flashes was longer or shorter than the interval between the second and the third flash (about 30 ms; Figure 2A of Zanker and Harris, 2002). The precision with which movements of the two hands can be synchronized is about 14 ms (Figure 4E of Doumas and Wing, 2007; Figure 6D of Doumas et al., 2008), as is the precision with which expert pianists can time their keystrokes (Figure 3B of Goebl and Palmer, 2013). To our knowledge, temporal precision is only less than 7 ms for judging which of two adjacent targets was flashed first (Figure 1 of Westheimer and McKee, 1977a), in which case the temporal order is presumably judged from the perceived motion (Brenner and Smeets, 2010).

Moreover, there is abundant evidence that the human visual system is quite poor at judging the instantaneous acceleration (Gottsdanker et al., 1961; Werkhoven et al., 1992), yet the abovementioned high temporal precision of interception is achieved with a falling ball that is accelerated by gravity. Thus, people must be relying on their experience with previous balls or with falling objects in general to judge the acceleration (Zago et al., 2004, 2009), or continuously be adjusting their movements to minimize the influence of misjudging the acceleration (Lee et al., 1983). Altogether, the temporal precision in intercepting falling balls appears to be at the very limit of what one could expect considering the required visual judgments about the ball's approach and the need to move the bat accordingly.

Not all interception studies report a temporal precision of about 7 ms. Poorer precision has been found when hitting falling balls under more constrained conditions (23 ms at best; Table 1 of Katsumata and Russell, 2012), when hitting virtual targets that move at a constant velocity (22 ms; Table 2 of Brenner and Smeets, 2009), and when hitting real targets that move at a constant velocity (12.5 ms at best; data including misses in Figure 4 of Tresilian et al., 2003). Moreover, a high precision appears to require continuous updating of sensory information (Bootsma and van Wieringen, 1990; Land and McLeod, 2000; Brenner and Smeets, 2011).

McLeod and Jenkins (1991) argued that the temporal precision in batting a (falling) ball is limited by the spatial resolution of the human eye. They did so on the basis of calculations involving the rate of expansion of the ball's retinal image and their estimate of the latencies involved in guiding the hitting movement. Michaels et al. (2001) observed differences between monocular and binocular performance when intercepting a falling ball, suggesting that retinal image size may not be the only relevant information. For catching, Rushton and Wann (1999) proposed that information based on retinal image size is combined with binocular information to improve performance (also see Regan, 1997; van der Kamp et al., 1997; Bennett et al., 2000; Regan and Gray, 2000). In a similar way, we here consider that the ball's changing elevation angle might provide critical information, because the ball does not fall straight toward the eye (since it is hit at some distance from the body).

In one of the previous studies in which a temporal precision of about 7 ms was found (Brenner et al., 2012), balls were released from a height of 5.7 m and were hit about 1.24 m above the ground. Three parameters change smoothly as the ball approaches: the ball's angular size and the binocular convergence that is required to keep looking at a position on the ball (binocular disparity) increase until the ball passes the batsman's eyes, while the ball's elevation angle continuously decreases. Assuming that the ball (6.6 cm diameter) passed about 0.7 m from the batsman's eyes, and that the eyes were 1.6 m above the ground with an inter-pupil distance of 6.6 cm, we can calculate that 200 ms before the ball is hit, the diameter of its image is increasing at 12◦/s. The binocular disparity at that time is also changing at 12◦/s, and the elevation angle is changing at 235◦/s. Similarly, 100 ms before the ball is hit, its image size and binocular disparity are changing at 24◦/s and its elevation angle at 349◦/s. These calculations are based on a ball (0.057 kg) falling under gravity (9.81 m/s2) with air resistance [drag force <sup>=</sup> <sup>½</sup>ρ*v*<sup>2</sup>*CDA*, using 1.225 for the density of air (ρ), 0.6 for the drag coefficient (*CD*; based on Goodwill et al., 2004) and 0.0034 m2 for the ball's cross-sectional area (*A*). *v* is the ball's speed]. The calculations were verified by comparing the calculated speed of the ball with the measured speed near the height at which the ball was hit.

Although the calculations show that the rate of change is an order of magnitude larger for the elevation angle than for the other two parameters, one must keep in mind that in terms of angles, the precision with which people can judge changes in image size is presumably much higher than the precision with which they can judge changes in elevation or ocular convergence. The former is probably limited by the retinal resolution (about 1 arc in the fovea, although higher precision can be achieved in some tasks; Westheimer and McKee, 1977b) whereas the latter are probably limited by the resolution of judging eye orientation (about 6 arc at best; Brenner and Smeets, 2000). We therefore started the current study, in which people tried to hit falling balls with a bat (as in Brenner et al., 2012), by varying the circumstances in ways that are likely to affect the above-mentioned sources of information. In subsequent experiments we examined how various other manipulations influence people's timing when hitting falling balls.

In the first experiment, we evaluated the role of binocular information by comparing how well subjects hit tennis balls with one eye closed, with how well they did so when they had both eyes open. We evaluated the importance of a high resolution with which to detect changes in the ball's retinal image size in two ways: by having subjects wear reading glasses that blurred the images on their retinae, and by reducing the ball's retinal image size by having subjects hit smaller balls. The manipulations hardly affected the subjects' performance. In the first experiment the ball always appeared at the same height, moving at the same speed. In the second experiment, we varied the ball's speed at the time it appeared, to check that people were not just hitting a fixed time after the ball appeared, and more generally to evaluate to what extent people were relying on feedback from previous trials. Whether all balls were moving at the same speed or not made no difference, but slower balls were hit more precisely. In the third experiment, we examined whether the fact that people's timing was less precise when the ball moved faster was because of the speed itself, or because a faster ball had to be hit sooner after it had appeared. We found that it was clearly the time that the ball was visible, and not its speed, that determined the temporal precision. In the second and third experiments, increasing the time that the ball was visible meant that one saw it longer before the hit. In the fourth experiment, we examined whether seeing the ball earlier was particularly important because it allowed one to better initiate the hit.

# **METHODS**

The task was always to hit a falling ball with a bat. The subject's aim was for the ball to hit a target that was at waist height, several meters away (shown schematically in **Figure 1A**). Except when mentioned otherwise, the balls were regular sized tennis balls. The bat was a children's foam-covered baseball bat that we bought in a toy shop (total length: 68.5 cm; diameter of relevant section: 5 cm). The experiments were conducted in a well-lit sports hall within our department. The balls were released from various heights and fell through tubes of various lengths, allowing us to independently vary the balls' speeds and the times for which they were visible before being hit. The subjects were all young adults. None of the subjects were aware of the hypotheses, but the manipulations were quite evident. Subjects were instructed to stand and hit in such a manner that the bat would be oriented approximately horizontally when it hit the ball, but received no further instructions about how to perform the task. The study is part of a research program that has been approved by the local ethical committee, and all subjects signed the standard informed consent form. All tested conditions and exclusions are reported.

**FIGURE 1 | Overview of task and conditions.** Subjects had to try to hit a falling ball toward a target region. The ball was released through a tube (indicated in black). **(A)** The configuration of the corresponding condition in Brenner et al. (2012). **(B–E)** The four conditions of Experiment 1: *baseline*, *monocular*, *blurred*, and *small ball*. **(F,G)** The two conditions of Experiment 2: *varying speed* and *fixed speed*. (**H–J**) The three conditions of Experiment 3: *fast*, *fast & short*, and *slow & short*. **(K–M)** The three conditions of Experiment 4: *transparent tube*, *early vision*, and *late vision*. The drawings are approximately to scale, showing the lengths and heights of the release tubes as well as the approximate height of the horizontal "bat" at the anticipated moment of impact. The gray disk represents the ball at an arbitrary moment. The number of participants that were included in the analysis is indicated at the bottom right for each experiment.

#### **EXPERIMENT 1**

Eleven male, right-handed subjects (with normal binocular vision as tested with the "Stereo fly test") agreed to participate in Experiment 1, but two subjects' data were not analyzed. In one case this was due to equipment failure. In the other it was because we only noticed half way through the session that the bat was not oriented approximately horizontally at the time of the hit, which made it impossible to reliably estimate the timing precision. We will henceforth only consider the remaining 9 subjects.

There were four conditions (**Figures 1B–E**) that were performed in separate blocks of trials. The order in which the conditions were presented was selected at random for each subject. There were breaks both between and half way through the blocks, while the experimenters gathered the balls. Each block of 60 trials was preceded by 12 practice trials that were not analyzed. In one condition (*baseline*) subjects hit the tennis balls with no additional restrictions. In a second condition (*monocular*) they kept one eye closed when doing. They were free to choose which eye to close (four closed their right eye and five closed their left eye). In a third condition (*blurred*) two near-sighted subjects who normally wore spectacles (−2.5 and −4.5 D) removed their spectacles and the others wore +2.5 D reading glasses (making them near-sighted, so that the ball's retinal image will have been blurred, especially early during its fall). In the fourth condition (*small ball*) the tennis balls (diameter of 6.6 cm) were replaced by bouncing balls (diameter of 3.6 cm). Calculations suggest that the latter were not only smaller but were also moving almost 3% faster than the tennis balls at the time of the hit.

The balls were released from a height of 4.9 m. The release tube was 58 cm long, and had a diameter of 7 cm for the tennis ball and of 5 cm for the bouncing ball. The target was at a distance of 5.6 m. Except for scoring whether or not the bat touched the ball, we also used an Optotrak 3020 system (Northern Digital Inc., Waterloo, Ontario, Canada) to measure the movement of an infrared light emitting diode that was attached to the tip of the bat (positions determined at 800 Hz). For each subject and condition we determined the fraction of balls that were "touched" by the bat. We used the measured horizontal and vertical speeds of the tip of the bat, and the curvature of the tip of the bat's path (determined during the time at which it moved faster than 4 m/s, assuming a constant curvature), to estimate the speed of the relevant part of the bat (the part with which the ball is hit) at the time at which the bat hit or passed the ball. In doing so we assumed that the relevant part of the bat was 20 cm from the tip, and that the bat's movement was a rotation around a point along an extension of the bat's main axis (so this point can be derived from the measured curvature). For trials in which the ball was hit, we also measured the direction of the acceleration of the bat caused by the impact with the ball.

We need to know the vertical velocity of the ball (relative to the relevant part of the bat) and the horizontal velocity of the relevant part of the bat in order to estimate the time window for hitting the ball. Knowing the time window and the fraction of touched balls, and assuming a normal distribution of timing errors, would be enough to estimate the temporal precision if we could be sure that subjects' average timing was correctly chosen so as to touch as many balls as possible with the bat. However, the subjects' aim was not to touch as many balls as possible, but to hit the target with the balls, so we also used information about the direction of the impact between the bat and the ball to estimate the average timing.

We are primarily interested in the standard deviation of the temporal errors (σ). We assume that these errors are normally distributed, but consider the possibility of a bias (μ) to arrive a bit later than would be optimal for touching as many balls as possible, because the participants' task was not to touch the balls but to hit them toward a target. Thus, the probability of arriving at time *t* with respect to the time for which the bat would touch the most balls is:

$$P(t) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(t-\mu)^2}{2\sigma^2}}$$

The range of times for which the bat will touch the ball and the direction in which the bat will hit the ball for a given time *t* depends on the horizontal speed of the bat (*Vbat*) and the vertical speed of the ball relative to the bat (*Vball*). For a sum of the radii of the bat and the ball of *r*, the ball will be hit if the value *h(t)* is positive:

$$h(t) = V\_{bat}^2 t^2 - \left(V\_{bat}^2 + V\_{ball}^2\right) \left(V\_{bat}^2 t^2 - r^2\right)$$

The direction of the acceleration at impact (assuming a perfectly elastic collision) is then:

$$d(t) = \arctan\left(\frac{V\_{ball}(\sqrt{h(t)} + V\_{bat}^2 t)}{V\_{bat}\left(\sqrt{h(t)} - V\_{ball}^2 t\right)}\right)$$

In order to estimate σ from the number of balls that were hit (*nhit*) and the mean measured direction of the acceleration of the bat at impact (*d*¯*m*) we solve:

$$\int\_{h(t)>0} p(t)dt = \frac{n\_{\text{hit}}}{n\_{\text{total}}}$$

and

$$\int\_{h(t)>0} p(t) \, d(t) dt = \bar{d}\_m$$

where *ntotal* is the total number of balls presented. We solved these two integrals simultaneously for each subject and condition to estimate σ (and μ). We compare the timing precision (σ) across the conditions with a repeated measures analysis of variance.

## **EXPERIMENT 2**

Six male and six female subjects (one left-handed) participated in Experiment 2. There were two conditions (**Figures 1F,G**) that were performed in separate blocks of trials in a counterbalanced order. Again, there were breaks both between and half way through the blocks while the experimenters gathered the balls. Each block of 100 trials was preceded by 12 practice trials that were not analyzed. In one condition (*varying speed*) 40 balls fell from a height of 5.79 m, and 20 balls each from heights of 4.74, 5.27, and 6.34 m, so they were moving at different speeds at the time that they were to be hit. The 100 trials were presented in a random order, so that the different speeds were interleaved. In a second condition (*fixed speed*) all 100 balls fell from a height of 5.79 m. In all cases the ball came into view (exited the release tube) at a height of 4.32 m. The target was at a distance of about 5 m.

In this experiment we filmed the hits at a high temporal resolution (1000 Hz) and low spatial resolution (224 by 64 pixels) with a Casio Exilim EX-ZR1000 camera. To compensate for the low spatial resolution we zoomed in on the region in which we expected the ball to be hit. As a result, data were lost if subjects hit a ball much higher or lower than expected. A calibration panel that was placed in the ball's path before the experiment allowed us to convert pixels in the image into distances in the world. The camera's frequency was verified by filming a rapidly flashing light emitting diode of which the frequency was determined with an oscilloscope. The camera was about 4 m from the ball's path, orthogonal to the direction toward the target, so that at the time of the hit the ball was moving downwards in the image, in front of the subject, and the tip of the bat was more or less facing the camera, moving to the right in the image for the right-handed subjects, and to the left for the left-handed subject (in which case the subject and camera switched sides with respect to the ball's path).

The timing of the bat with respect to the ball was determined by stepping through the recordings while counting the images. For each trial we determined the difference in time between when the ball and the bat reach the point at which their paths cross. If the ball was hit before reaching that point, the spatial calibration was used to calculate the time at which the ball would have crossed that point. The precision is the standard deviation of these time differences. We compared the timing precision for the balls that fell from a height of 5.79 m across the two conditions with a repeated measures analysis of variance. We compared the timing precision for the balls that fell from the four different heights in the *varying speed* condition with a second repeated measures analysis of variance. We used *t*-tests with Bonferroni correction to identify the heights for which the subjects had significantly different precision.

# **EXPERIMENT 3**

Nine male and five female subjects (one left-handed) participated in Experiment 3. One female subject's data was not included in the analysis, because a majority of her hits took place outside the image. There were three conditions (**Figures 1H–J**) that were performed in separate blocks of trials. The order in which the conditions were presented was selected at random for each subject. Each block of 60 trials was preceded by 4 practice trials that were not analyzed. There were breaks between the blocks while the experimenters gathered the balls.

In one condition (*fast*) the balls fell from a height of 5.85 m, so they were visible for almost a second before they reached a height of 125 cm (the height at which we anticipated that the ball would be hit). In a second condition (*fast & short*) the ball fell from the same height, but during the first 1.39 m it was within a release tube, so it was moving at the same speed but had only been visible for 450 ms when it reached a height of 125 cm. In a third condition (*slow & short*) the ball was released at a height of 3.13 m, with a 15 cm release tube, so it moved more slowly but was also visible for 450 ms by the time it reached a height of 125 cm.

The target was at a distance of about 4.4 m and the Casio Exilim EX-ZR1000 camera at a distance of about 3.2 m from the ball's path. For trials in which the ball was hit, estimates of the timing of the hit were refined by also measuring the direction in which the ball moved after the hit. Otherwise the procedure was identical to that of Experiment 2. We compared the timing precision across the conditions with a repeated measures analysis of variance, and then used *t*-tests with Bonferroni correction to identify the conditions in which the subjects had significantly different precision.

#### **EXPERIMENT 4**

Seven male and two female subjects (one left-handed) participated in Experiment 4. In this experiment the ball always fell from a height of 3.95 m, first falling 15 cm through an opaque release tube, and then 200 cm through a (also 7 cm diameter) transparent tube, which it exited at a height of 1.8 m. The ball's motion in this experiment was not precisely as described by the equation given in the introduction, because the balls clearly moved differently within the tube (probably due to the larger air resistance), so the positions at which the tube had to be covered to achieve the desired viewing times were based on measurements (from camera images) rather than on calculations.

There were three conditions (**Figures 1K–M**) that were performed in separate blocks of trials in a random order. Each block of 60 trials was preceded by 10 practice trials that were not analyzed. There were breaks between the blocks while the experimenters gathered the balls. In one condition (*transparent tube*) the balls were visible for 600 ms within the transparent tube and then for about 67 ms after exiting the tube. In a second condition (*early vision*) the lower 1.3 m of the tube was covered, so that the ball was only visible for the first 300 ms of its path through the transparent tube. In a third condition (*late vision*) the top 0.7 m of the tube was covered, so that the ball was only visible for the last 300 ms of its path through the transparent tube.

In this experiment we set the camera so that we could see when the bat started moving forward (by zooming in less). We used this to estimate about how long it took to hit the ball for five trials of each subject in each condition, and determined the overall median of these estimates for each condition. We did not try to determine this time more precisely because it is difficult to tell when exactly the true hitting movement started, but these values provide an indication of the timing of the swing of the bat. Knowing this can help interpret the influence that seeing the ball at different times has on the temporal precision. The target was at a distance of about 5 m and the camera at a distance of about 3.5 m from the ball's path. The further procedure was the same as in Experiment 2. Again, we compared the timing precision across the conditions with a repeated measures analysis of variance, and then used *t*-tests with Bonferroni correction to identify the conditions in which the subjects had significantly different precision.

# **RESULTS**

# **EXPERIMENT 1**

For each subject and condition, the fraction of balls that were touched by the bat before reaching the ground was combined with the motion of the bat to obtain an estimate of the temporal precision (bars **B** to **E** in **Figure 2**). Even in the *baseline* condition (bar **B**), the temporal precision was considerably poorer than in our previous study (bar **A**). Within the experiment, there were no significant differences between the conditions [*F*(3, 24) = 1.19, *p* = 0.33]. Performance in the *blurred* and *small ball* conditions (bars **D** and **E**) was very similar to the baseline performance, but performance in the *monocular* condition (bar **C**) looks a bit poorer. Although this could be considered to suggest that binocular vision is critical for interception, but that our study does not have enough power to demonstrate this, we do not interpret it in that manner, because closing one eye does not only remove purely binocular information (Rose, 1978; van Mierlo et al., 2011). If the two eyes give independent judgments of the ball's trajectory with a similar resolution, and the two judgments are combined optimally, then using both eyes could lead to an improvement in precision of about a square root of two (Blake et al., 1981; Simpson et al., 2009). The dashed line within bar **C** of **Figure 2** indicates the performance that one could expect from optimally combining two independent and equal estimates which each have the precision that we measured for the *monocular* condition. This value is very close to the baseline.

Since neither blurring the image nor using a smaller ball decreased the subjects' temporal precision, it is unlikely that

equally precise). Error bars are standard errors across subjects.

the retinal resolution is critical. That blurring the image did not decrease the subjects' temporal precision also confirms that binocular disparity is unlikely to be critical, because blurring the images can be expected to decrease the resolution for judging distance from binocular disparity (Schor and Wood, 1983; Watt et al., 2005). Not finding any significant differences between the conditions suggests that the ability to estimate the changing angular elevation might be critical, because judging the rapidly changing elevation does not evidently benefit from a sharper or larger retinal image. The resolution with which the angular elevation can be judged is probably limited by the precision with which the orientation of (each of) the eyes is known. On average, the bat was moving at about 18 m/s when it hit (or passed) the ball. The differences between individual subjects' mean hitting speeds in the four conditions (standard deviation of 1 m/s) were much smaller than the differences between the different subjects' hitting speeds within each condition (standard deviation of 5 m/s).

# **EXPERIMENT 2**

Only one trial could not be analyzed because the hit took place outside the image. We estimated the temporal precision for each subject, condition and release height (in the *varying speed* condition; bars **F1** to **F4** in **Figure 2**). The critical comparison is between the trials with the same release height in the two conditions (bars **F3** and **G**). Precision did not differ between these trials [*F*(1, 11) = 0.44, *p* = 0.52]. Thus, precision for trials with a given release height does not depend on whether or not such trials are interleaved with trials with different release heights. This implies that even in the *fixed speed* condition, subjects were relying on the ball's motion, rather than for instance learning to swing a fixed time after the ball appeared.

Within the *varying speed* condition, precision was lower (the standard deviation was larger) when the ball was moving faster at the time that it appeared [bars **F1**–**F4**; *F*(3, 33) = 9.53, *p* = 0.00011]. Precision for the lowest speed (bar **F1**) was significantly better than for the two highest speeds (bar **F3**, *p* = 0.0051; bar **F4**, *p* = 0.00022). There was also a tendency to hit later with respect to the ball if the ball was moving faster, but this was not significant. Such a tendency could mean that subjects relied on the velocity on previous trials to some extent (de Lussanet et al., 2001), but it could also just arise because subjects had (too) little time to hit the fastest targets. We estimate that the time between the ball coming into sight and it being hit is about 530, 450, 400, and 370 ms for drops from heights of 4.74, 5.27, 5.79, and 6.34 m, respectively. We intentionally gave subjects so little time to encourage them to use other than visual information.

# **EXPERIMENT 3**

After excluding the subject for whom most trials could not be analyzed because the hit took place outside the image, there were only two additional trials that could not be analyzed. Again, we estimated the temporal precision for each subject and condition (bars **H**, **I,** and **J** in **Figure 2**). Precision differed significantly between the conditions [*F*(2, 24) = 8.89, *p* = 0.0013]. Subjects timed the hit more precisely in the *fast* condition (bar **H**) than in either the *fast & short* condition (bar **I**; *p* = 0.0055) or the *slow & short* condition (bar **J**; *p* = 0.018). Precision was very similar in the two latter conditions that were matched in terms of the time that the ball was visible. The time that the ball was visible was not exactly the same in both conditions, because subjects hit slightly higher than we had anticipated on the basis of the previous experiments. Consequently, the time that the ball was visible was about 6 ms shorter in the *slow & short* condition than in the *fast & short* condition. Nevertheless, it is evident that the time that the ball is visible is critical, rather than the ball's speed.

#### **EXPERIMENT 4**

In this experiment 132 trials could not be analyzed because the hit took place outside the image (8%). We estimated the temporal precision for each subject and condition from the remaining trials (bars **K**–**M** in **Figure 2**). Precision differed significantly between the conditions [*F*(2, 16) = 22.2, *p* = 0.000024]. Subjects timed the hit more precisely in the *transparent tube* condition (bar **K**) than in the *early vision* condition (bar **L**; *p* = 0.00026). They also timed the hit more precisely in the *late vision* condition (bar **M**) than in the *early vision* condition (*p* = 0.022).

These results show that it is not just better to see the ball earlier, for instance in order to time the onset of the batting movement more precisely (Caljouw et al., 2004; Tresilian and Plooy, 2006). In the *transparent tube* condition, the bat's forward motion took about 240 ms. In the *early vision* condition it took about 300 ms and in the *late vision* condition about 180 ms. That the movement started later and took less time in the *late vision* condition is logical because subjects must wait for the ball to appear before really starting their movement. Why the movements started earlier and took longer in the *early vision* condition is less clear. Perhaps subjects tried to time their hit in relation to the ball disappearing.

The last 67 ms of the ball's motion were always outside the tube (this time did not differ systematically between the conditions). Since it takes at least 100 ms to adjust a movement to new visual information (Brenner and Smeets, 1997; Oostwoud Wijdenes et al., 2011), this does not affect our interpretation. In general, performance was poorer in this experiment than in the previous ones. This is probably because the ball's fall was less consistent within the tube, so that a prediction based on the visible motion within the tube was less accurate than a prediction based on visual motion outside the tube in the other experiments. The precision may also have been affected by it being more difficult to see the ball within the transparent tube. For these reasons, adjusting the on-going swing may have been exceptionally important, and the consequences of doing so therefore exceptionally clear, in this experiment.

# **DISCUSSION**

In Experiment 1 we examined the three sources of visual information that we considered to most likely underlie, and therefore limit, temporal precision in hitting a falling ball. We found that temporal precision is not limited by the retinal resolution for judging size (as proposed in McLeod and Jenkins, 1991). If that were the case, using a smaller ball or blurring the image (by making subjects near-sighted) would have resulted in poorer precision. If the (non-significant) reduction in precision for monocular viewing is really due to having two, largely independent estimates of the relevant monocular estimate when looking with both eyes, information from ocular convergence and binocular disparity is probably also not critical. This does not mean that retinal expansion and binocular information are not used in interception (Lee et al., 1983; Regan, 1997; Rushton and Wann, 1999). It just means that in our task the critical visual information is probably the changing angular elevation (in this context, it is worth mentioning that a similar lack of sensitivity to blurring the image has also been found for a more conventional batting configuration; Mann et al., 2007, 2010). Even if all three cues are always considered, if one of the three is much more precise than the others, and the three are combined in anything close to an optimal manner, only removing the most precise cue will affect the precision noticeably. If timing in our task is indeed based on changing angular elevation, the results of the monocular condition imply that elevation is judged independently for each eye (considering the orientation of the eye as well as the retinal position of the ball's image) and the two judgments are then averaged.

Experiment 2 rejected an alternative cue, learning to hit at a fixed time after the target appeared, that could have been used in the first experiment and in the previous experiments that found a high temporal precision, because in all those studies targets were dropped from a fixed height. We found that interleaving targets falling from different heights did not make any difference. At the same time, we found that the temporal precision was lower for faster balls. Experiment 3 shows that this is not directly because of the balls' speeds, but because the faster balls were visible for less time (in Experiment 2 they always appeared at the same height, but moving at different speeds). Note that this does not mean that people hit as many of the faster balls as of the slower balls. The time window for hitting the faster balls is shorter, so fewer of the faster balls were hit despite the equal temporal precision. Experiment 4 shows that it is seeing the ball longer that is beneficial, not just seeing it earlier. Thus, visual information is not only used to initiate the swing of the bat at a more precise time, but also to guide the bat during the swing, presumably primarily on the basis of the ball's angular elevation.

By plotting the raw time differences of Experiments 2–4, we confirmed that the timing errors were approximately normally distributed. This justifies the analysis that we used in Experiment 1 and in our previous study (Brenner et al., 2012). To judge how reliably the time differences are determined from the images, we compared repeated estimates for the same trials on different days, by the same person. We found a mean standard deviation of 0.7 ms (averaged across trials). Thus, the contribution of uncertainty in judging the timing from the images is negligible.

In general, performance in this study was slightly worse than in the earlier, comparable studies (McLeod et al., 1985; Brenner et al., 2012). One obvious reason for this is that in many of the current experiments the time for which the ball was visible was quite short. The condition of the current study with the longest time for which the ball was visible before reaching the position at which it was hit is the *fast* condition of Experiment 3 (it was visible for about 970 ms). This is also the condition in which the standard deviation in the timing was smallest. However, even the precision in this condition was poorer than in the former study, in which the ball was visible for 830 ms (Brenner et al., 2012).

The fact that even the best performance was poorer than performance in our former study could be a coincidence, because different subjects took part in the former study and there are considerable differences in precision between subjects, but it is also possible that directly measuring the precision, rather than inferring it from the number of hits, results in a slightly poorer value for the estimate of the precision. There are at least two reasons why measuring the precision directly could give rise to a poorer estimate of the temporal precision. The first reason is that outliers influence the standard deviation that is calculated from individual values considerably, whereas they do not influence the number of hits differently than any other misses. The second possible reason for temporal precision being worse when measuring timing directly, is that we might be overestimating the variability when we calculate the standard deviation of the individual times, because in doing so we implicitly assume that people try to hit in the same way on all trials. We tried to encourage our subjects to do so by asking them to hit a target with the ball, not just to hit the ball. Nevertheless, our subjects may have varied the speed at which they tried to hit the ball, even across identical trials, and therefore intentionally aimed for a slightly different timing relative to the ball on different trials, because if a ball is hit harder it must also be hit later in order to hit the target (see Brenner et al., 2012). A target that is hit earlier and more gently will reach the target along a more curved path.

Considering a preferred duration of the bat's forward motion of about 240 ms (as determined for the *transparent tube* condition of Experiment 4), and a minimal reaction time of about 200 ms (as determined by subtracting the average movement time from the time the ball is visible in the *late vision* condition of Experiment 4; also see Marinovic et al., 2009), we can understand the decrease in precision with ball speed in Experiment 2 (ball visible for about 530, 450, 400, or 370ms before being hit). Assuming that subjects naturally select the optimal movement time for the task, given the prevailing task constraints (Brouwer et al., 2005; Faisal and Wolpert, 2009), the above values indicate that at least for the two fastest ball speeds, subjects will have been forced to move faster than is optimal. However, in Experiment 3, subjects were more precise when the ball was visible for about 970 ms than when it was visible for about 450 ms, suggesting that it is advantageous to see the ball for some time before initiating the forward movement of the bat.

The results of Experiment 4 show that seeing the ball earlier, and therefore having more information with which to select the optimal moment to initiate the swing, is less important than seeing the ball throughout the bat's movement, probably because subjects adjust their bat's motion to that of the ball throughout the movement (Bootsma and van Wieringen, 1990; Peper et al., 1994; Caljouw et al., 2004; Brenner and Smeets, 2011). Taken together, the results of our four experiments suggest that people primarily time their hits so precisely by using the perceived changing elevation of the ball throughout the swing to adjust the bat's movement to that of the ball.

#### **ACKNOWLEDGMENTS**

We thank Lieke Hollander, Samantha van Schie, Lies van Rijsbergen, Kristine Wijdenes, Naomi de Valk, Lucas Haagen, and Christiaan van de Vecht for their help in running the experiments and conducting the preliminary analysis of the data. We thank Frans-Jozef Halkes for technical assistance.

# **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 17 February 2014; accepted: 05 May 2014; published online: 22 May 2014. Citation: Brenner E, Driesen B and Smeets JBJ (2014) Precise timing when hitting falling balls. Front. Hum. Neurosci. 8:342. doi: 10.3389/fnhum.2014.00342 This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Brenner, Driesen and Smeets. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Different roles of alpha and beta band oscillations in anticipatory sensorimotor gating

# **Verena N. Buchholz 1,2\*, Ole Jensen<sup>1</sup> and W. Pieter Medendorp<sup>1</sup>**

<sup>1</sup> Cognition and Behaviour, Donders Institute for Brain, Radboud University Nijmegen, Nijmegen, Netherlands

<sup>2</sup> Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany

#### **Edited by:**

Irene Sperandio, University of East Anglia, UK

#### **Reviewed by:**

Gregor Thut, University of Glasgow, UK Gunnar Blohm, Queen's University, Canada

#### **\*Correspondence:**

Verena N. Buchholz, Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Martinistr. 52, 20246 Hamburg, Germany e-mail: v.buchholz@uke.de

Alpha (8–12 Hz) and beta band (18–30 Hz) oscillations have been implicated in sensory anticipation and motor preparation. Here, using magneto-encephalography, we tested whether they have distinct functional roles in a saccade task that induces a remapping between sensory and motor reference frames. With a crossed hands posture, subjects had to saccade as fast and accurate as possible toward a tactile stimulus delivered to one of two non-visible index fingers, located to the left or right of gaze. Previous studies have shown that this task, in which the somatotopic stimulus must be remapped to activate oculomotor system in the opposing hemisphere, is occasionally preceded by intrahemispheric remapping, driving a premature saccade into the wrong direction. To test whether the brain could anticipate the remapping, we provided auditory predictive cues (80% validity), which indicated which finger is most likely to be stimulated. Both frequency bands showed different lateralization profiles at central vs. posterior sensors, indicating anticipation of somatosensory and oculomotor processing. Furthermore, beta band power in somatosensory cortex correlated positively with saccade reaction time (SRT), with correlation values that were significantly higher with contralateral vs. ipsilateral activation. In contrast, alpha band power in parietal cortex correlated negatively with SRT, with correlation values that were significantly more negative with ipsilateral than contralateral activation. These results suggest distinct functional roles of beta and alpha band activity: (1) somatosensory gating by beta oscillations, increasing excitability in contralateral somatosensory cortex (positive correlation); and (2) oculomotor gating by posterior alpha oscillations, inhibiting gaze-centered oculomotor regions involved in generating the saccade to the wrong direction (negative correlation). Our results show that low frequency rhythms gate upcoming sensorimotor transformations.

**Keywords: human, MEG, parietal cortex, sensorimotor, gating, reference frame**

# **INTRODUCTION**

Saccadic eye movements serve to bring objects of interest into our focus. To make these movements, the object's sensory coordinates must be converted into gaze-based oculomotor coordinates. For visually-guided saccades, this transformation is fairly straightforward because the visual and motor coordinates are the same (Andersen and Buneo, 2002). In contrast, making saccades toward something felt on the skin, e.g., to inspect the insect landed on your hand, involves a more complex transformation. In this case, the tactile information, as sensed in a body-based somatotopic frame (i.e., relative to the body's surface), must be transformed into the oculomotor representation, which depends on the position of both hand and gaze (Groh and Sparks, 1996; Ren et al., 2006; Azañón et al., 2010; Harrar and Harris, 2010). What are the neural implications?

Because the body-based somatosensory and gaze-centric oculomotor maps are lateralized in the cortical brain (Medendorp et al., 2003; Eickhoff et al., 2008), these transformations sometimes require interhemispheric remapping. For instance, when the right hand (RH) is to the left of gaze (in the left visual hemifield), the tactile stimulus is sensed in a somatosensory map in the left hemisphere, but must be remapped to an oculomotor representation in the right hemisphere. Previous studies suggest that early somatotopic processing activates the oculomotor system in the same hemisphere, occasionally even strong enough to drive a saccade into the wrong direction (Overvliet et al., 2011; Buchholz et al., 2012). To account for the integration of postural information in the sensorimotor transformation, activity must build up on the other side of the oculomotor system, and by superseding the erroneous activity, it could initiate a saccade into the correct direction. But this process takes time and delays the saccade. Therefore, it would be beneficial if the brain could anticipate the upcoming sensorimotor transformation and regulate which regions need to be engaged and disengaged (Jensen and Mazaheri, 2010), even before the stimulus arrives. So far, these anticipatory mechanisms have been identified for sensory, as well as motor processes independently, but never in the context of a sensorimotor task, requiring spatial transformations. Importantly, such mechanisms across spatial maps could not only be important for sensorimotor behavior, but could also be instrumental in supra-modal attention networks.

A mechanism that has been proposed to reflect gating is lowfrequency oscillatory activity. Relative power suppression in the alpha (10 Hz) and beta-band (18–30 Hz), which is linked to cortical excitability during sensory and motor tasks (Gilbertson et al., 2005; Romei et al., 2008, 2010; Engel and Fries, 2010; Haegens et al., 2011b; Jensen et al., 2012; van Ede et al., 2012) has been shown to correlate with various perceptual benefits, including faster and better detection in tactile and visual tasks (Thut et al., 2006; van Ede et al., 2010, 2012; Haegens et al., 2011a; Händel et al., 2011). Both rhythms show lateralization with less power in the contralateral than ipsilateral respective sensory cortex in anticipation of visual or tactile stimuli (Worden et al., 2000; Thut et al., 2006; Hanslmayr et al., 2007; Jensen and Mazaheri, 2010; Haegens et al., 2011a; van Ede et al., 2011; Bauer et al., 2012).

In all these visual and tactile studies, the low-frequency power suppression involved the same hemisphere (van Ede et al., 2011; Bauer et al., 2012). Does this mean that alpha and beta band generally play similar roles in gating? This conclusion could be premature. In these studies, tactile tasks were typically performed with the hands in natural, uncrossed position, while the visual tasks were tested with gaze fixating straight ahead. This means that the power modulations, which were observed in the same hemisphere, were not functionally dissociated in terms of the reference frame that they deploy.

In the present study we used a crossed hand position to examine the lateralization profiles of alpha and beta band power while subjects anticipate the tactile remapping for a saccade. Recently, we reported alpha and beta band power modulations in bodyand gaze-centered reference frames induced by a tactile stimulus for a saccade (Buchholz et al., 2011). Here we test whether these oscillations also prepare the brain for upcoming tactile remapping in these different frames, thereby setting a gate for sensorimotor behavior at different stages of the sensorimotor transformation.

Under continuous recording of magnetoencephalography (MEG), human subjects executed speeded saccades to tactile stimuli (predictively cued with 80% validity) for which correct saccades require interhemispheric remapping. If alpha and beta oscillations play a role in anticipating tactile remapping for saccades, their modulations should (1) not only be evident in body-based somatotopic but also in gaze-centered oculomotor structures, even if the prediction is probabilistic; (2) take position of the hand relative to gaze into account; and (3) facilitate tactile remapping for saccades.

# **MATERIALS AND METHODS**

## **PARTICIPANTS**

Twenty-two subjects (age range 19–50 yrs, 12 female, 3 left handed), free of any known sensory, perceptual, or motor disorders, volunteered to participate in the experiment. All subjects provided written informed consent according to institutional guidelines of the local ethics committee (CMO Committee on Research Involving Human Subjects, region Arnhem-Nijmegen, the Netherlands).

# **SETUP**

Participants sat in the MEG system that was placed in a magnetically shielded dark room. They wore ear tubes attached to earplugs for auditory instructions. Their elbows were resting on platform in front of them. Forearms were crossed at the level of the wrists and supported by a wooded board, pitched away by 30◦ relative to the subject's body. In this configuration the hands were about 25 cm in front of the body. The index fingers were stretched. Due to the crossed hands posture, the two index fingertips were positioned 10 cm contralateral relative to the body midline (i.e., the sagittal plane).

One fiber optic light (Omron e3x-na, GB) was located just above the center between the two hands and served as a fixation point. Subjects viewed this light with a comfortable, slightly downward gaze direction.

We induced a tactile stimulus by means of electrical stimulation (single pulse, duration 200 µs) of the nerve endings in the skin of either index fingertip. The simulation was applied using two constant-current high voltage stimulators (Digitimer Ltd., Hertfordshire, UK). Stimulus intensity was set beyond individual perceptual threshold, but below pain threshold. Stimulus levels were adjusted during the experiment to account for adaptation effects of the tactile sense.

Continuous MEG data were recorded using a whole head system with 275 axial gradiometers (Omega 2000, CTF Systems Inc., Port Coquitlam, Canada). Head position relative to the sensor array was measured using localization coils fixed at anatomical landmarks (nasion, and left and right ear). Horizontal and vertical electrooculograms (EOG) were recorded using electrodes placed below and above the left eye and at the bilateral outer canthi. Impedance of all electrodes was kept below 5 k. During the experiment, eye recordings were continuously inspected to ensure the subject was vigilant and performed the task correctly. MEG and EOG signals were low-pass filtered at 300 Hz, sampled at 1200 Hz, and then saved to disk.

Structural full-brain MRIs were acquired with a 1.5 T Siemens Sonata scanner (Siemens, Erlangen, Germany) using a standard T1-weighted scan sequence (*FA* = 15◦ ; voxel size: 1.0 mm inplane, 256 × 256, 164 slices, *TR* = 760 ms; *TE* = 5.3 ms). These anatomical MRIs were recorded with anatomical reference markers at the same locations as the head position coils during the MEG recordings. The reference markers served alignment of the MEG and MRI coordinate systems.

# **EXPERIMENTAL PARADIGM**

Subjects performed a speeded response task in the dark, in which they had to saccade toward a tactile stimulus, delivered to one of the invisible index fingertips. Each trial began with the presentation of a high or low pitch tone, indicating with 80% validity which index finger was to be stimulated. Prior to the experiment, subject learned this relationship, which was counterbalanced across subjects. While the subject fixated centrally, after a 1.3–1.6 s interval the stimulus was delivered. Subjects were instructed to change their gaze as fast and accurate as possible to the invisible target location. After a brief fixation, the auditory cue of the next trial, instructed them to return to central fixation again. Subjects performed 10 blocks of 100 trials each, in which target locations were pseudo-randomly interleaved. Each trial lasted for 3000– 3300 ms. A brief rest was provided between the blocks during which the subjects could move their hands and eyes freely.

Thus the paradigm contains valid trials, in which the actual tactile stimulus location matches the expected location, and invalid trials, in which the actual tactile stimulus location is diametrically opposite from the expected location. **Figure 1** illustrates the conditions of the paradigm, which are defined by the location of the target relative to the body (left vs. right hand) and cue validity (valid vs. invalid). That is, the location of the potential tactile target could be represented to the body, Left hand (LH) vs. right hand (RH), or alternatively, right or left relative to gaze. Due to the crossed hands posture, the hemisphere contralateral to the hand is ipsilateral to the target relative to gaze, and vice versa.

The present paper is based on the well-accepted notion in the literature that alpha and beta suppression in rolandic and posterior regions reflect cortical increased excitability (Gilbertson et al., 2005; Romei et al., 2008, 2010; Engel and Fries, 2010; Haegens et al., 2011b; Jensen et al., 2012; van Ede et al., 2012). Therefore, by dissociating relative suppression in the hemisphere contralateral to the hand or contralateral to the target in gaze coordinates, we can distinguish between gaze- and body-centered reference frames in the regions that anticipate the tactile remapping for the saccade.

#### **BEHAVIORAL ANALYSIS**

Trials were rejected if subjects broke fixation 500 ms prior to stimulus presentation, as identified by semi-automatic analysis. On average, 74 ± 33 trials out of 1000 were excluded from further analysis. Supporting the effectiveness of this rejection method, all the reported effects show a topographic distribution that is inconsistent with residual saccadic eye movement contamination, as it is described extensively for (micro-) saccades in Carl et al. (2012) for MEG data. Of the included trials, saccade behavior was classified as "correct" when subject responded with a saccade into the correct direction after stimulus presentation. Trials were characterized as "error trials" when the saccade was initiated into the wrong direction, even when corrected during the movement. Trials that were classified as premature or too slow due to lack of subject alertness (RTs < 50 or > 450 ms, respectively) or trials in which subjects did not made a saccade at all were excluded. Based on these criteria, per subject 606 ± 45 valid trials, and 130 ± 14 invalid trials were correctly performed. We determined the reaction time of these correct saccades using a computer algorithm that detects a two degrees difference to fixation values, on a trial by trial basis.

## **MEG DATA ANALYSIS**

Open source Fieldtrip software<sup>1</sup> (Oostenveld et al., 2011) was used to analyze the MEG data. Planar gradient estimation was calculated from the axial gradiometer signals using the nearestneighbor method described by Bastiaansen and Knösche (2000) to simplify interpretation of the sensor-level data. With this conversion, the maximal signal is located above the source

**FIGURE 1 | Experimental design.** Subjects adopted a crossed hand posture, with the index fingers each at 10 cm distance from straight ahead. They had to fixate centrally, at a dim light between the two hands. Hands were resting on a tilted support, such that fixation was only slightly downward. Subjects had to saccade as fast and accurate as possible toward the tactile stimulus presented to the invisible fingertip. A tone cued with 80% validity the side of stimulation, such that subjects could anticipate the location of the sensory stimulus.

(Hämäläinen et al., 1993). The sum of the calculated horizontal and vertical planar MEG field gradients was computed to obtain the power at each virtual planar gradiometer location. Semiautomatic artifact rejection was done, rejecting high noise levels in MEG data by identifying outliers when calculating variance per trial.

Low frequency analysis (2–40 Hz) was computed based on a Fourier approach, applied to the 500 ms interval before stimulation and a Hanning taper, resulting in a spectral smoothing of approximately 3 Hz. Frequency bands of interest were the alpha band (10 ± 2 Hz) and the beta band (18–30 ± 2 Hz). To reduce data dimensionality and increase sensitivity of the analysis, we defined sensor clusters of interest based on previous results on tactile remapping, as reported by Buchholz et al. (2013), overlaying somatosensory ("central") and posterior parietal ("posterior") regions.

At the sensor level, we computed the pre-stimulus changes in power in the two frequency bands comparing activity in the contralateral and ipsilateral hemisphere for each condition. This analysis of pre-stimulus power was performed on a trial-bytrial basis, involving log-transformed power values of the last 500 ms preceding the stimulus in order to be as temporally close as possible to the transformation process. Both valid and invalidly cued trials were incorporated since subjects had the same expectancy during the pre-stimulus period, irrespective of trial type. To increase signal-to-noise ratio, we pooled this hemispheric difference across conditions and projected it onto a left standardized hemisphere, as in Buchholz et al. (2013). This spatially specific lateralization was compared across central and posterior sensors for both frequency bands separately with a simple *t*-test across subjects.

<sup>1</sup>http://fieldtrip.fcdonders.nl/

In a subsequent analysis, we correlated pre-stimulus activity with saccadic reaction time (SRT) of the correct saccades of the valid trials. To allow this relationship to be nonlinear, we calculated the Spearman rank correlation between log-transformed power values and SRT, only including validly cued trials. These correlation values were then Fisher *z*-transformed. Statistical effects were tested using paired *t*-tests.

To reconstruct the neural sources of the spectral components of interest at source level, we applied an adaptive spatial filtering (or beamforming) technique (Dynamic Imaging of Coherent Sources (DICS; Gross et al., 2001; Liljestrom et al., 2005)). We divided each participant's brain volume into an individually spaced three-dimensional grid using SPM8,<sup>2</sup> in which each location corresponds to a location in the regular 1 cm grid based on a brain template (International Consortium for Brain Mapping; Montreal Neurological Institute (MNI), Montreal, Canada). Then individual MRIs were warped to fit this template MRI and the template's grid. We subsequently warped the grid back to fit every subject's original MRI to obtain a grid in MNI coordinates for each subject. This procedure does not require normalization, as grid points are comparable across subjects. The individual spatial filters were computed from forward models with respect to dipolar sources at each individual grid point (the leadfield matrix) and the cross spectral density between all combinations of sensors at the frequency of interest (Nolte, 2003). This filter fully passes activity from the location of interest, while attenuating activity from all other locations (Van Veen et al., 1997). We used single-sphere head models based on individual MRIs to calculate the lead field matrix (Nolte, 2003). For every single subject, the source power was estimated relative to the source power in the other hemisphere, without the use of a baseline interval. Individual trial source power was log transformed and averaged across trials before averaging across subjects.

# **RESULTS**

#### **BEHAVIORAL ANALYSIS**

With their arms in a crossed posture, subjects performed a speeded saccade task to tactile stimuli, presented to the fingertips. Auditory predictive cues indicated with 80% validity which finger is most likely to be stimulated. For the valid trials, mean saccadic reaction time (SRT) of correct responses did not differ between stimuli presented to the left and right hand (*t*-test, *P* > 0.05). Average SRT in valid trials (256 ± 9 ms; Mean ± SD) was significantly shorter (*t*-test, *P* < 0.05) compared to invalid trials (263 ± 9 ms), i.e., the trials with the unexpected stimulus location. This validates our design, indicating that subjects used the auditory cues to anticipate the most probable stimulus location. Furthermore, percentage error trials for expected and unexpected LH stimuli were 3 and 6%; percentage error trials for expected and unexpected right hand stimuli were 4 and 9%. Given the low number of trials in the unexpected condition, in the following section, we will focus on the power modulation during the valid trials only.

# **LOW-FREQUENCY POWER MODULATIONS IN SENSORY AND MOTOR FRAMES**

By design of the paradigm, the crossed posture imposes an interhemispheric remapping of tactile stimuli between the bodybased somatosensory and gaze-centric oculomotor maps. We describe the lateralization of our two frequency bands of interest (alpha, beta) at the sensors of interest (central and posterior) during the prestimulus period. Under the assumption that relative suppression of alpha and beta oscillations reflects increased cortical excitability, and conversely that relative enhancement reflects cortical inhibition, we dissociated the hemispheric lateralization of these rhythms in terms of body-centered or gaze-centered anticipation.

**Figure 2A** shows the scalp topography of power in the beta band (averaged across 18–30 Hz) in the 500 ms prestimulus period, comparing log-transformed power when subjects were expecting a stimulus on the contralateral hand as compared to the ipsilateral hand. Thus, for the left hemisphere we compare right-hand (RH)—LH stimulation, and for the right hemisphere: LH—RH. Regions with cooler colors indicate lower power values for anticipating contralateral hand stimuli, while regions with warmer colors signify lower power values for stimuli on the ipsilateral hand. The scalp topography shows lower beta-band power for contralateral hand stimuli than for ipsilateral stimuli (cooler colors), most prominently over central regions. This is consistent with increased excitability in the hemisphere contralateral to the hand in a body-centered (somatotopic) representation format, or a decreased excitability in the hemisphere ipsilateral to the hand. To examine consistent effects across hemispheres, and improve the signal-to-noise ratio, data were combined by averaging across the two halves, resulting in a cleaner topography of the lower power for anticipated stimuli to the contralateral hand (**Figure 2B**). As shown, opposite modulations across hemispheres, which are inconsistent with either reference frame, and just reflect a general spatial bias have cancelled out. The observed lateralization was significantly different between central and posterior sensors (indicated by dots; *t* = 3.55, *P* = 0.0019). In fact, there was a clear lateralization at central sensors, but not at posterior sensors, which is consistent with sensory anticipation at central regions by beta band activity in a somatotopic reference frame.

We used spatial filtering techniques to estimate the sources underlying these anticipatory power changes, which are projected on a rendered representation of a standardized left hemisphere (**Figure 2C**). This suggests that the somatotopic pre-stimulus power modulation in the beta band originate from somatosensory areas, extending into inferior parietal cortex.

Whereas these body-centered modulations in somatosensory areas support previous findings, the crucial question here is whether the anticipation exceeds the sensory (somatotopic) level, and accounts for the transformations needed to operate at the motor level. In other words, does the brain also anticipate the gaze-centered motor representation of the potentially upcoming stimulus, taking into account the posture configuration between body and gaze? Or, in terms of topography, is there evidence for higher power values contralateral to the target in bodycoordinates (warmer colors), corresponding to lower power values contralateral to the target in gaze-coordinates?

<sup>2</sup>http://www.fil.ion.ucl.ac.uk/spm

**(D–F)** Scalp topography and source reconstructions of the alpha oscillations (10 Hz, and time −0.3 s) in the same format as A–C. CS, Central sulcus; IPS, intraparietal sulcus.

The beta band did not show any gaze-centered modulation at central or parietal sensors, i.e., relative power suppression contralateral to the saccade direction. The alpha band, however, showed a different pattern. **Figures 2D–F** plots alpha band topography when subjects were expecting a stimulus on the contralateral vs. the ipsilateral hand, in the same color format as **Figures 2A–C**. Lateralization of alpha-band power lateralization was significantly different between central and posterior sensors (*t* = −2.61, *P* = 0.01), by showing opposite modulation profiles. At the central sensors, the alpha band modulations mimic those of the beta band. At posterior sensors, mean alpha power is relatively higher when expecting contralateral compared to ipsilateral hand stimuli, consistent with a relative suppression contralateral to the target in gaze-coordinates. Furthermore, along the posterior midline, alpha band activity shows more power for left than for right stimuli, in both hemispheres, which is inconsistent with either reference frame, as seen in **Figure 2E**. Finally, the alpha band activity that is lateralized like the beta band, seems to originate from the hand region of primary somatosensory cortex and the operculum, whereas the opposite lateralization profile is observed in posterior intraparietal sulcus (pIPS; **Figure 2F**).

**Figure 3** illustrates the spectral extent of the spatially specific alpha and beta band effects. Thus, at central sensors, beta band power is relatively lower for contralateral than ipsilateral hand stimuli, whereas the alpha band shows the opposite pattern at posterior sensors. Under the assumption that suppressed alpha and beta corresponds to increased excitability (see Section Materials and Methods), the beta band modulations are consistent with increased excitability of the somatotopic areas that will be engaged. In contrast, alpha band modulations observed at posterior sensors are consistent with regulating excitability of

areas in a gaze-centered reference frame. Taken together, our data show that modulations do not only take place at the sensory level, but that the brain calculates the sensorimotor transformation in anticipation of a sensory event, modulating excitability at the level of the gaze-centered oculomotor structures.

It is important to realize that our analysis of lateralization cannot distinguish which hemisphere has caused the effect. In other words, the lower power contralateral than ipsilateral could equally be due to an ipsilateral power increase or a contralateral power decrease. In the following section, we will investigate the hemisphere-specific contributions to sensorimotor behavior, by examining the correlations with saccadic reaction time.

# **PRESTIMULUS MODULATIONS CORRELATE WITH SACCADE REACTION TIME**

If the observed power modulations indeed gate upcoming sensorimotor processing at sensory and motor stages by changing excitability of the cortical pathways, we should observe facilitating effects on saccade behavior. To test this, we correlated the prestimulus power modulations during the valid trials with the changes in reaction times of the correctly-directed saccades.

**Figure 4** demonstrates the correlation values between changes in beta band power and changes in SRT for valid trials with LH stimuli (A) and valid trials with right hand stimuli (B). In both there is a small but positive correlation between the beta power at the contralateral central area and the SRT. Consistent with the inverse relationship between beta band power and cortical

excitability, the more beta band suppression at contralateral central sensors, the higher the excitability and the faster the saccade is initiated (or the higher the beta, the slower). We pooled the data of both conditions by averaging the right-hand pattern with the mirrored pattern corresponding to left-hand stimulation. **Figure 4C** shows pooled power-SRT correlation values, in a format that renders left hemisphere contralateral and the right hemisphere ipsilateral to the stimulated hand (depicted in D). The correlation values at contralateral central sensors were significantly higher than their ipsilateral counterparts (*t* = 2.90, *P* = 0.009). This suggests that the ipsilateral hemisphere (here right) does not cause the behavioral benefits, but that the suppressed beta band power in contralateral areas is associated with expediting behavior. Correlation values at posterior sensors did not differ between hemispheres (*P* > 0.1).

Do the alpha band modulations show a similar relationship with saccade reaction time (SRT)? We performed the same analysis as for the beta band. Averaged alpha-SRT correlation values for the expected LH stimuli and right hand stimuli were small, as shown in **Figures 4E,F**. **Figure 4D** depicts the combined pattern. Correlation values differed significantly between the two hemispheres at posterior sensors (*t* = −2.32, *P* = 0.03), but not at central sensors (*P* > 0.1). Even at the source level, taking the voxel with maximum power, the central effect was not significant, ruling out that spatial summation at sensor level of central and posterior alpha sources has obscured such an effect (*P* > 0.1).

This significant difference at the posterior region was mainly caused by negative correlation values over the hemisphere contralateral to the stimulated hand. Importantly, this is the hemisphere that is ipsilateral to the direction of the required oculomotor response, and needs to be disengaged in an oculomotor reference frame. The more alpha power contralateral to the hand, and thus ipsilateral to the direction of the saccade, the shorter SRTs (or the less alpha band power the slower). These data suggest that the brain has a behavioral benefit of inhibiting gaze-centered oculomotor areas that should not become activated by premature, default intrahemispheric remapping of stimulus information.

# **DISCUSSION**

We examined alpha and beta oscillations in the brain of human subjects anticipating a complex sensorimotor mapping: speeded saccades to tactile stimuli in a crossed arm posture. Our analysis was based on the increasing evidence that suppression of these oscillations is associated with higher cortical excitability (Gilbertson et al., 2005; Romei et al., 2008, 2010; Engel and Fries, 2010; Haegens et al., 2011b; Jensen et al., 2012; van Ede et al., 2012) and a recent study about the underlying reference frames of these rhythms (Buchholz et al., 2011). Behavioral data from this task suggest that early response conflict arises due to premature remapping of the tactile stimulus toward the oculomotor structure in the same hemisphere, preceding the interhemispheric remapping that is required because of the crossed arm posture (Overvliet et al., 2011; Buchholz et al., 2012). Here, we show that the brain anticipates this remapping by presetting excitability in both somatotopic (sensory) and retinotopic (motor) reference frames, as reflected by the spatial selectivity in the alpha and beta band. The positive correlation between beta band activity in the somatosensory area and SRT is consistent with somatosensory gating by beta. Conversely, alpha band activity in irrelevant oculomotor regions correlated negatively with SRT, indicating that alpha band activity gates the sensorimotor transformation by inhibition of interfering areas. Even though the correlation values and modulations were small, they show a clear topological difference between the two frequency bands. Importantly, our results were observed with predictive cues that were valid for 80%, so they contain the risk that the sensory event happens elsewhere and movement plans have to be inhibited. This could also explain the small effect sizes in comparison with other studies without such manipulations.

The slightly shorter SRTs for expected than unexpected stimuli indicate that we successfully manipulated stimulus expectation in our paradigm. Furthermore, low error rates and the size of the effect suggest that subjects used sensory evidence to drive their response and not just executed preprogrammed responses. Conversion of unexpected tactile stimuli into gaze-centered coordinates takes more than 100 ms (Heed and Röder, 2010). Our results suggest that this remapping is expedited by expectation through anticipatory neuronal population dynamics. That is, the posterior alpha band lateralization was not the same as at central regions, in contrast to intrahemispheric co-modulations observed previously (Bauer et al., 2012). The gaze-centered modulations at posterior sensors take the current eye-hand configuration into account.

Lateralization of power in the alpha band has been observed during visual (Thut et al., 2006; Hanslmayr et al., 2007; Mazaheri et al., 2009; van Dijk et al., 2010) and tactile paradigms (Haegens et al., 2010, 2011a), linking alpha band activity to modulation of cortical excitability. Scrutiny of the alpha band at central sensors suggests that it co-modulates locally with the beta band in a somatotopic manner. Moreover, across hemispheres, alpha power at central sensors did not correlate differently with SRT. This might be surprising given previous results on (dis)-engagement of somatosensory regions by alpha oscillations (Haegens et al., 2010, 2012; Jones et al., 2010; Anderson and Ding, 2011; van Ede et al., 2012). However, some of these studies used distractors on the opposite side, suggesting a specific role of alpha oscillations in functional gating by inhibition of distractor-related activity. On the other hand, alpha band activity might also be behaviorally relevant by disengaging regions that would become co-activated due to anatomical connections between regions, for example between left and right primary sensory areas (Jensen and Mazaheri, 2010).

While the observed difference in trial-to-trial power-SRT correlations were significant, they only explained a small fraction of the variance in the respective relationships (*r* ∼ 0.05). Because these effects refer to condition specific differences in correlation values, the size of effect is not expected to be high (also for the reasons indicated above). Even though quantitative inferences based on extracranial recordings are limited due to methodological constraints (van Ede et al., 2012), the topography of these correlations and their polarity provide an essential insight from a functional perspective. They indicate that the brain computes its predictions about future events not only in the reference frame of the stimulus, but also simulates the coordinate transformation to anticipate the processing at the motor level.

Here, we did not use somatosensory distractors, but the sensorimotor transformation of our task contains early interference or competition from motor activity at the wrong side induced by sensory input that is not yet integrated with postural information (Overvliet et al., 2011; Buchholz et al., 2012). Therefore, the difference in correlation values observed between hemispheres was driven by negative correlation values ipsilateral, not contralateral, to the target in gaze-coordinates. Higher alpha band power in the oculomotor structures that should not become activated by early (erroneous) sensorimotor mapping was associated with shorter SRTs. This is consistent with an inhibitory role of alpha in the gating of the interhemispheric remapping process. That is, the default, but erroneous intrahemispheric remapping here might be prevented through inhibition by alpha oscillations.

Not only the alpha band at oculomotor regions expedited behavior in this task. We observed that the expectation of a somatosensory event leads to lateralization of beta band activity in central regions that is consistent with somatotopic anticipation, independent of posture. Furthermore, we found positive correlations between beta oscillations in S1 and saccadic reaction time. The difference in the correlation values across the two hemispheres was driven by positive correlations in the hemisphere contralateral to the hand. The lower the beta band power in S1 contralateral to the upcoming stimulus, the higher the excitability of this region, and the faster the saccade responses are initiated. This suggests that the local beta band power, rather than the balance between hemispheres, influences SRT.

In addition to earlier reports of behavioral benefits by beta band suppression on subsequent tactile processing (van Ede et al., 2011, 2012; Haegens et al., 2012), beta band modulations are also associated with eye-movement planning, spatial attention (Donner et al., 2007; de Lange et al., 2008; Zhang et al., 2008; Buschman and Miller, 2009; Gregoriou et al., 2012) and the facilitation of movements (Gilbertson et al., 2005). In the present paradigm, beta oscillations seem to gate only somatosensory processing and not the saccadic motor output by fronto-parietal regions.

Finally, could our results be simply explained by attentional modulations? According to the premotor theory of attention (Rizzolatti et al., 1987) preparing a saccade involves similar processes as orienting selective attention, regardless of whether the saccade is subsequently executed or not. We consider it entirely plausible that also spatial attention to the stimulus was involved, even though only foveal visual input was available. Although saccades were studied in our task, the observed gazecentric motor code might also be part of a supramodal spatial attention network, which is activated during attentional orienting in tactile space, without explicit eye movement planning. Indeed, previous findings indicate the use of a spatial code external to a somatotopic format during tactile attention (Kennett et al., 2001; Heed and Röder, 2010). Along these lines, tactile attention might be supported by several spatial maps in parallel, prioritizing the stimulus on multiple scales, to optimally prepare the system for multisensory inputs and flexible behavioral output. In support, a recent study by Ruzzoli and Soto-Faraco (2014) showed that stimulation of parietal cortex at alpha frequencies influenced tactile attention in external space coordinates, using a task in which no saccades were involved. From a different perspective, however, the source reconstruction here matches with previous alpha band sources during saccade planning as opposed to reach planning. Alpha band activity was confined to pIPS during saccade planning (Buchholz et al., 2011); an additional source in anterior intraparietal sulcus (aIPS) has been observed during reach planning (Buchholz et al., 2013). These observations also fit nicely with imaging work showing a gradient from anterior to posterior IPS for reaches vs. saccades and proprioceptive vs. visual targets (Filimon et al., 2009). Following this reasoning, tactile attention without saccade planning might activate an intraparietal source more anterior to what we observed here. Only future work can verify this interpretation. We believe that our results demonstrate oscillatory mechanisms that could gate remapping across regions needed for both directing attention to multimodal input and preparation of potential motor acts. Via oscillatory activity, the brain could gate information flow throughout the sensorimotor network, presetting excitability of regions in this pathway that operate with different frames of reference.

# **ACKNOWLEDGMENTS**

This work was supported by grants from the European Union (ERC-2010-AdG-269716) and the German Research Foundation (SFB936/A3); an internal grant from the Donders Centre for Cognition. Pieter Medendorp is supported by the European Research Council (EU–ERC 283567), EU-FP7-FET grant (SpaceCog 600785), and the Netherlands Organization for Scientific Research (NWO–VICI: 453–11–001).

# **REFERENCES**


rhythm in primary somatosensory cortex. *J. Neurosci.* 30, 13760–13765. doi: 10. 1523/jneurosci.2969-10.2010


bias and predicts visual target detection. *J. Neurosci.* 26, 9494–9502. doi: 10. 1523/jneurosci.0875-06.2006


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

#### *Received: 18 April 2014; accepted: 02 June 2014; published online: 17 June 2014*. *Citation: Buchholz VN, Jensen O and Medendorp WP (2014) Different roles of alpha and beta band oscillations in anticipatory sensorimotor gating. Front. Hum. Neurosci. 8:446. doi: 10.3389/fnhum.2014.00446*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Buchholz, Jensen and Medendorp. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# It's too quick to blame myself—the effects of fast and slow rates of change on credit assignment during object lifting

# **Kelene Fercho1,2 and Lee A. Baugh1,2\***

<sup>1</sup> Sanford School of Medicine, University of South Dakota, Vermillion, SD, USA

<sup>2</sup> Center for Brain and Behavior Research, University of South Dakota, Vermillion, SD, USA

#### **Edited by:**

Gavin Buckingham, Heriot-Watt University, UK

#### **Reviewed by:**

Alexandra Reichenbach, University College London, UK Frederic Danion, CNRS, France

#### **\*Correspondence:**

Lee A. Baugh, Sanford School of Medicine, University of South Dakota, Lee Medical Building, Vermillion, SD 57069, USA e-mail: Lee.Baugh@usd.edu

Although there have been substantial research efforts examining the effect of various rates of change in reaching movements, there has been little to no research devoted to this issue during object manipulation tasks. In force-field and visuomotor adaptation studies, two parallel processes have been identified: first, a fast process that adapts and de-adapts quickly is thought to enable the actor to deal with potentially transient perturbations. Second, a slower, but longer lasting process adapts if these initial perturbations persist over time. In a largely separate body of research, the role of credit assignment has been examined in terms of allotting the cause of errors to changes in the body vs. changes in the outside world. Of course, these two processes are usually linked within the real world, with short lasting perturbations most often being linked to external causes and longer lasting perturbations being linked to internal causes. Here, we demonstrate that the increases in load forces associated with a gradual increase in object weight during a natural object lifting task are transferred when lifting a novel object, whereas a sudden increase in object weight is not. We speculate that gradual rates of change in the weight of the object being lifted are attributed to the self, whereas fast rates of change are more likely to be attributed to the external environment. This study extends our knowledge of the multiple timescales involved in motor learning to a more natural object manipulation scenario, while concurrently providing support for the hypothesis that the multiple time scales involved in motor learning are tuned for different learning contexts.

**Keywords: credit assignment, object lifting, load force, rates of chage, motor learning**

# **INTRODUCTION**

Although our motor system is finely tuned to generate accurate movements when interacting with our environment, we inevitably make many mistakes on a daily basis when manipulating our surroundings. Luckily, the motor system is capable of adapting future movement based on the errors experienced in previous interactions with the world. To fully-benefit from this type of error-based learning, the underlying cause of experienced errors must be identified. For example, suppose you are playing a round of golf, and on the 10th hole your drive off the tee falls much shorter than you had predicted. When the swing is made, the sensorimotor system is capable of comparing the movement's outcome to a desired and/or predicted state. The information that results from this comparison can be used to inform the motor system that the target goal was not attained, and provides some initial information as to how the target was missed. In our golf example, this error-based learning could be used to adjust the motor commands for the following swing. Errorbased learning is well-understood, having been examined in many variants of adaptation paradigms including saccadic adaptation (Pélisson et al., 2010), reaching in force fields (Thoroughman and Shadmehr, 2000), and grip force modulation (Flanagan and Wing, 1997).

It is, therefore, apparent that our motor system is capable of adapting future movements based on errors experienced in previous scenarios. However, in the real world, there are a number of potential causes for a given motor movement that could result in an unexpected outcome. If we return to the golf example, a strong headwind or fatigue could both result in the shot falling shorter than expected, but the corrections the motor system should engage in would be different for each situation. Although the end goal of needing to hit the ball further is equivalent in both scenarios, the optimal way to achieve that goal is not. If a gust of wind is responsible, any changes in the motor plan related to the swing should be temporary. However, if general fatigue is to blame, motor plans should be adjusted for the remainder of the game. In order to maximize motor performance, assigning blame to the correct cause is essential, and is a credit assignment problem. Research has shown that when we learn new dynamics related to a movement, we are able to link learning to appropriate contextual cues. This, in turn, allows for the cause of any errors to be linked to the self, vs. the external world. For instance, aftereffects, the hallmark of motor adaptation, are commonly seen following adaptation to both visual and force perturbations in a number of tasks (Shadmehr and Mussa-Ivaldi, 1994; Scheidt et al., 2000; Krakauer et al., 2005; Smith et al., 2006). However, such after-effects can be substantially reduced through contextual cues that link the perturbation to the external world vs. the participant's body (Lackner and Dizio, 2005; Kluzik et al., 2008). In other words, if a reliable external source of a perturbation is provided to the actor, they only adjust their motor plans when in that specific context. In comparison, if no such cues are available, the error is attributed to the self, and after-effects are observed.

Recent work has also shown that errors appear to be allocated with differing time scales. Specifically, using both visuomotor and force-field adaptation, two parallel processes have been identified. A fast process that adapts and de-adapts quickly, and an aptly named slow process that adapts and de-adapts with a more gradual time scale (Newell, 1991; Shadmehr and Mussa-Ivaldi, 1994; Krakauer et al., 2005; Smith et al., 2006; Huang and Shadmehr, 2009). The benefit of a system with two (or more) processes that vary in their temporal characteristics is that rapid learning mechanisms would enable the individual to deal with potentially short-lived perturbation (such as a gust of wind), and the slower mechanism(s) could be used in situations where the source of the error is longer lasting (such as fatigue). Of course, these timescales themselves must be flexible, with research showing that they can be adjusted depending on the rate of change previously experienced (Huang and Shadmehr, 2009).

To date, there have been very few studies examining issues of credit assignment during object lifting tasks, although it is apparent that object lifting also requires solutions to the credit assignment problem. To lift an object efficiently, one must predict the weight of the object to be lifted (Johansson and Westling, 1988; Wolpert and Flanagan, 2001; Flanagan et al., 2006; Johansson and Flanagan, 2009). An efficient lift can be described as one consisting of a smooth increase in vertical load force to a level that just exceeds the predicted weight of the object. When lifting a novel object, people are quite accurate at predicting its weight, provided it falls within our long-term size-weight (Gordon et al., 1991; Flanagan and Beltzner, 2000; Mon-Williams and Murray, 2000) or material-density priors (Gordon et al., 1993; Buckingham et al., 2009). Despite this proficiency in predicting object weight, there are times when these predictions will contain errors. In order to maximize future lifting performance, the source of an error related to an incorrect initial prediction of the forces required to lift an object off a surface should be identified by the motor system. For example, if the error in lifting performance was a result of interacting with an object with an unusual size-weight relationship, sensorimotor memory can be used to adapt future lifts of the same object (Johansson and Cole, 1992; Flanagan et al., 2006; Johansson and Flanagan, 2009), or a combination of sensorimotor memory and long-term priors can be used when extrapolating to newly encountered objects (Baugh et al., 2012). However, in all of these scenarios, errors in lifting must be correctly assigned to the external environment, or to the self, appropriately.

The current study was designed to examine how the rate of change in an object's mass affects whether the experienced error in lifting performance are transferred to a novel to-be-lifted object. To address this issue, participants were asked to repeatedly lift a small cube. Unknown to the participant, the weight of that cube either increased from a weight of 400 g to a final weight of 570 g at a level below conscious perception over a series of 90 lifting trials, or suddenly increased from 400 g to the final weight midway through the experiment. After 90 lifts, all participants were then asked to lift a larger cube with an outer visual appearance that was different from the previously lifted blocks.

We hypothesized that if those participants in the gradual weight change condition interpreted the changes in object weight during the first 90 lifts to the self, they would lift the newly encountered larger block with greater lifting forces than those participants that were in the sudden weight change condition. Confirmation of this prediction would provide support for theories that posit the rate of change experienced plays a critical role in how the motor system solves the credit assignment problem.

# **MATERIALS AND METHODS PARTICIPANTS**

Thirty-nine participants (17 female; mean age 32; std. dev. 16) recruited from the University of South Dakota took part in this experiment after providing written informed consent. All participants performed the experiment with their dominant hand, as assessed by a modified Edinburgh handedness inventory (Oldfield, 1971). All experimental procedures were approved by the University of South Dakota's Institutional Review Board, and participants received financial compensation (20 USD per hour) or course credit for their time. Participants were randomly assigned to one of two experimental conditions (see below).

# **APPARATUS**

A total of 10 objects were used in this experiment. These included nine medium (216 cm<sup>3</sup> ) sized objects of identical outer appearance, varying in weight from 400 g to 570 g. A lead core was added to the center of each object increasing the weight by 4% from the previous block weight, a value known to be below the just-noticeable-difference threshold (JND) for weight estimations in hand-held objects (Brodie and Ross, 1984; Jones, 1986; Pang et al., 1991). One large cube (729 cm<sup>3</sup> ) was created with a different outer visual appearance (red vs. black), with a weight of 1354 g. The density of the small light-weight black cube (1.81 g/cm3) was chosen as it was unusually heavy for the apparent material (polylactic acid (PLA), 1.25 g/cm3) to ensure all participants were starting the experiment lifting a novel materialdensity relationship. The density of the heaviest black cube was 2.63 g/cm3, the resultant of the maximum change in mass possible within the JND threshold over the number of lifts participants performed. Finally, the density of the large red cube was also set to 1.81 g/cm3. As we have previously demonstrated (Baugh et al., 2012) when extrapolating to larger, unusually weighted blocks, weight predictions are brought down by the more stable longterm priors related to the apparent material. We anticipated that both groups (sudden and gradual) would predict that the large red block would have a lower density than the small black cube, as their estimates would be reduced by previous experiences with lifting plastic blocks. This allowed us to examine the differences in weight prediction between the two groups of participants, without having any participant over-estimate the weight of the block.

During each trial, participants were required to lift an object from a tabletop platform (**Figure 1A**) instrumented with two force sensors (Nano 17 F/T sensors, ATI Industrial Automation, Garner, NC, USA) to a height of approximately 2.5 cm, hold the object stationary for 1 s and then place the object next to the platform. The force sensors were capped with a flat rectangular surface, with a width of 15 cm and a length of 26 cm. These force sensors allowed for the precise measurement of the vertical load force applied to the object during lifting, up to the point when the object lifted off the supporting platform. Participants wore LCD shutter-glasses (Plato Technologies, Toronto, Ontario, Canada) that blocked vision during the inter-trial intervals.

# **PROCEDURE**

Participants were randomly assigned to one of two groups. One of the groups (Gradual, *N* = 20) lifted the entire range of 9 mediumsized objects 10 times in the training phase, with the object increasing to the next heavier weighted object every 10 lifts, for a total of 90 lifts. As all objects were identical in visual appearance, and the change in weight was below the JND threshold, participants were unaware of this change in object weight. The second group (Sudden, *N* = 19) completed the first half of the training phase (45 trials) with the lightest block and the second half of the training phase (45 trials) with the heaviest block. Following the 90 training lifts, both groups completed 10 lifts of the heavy red block.

The shutterglasses prevented participants from seeing the experimenter change the lifted object, and prevented any visual cues as to object weight. On all trials, the object was removed from the tabletop placed on a small table out of the participant's view, and then was either replaced or returned to the lifting surface. As this procedure was identical in both trials in which the weight of the object was changed and those when the weight remained the same, no auditory cues were available to the participant to suggest the object had been replaced in either of the two conditions.

Participants received both verbal instructions and a demonstration by the experimenter as to how to perform the lifting motion. Participants were asked to lift the test object 2.5 cm (100) off the sensor platform, hold it in mid-air for 1 s, and then place it on the tabletop. An auditory tone (500 Hz, 1 s) indicated when the participant was to begin the lift, and coincided with the shutterglasses turning translucent. A second tone (250 Hz, 1 s) indicated when the participant was supposed to place the object back on the tabletop. At the end of each trial, the shutter glasses turned opaque, preventing vision during the inter-trial interval.

# **DATA ANALYSIS**

Vertical forces from the sensors were sampled at 250 Hz. Raw force signals were low-pass filtered using a 4th order, zero-phase lag Butterworth filter with a cut-off frequency of 14 Hz, offline. A signal representing the vertical force applied to the object by the hand (i.e., the vertical load force) was obtained by subtracting the initial vertical force accounting for the weight of the object when fully supported by the lifting platform from the recorded signal. This processed signal was then differentiated with respect to time

**FIGURE 1 | Experimental apparatus and data analysis. (A)** While seated, participants lifted and replaced an object located on top of a platform instrumented with two force sensors. Shutter glasses were opaque between trials, preventing the participant from observing the to-be-lifted block being placed. **(B)** Load force function from two lifts of the large red object, in one lift (gray curves) the initial increase in load force was too low for the object weight, in the other lift (black curves), the initial increase in load force accurately reached the object weight. **(C)** Corresponding load force rate functions. Of importance, the initial peak in load force rate scaled with the initial increase in load force, which is dependent on predicted object weight.

using a 1st order central difference equation, resulting in the rate of change in the load force.

For each lift of the test objects, the first peak in load force rate and the load force associated with this peak were determined. The start of the load phase was defined as the time point in which the load force first exceeded 0.2 N (**Figure 1B**). Therefore, the first peak in load force rate (defined as a maxima followed by a decrease) had to occur after the load force exceeded 0.2 N. This threshold of 0.2 N was selected as load force values earlier in the trial are likely the result of initial finger placement on the block, rather than an obvious attempt to lift the object (**Figure 1C**). The end of the load phase was defined as the time, just before object lift-off, when load force reached within 0.2 N of the weight of the object (**Figure 1B**). Due to objects being lifted off the force-sensing platform, recording load forces after lift-off was not possible. When assessing initial predictions of object weight, such a method is adequate because the initial peak rate of change of load force occurred well before object lift-off in all of the trials examined.

When lifting objects, people tend to normalize the lift duration across object weight by scaling the load force rate, prior to object lift off, to the expected weight of the object. Further, by using a small target lift height, participants typically will reduce the load force rate so that it approaches zero at the expected lift-off time. Due to these task characteristics, the peak rate of change of load force during the initial increase in load force and the load force at the peak rate of change in load force rate are accurate reflections of the participants predicted weight of the object (Johansson and Westling, 1988; Flanagan and Beltzner, 2000; Flanagan et al., 2008).

In many object lifting studies, both vertical load forces and horizontal grip forces have been measured. This is typically accomplished by having participants lift an object via a handle instrumented with force sensors. In the present study, participants lifted the object directly off the force sensors, preventing the collection of grip force data. Justifying this approach, load force provides a more accurate measure of the participant's expected weight than grip force, because load force depends solely on object weight, whereas grip force depends on object properties not directly related to the mass, such as friction between the object and digits (Westling and Johansson, 1984). A primary advantage of the method utilized in this study is that participants directly manipulate the object, and therefore obtain a more natural lifting experience (Flanagan et al., 2008).

Data analysis focused on the first lift of the black cube as this reflects a participant's initial predictions as to the weight of the object. The last lift of the black cube was also examined, to establish no differences existed between our groups before changing to the novel large block, as these should be equivalent between the two groups of participants as they are lifting the same 547 g block immediately preceding the switch to the large block. Of critical importance were the first three lifts of the newly encountered large red block. This block was weighted to be unusually heavy for its size and apparent material at 1354 g, which allowed us to examine any differences the rate of change experienced in the previous lifting trials had on the initial weight predictions of the novel block. Following the experiment proper, participants in both the gradual and sudden groups were debriefed as to the true nature of the experiment.

# **RESULTS**

No participants within the gradual group reported sensing the object weight change during debriefing. The initial peak load force rate (PeakLFR),the load force at the initial peak load force (LF@PeakLFR) rate and the load phase duration (LPD) were submitted to a 3 (lift—first 3 lifts of the small black blocks, last 3 lifts of the small black block, and first 3 lifts of the large red block) × 2 (group—Gradual vs. Sudden) repeated-measures analysis of variance (rm-ANOVA) with lift as a within-subject factor and condition as a between subject factor. For all three measures (PeakLFR aLF@PeakLFR and LPD), a significant interaction between Lift and Condition was observed (*F*(2,74) = 5.435, *P* = 0.006; *F*(2,74) = 3.763, *P* = 0.028; *F*(2,74) = 4.395, *P* = 0.016, respectively), demonstrating the effect of lift was not consistent across our two groups of participants. There was also a main effect of lift, demonstrating that all participants adjusted their lifting forces to the weight as the presented block changed. Specifically, the heavier blocks used in later trials were lifted with greater forces when compared to the lighter blocks used in the earlier trials (*F*(2,74) = 177.305, *P* < 0.001; *F*(2,74) = 216.642, *P* < 0.001), and were associated with shorter lift durations (*F*(2,74) = 39.92, *P* < 0.001).

We expected that, following repeated lifting of the small black cubes, participants in all groups would learn to adequately predict the forces required to efficiently lift the object, as indicated by force output appropriately scaled to the actual weight of the object. To ensure this was the case, we examined the first and last lifts of the small black cubes, as these were equivalent in weight for both the Gradual and Sudden lifting groups. **Figure 2** shows that participants in both groups began the experiment with approximately equal load forces when lifting the initial training blocks (A) and efficiently increased load force up to the weight of the object, and showed no differences in lifting forces in the last lifts of the training block (B). Additionally, planned comparisons between conditions on the first lift of the training block and at the last lift of the training block revealed no differences in the PeakLFR (*t*(37) = 0.961, *P* = 0.343; *t*(37) = 1.294, *P* = 0.204, respectively), LF@PeakLFR (*t*(37) = 1.362, *P* = 0.181; *t*(37) = 0.214, *P* = 0.832, respectively), or LPD (*t*(37) = −0.661, *P* = 0.513; *t*(37) = −1.691, *P* = 0.099) (C and D; E and F; G and H ). These results demonstrate both groups of participants began the experiment without significant differences in lifting forces or lift durations and ended the training phase of the experiment without significant differences in lifting forces or lift durations.

Load force tracings of the first three lifts of the large red block for all participants can be seen in **Figure 3A**. To test our hypothesis that those participants in the gradual weight change group would lift the newly encountered red blocks with greater force than those participants in the sudden weight change condition, planned comparisons between the first three lifts of the large red block between each group were performed. Significant differences in PeakLFR, LF@PeakLFR, and LPD was found (*t*(37) = 2.223, *P* = 0.032; *t*(37) = 2.080, *P* = 0.044, respectively) (**Figures 3B–D**).

Finally, to examine the longevity of this effect, planned comparisons between the last three lifts of the large red block for the gradual and sudden groups were performed. Load force tracings of the last three lifts of the large red block for all participants can be seen in **Figure 4A**. No significant differences in PeakLFR or LF@PeaklLFR were found (*p'*s > 0.10) (**Figures 4B–D**), suggesting the observed effect was short-lived.

# **DISCUSSION**

Although temporal credit assignment has been examined in a number of different scenarios, little to no research has examined these issues during natural object lifting tasks, despite such scenarios also requiring a solution to the credit assignment problem. Specifically, in order to maximize future lifting performance with an object, the ability to accurately predict the forces necessary to lift said object is an essential component of dexterous object manipulation (Johansson and Westling, 1988; Johansson and Flanagan, 1999; Wolpert and Flanagan, 2001; Flanagan et al., 2006), and the temporal nature of the errors applied is likely to be an important factor within this prediction, as has been demonstrated in other tasks. This study is the first to our knowledge that demonstrates experience with an object lifting task is also sensitive to the rates of change in object weight. The role of the temporal nature of changes in object weight was influential in how the errors experienced during object lifting were applied to a novel lifting scenario. Further, we believe how the experienced errors were credited are related to whether the perturbation was seen as arising from the self or from the external environment. As we had predicted, we found the rate of change in object weight that the participants experienced had a strong influence on participants' weight predictions when encountering a newly presented object. Specifically, we demonstrated that those participants that experienced a scenario in which object weight was slowly increased lifted a newly encountered object with a greater initial peak in load force rate, and a greater load force at the initial peak in load force rate than those participants who experienced a faster change in object weight in the training phase. When participants were required associate a greater weight with the object to be lifted, they were required to link this learning to appropriate contextual cues for it to be used in later interactions. Due to participants lifting objects off force sensors, we chose to make the order of weight change always go from lighter to heavier in both the gradual and sudden conditions. This effectively ensured that a participant's predicted weight of the test object was either equal to or less than the actual weight, allowing accurate load force measurements to be obtained from the sensor before object lift-off occurred. However, had we made the order reversed (going from heavier to lighter), we would expect that rates of change in object weight would have the same effect as what we observed in the present study.

Recent models of temporal credit assignment provide a possible mechanism by which this linking may occur (Smith et al., 2006; Lee and Schweighofer, 2009). Under these models, fast and slow learning processes act in parallel in response to error signals, but differ in both their rates of learning and unlearning. As the name would suggest, slow learning processes are slower to adapt but also have a slower decay rate. In contrast, the fast system is quick to adapt and to de-adapt. In support of these models, research has shown that learning in a rapidly

**FIGURE 3 | Lift performance on first lifts of the large red block. (A)** Load force records as a function of time from the first three lifts of the large red block for all participants in both the sudden (black dashed line) and gradual (solid gray line) lifting conditions. **(B–D)** Mean

initial PeakLFR **(B)**, load force at initial peak in load force rate **(C)**, and LPD **(D)** averaged across participants. Vertical lines represent ±1 standard error, asterisk denotes statistical significance at the p < 0.05 level.

changing environment is affected by the temporal features of the task. Specifically, when participants made reaching movements in an environment which contained rapid changes, the decay rate of motor memories was greater than when participants were exposed to an environment with gradual changes (Huang and Shadmehr, 2009).

When examining the magnitude of the effect reported here, it is interesting to note that the increase in load forces utilized in lifting the novel large red cube is consistent with the weight change experienced by the gradual participant group during the training period. An efficient object lift typically consists of unimodal, bellshaped distribution when examining the rate of change in load force. Therefore, the initial peak in load force is scaled to the predicted object mass, with the load force at the initial peak in load force rate being approximately half of the predicted object mass (Johansson and Westling, 1984, 1988). When comparing the LF@PeakLFR between the gradual and sudden participant groups, we observed a difference of approximately 1 N, which is quite close to the 0.83 N that one would expect based on this simple relationship between load forces and predicted object weight.

It is important to note that we are not claiming that the motor system is unable to adapt to the increasing object weight when participants were exposed to a gradual change in object weight. In fact, when examining the lifting forces utilized at the end of the training session, those participants in the gradual and sudden weight change condition were applying equal lifting forces appropriately matched to the actual object weight (see **Figure 2**). Additionally, when examining the longevity of this effect, after 10 lifts of the large red cube differences between our groups in any of our measures were not present. This suggests that even though there were differences between our two experimental groups, these differences were short-lived and in both the gradual and sudden change participant groups, lifting forces were appropriately scaled for the object to be lifted. This is in congruence with previous studies showing that when lifting objects with poorly predicted weight, the motor system adapts to the actual object weight within approximately 10 trials (Flanagan and Beltzner, 2000; Grandy and Westwood, 2006; Flanagan et al., 2008).

In the present study, we did not directly assess which features of the gradual and sudden weight change conditions are used by the motor system to determine generalizability. For instance, in the sudden condition, in addition to differences in the temporal dynamics, the size of the error subjects experienced is much larger than the error experienced in the gradual condition. That is to say, that the difference between the predicted forces necessary to lift the object, and the eventual force required on a trial in which the object changed weight was much greater than the difference experienced by those participants in the gradual weight change condition. There is mounting evidence that suggests small errors affect learning in a fundamentally different way when compared to large errors (Criscimagna-Hemminger et al., 2003; Malfait and Ostry, 2004; Hatada et al., 2006; Michel et al., 2007; Huang and Shadmehr, 2009), and there is some evidence to support the neurological correlates related to error correction in these two scenarios is distinct (Criscimagna-Hemminger et al., 2010). Another important distinction between our rapid and gradual adaptation conditions is whether the subject is cognitively aware of the error. Our gradual condition was designed so that participants were unaware of the change in object weight over time, in contrast to those participants in the sudden condition. Previous adaptation work has shown that whether a participant is aware of a perturbation can change generalization patterns (Malfait and Ostry, 2004), and in some cases result in improved performance in a reaching task (Hwang et al., 2006), and in others result in decreased performance (Mazzoni and Krakauer, 2006).

Although we believe the presented results provides evidence that fast rates of change in an object lifting task are attributed to external sources of error, whereas slow rates of change are attributed to internal sources (in agreement with previous reaching work) due to the increased complexity during skilled object manipulation a number of alternative explanations warrant discussion. Firstly, it is possible that those participants in the gradual weight change condition could adjust their internal representation of object density—an external attribution. We would predict if such a process were responsible for the differences between our two groups, we would have observed much higher load forces during the first three lifts of the large red block. Extrapolating from the final density of the small black cube, the predicted weight of the large red cube would be approximately 2000 g. In opposition to this, the magnitude of the observed effect was much smaller than this value. A second alternative hypothesis could be that participants in the sudden weight change condition developed an average sensorimotor memory of object density that was utilized when extrapolating lifting forces to the large red cube. We are unaware of studies which show such an effect, and most research has demonstrated that during conditions of unpredictable object weight changes, load forces are largely correlated with the immediately preceding lift (Johansson and Westling, 1988; Forssberg et al., 1992; Gordon et al., 1994; Salimi et al., 2000). Nevertheless, as the present study was not directly designed to rule out such alternatives, some questions as to the root source of the differences in load forces between the sudden and gradual participant groups. Future research will examine the distinct roles each of these features play when the motor system attempts to assign error to motor predictions in object lifting tasks.

The presented research helps to move the examination of motor learning away from a fairly limited number of scenarios tested in the laboratory (such as reaching under perturbation) and into the more complicated realm of real-world motor control. In our day-to-day lives, we are often presented with objects that may adjust in mass with or without our knowledge, and the ability for the brain to be able to correctly attribute errors in prediction under these circumstances is critical for dexterous manipulation of our surroundings. Bilateral hemispheres and the right vermis of the cerebellum are known to become active during object lifting (Kinoshita et al., 2000; Schmitz et al., 2005) and cerebellar damage can result in precision grip deficits, especially in the coordination of grip force and load force during perturbation (Müller and Dichgans, 1994; Babin-Ratté et al., 1999; Serrien and Wiesendanger, 1999; Fellows et al., 2001; Rost et al., 2005). These results are consistent with theories that posit an internal model related to limb dynamics is implemented within the cerebellum (Wolpert et al., 1998; Blakemore et al., 2001; Wolpert and Flanagan, 2001; Kawato et al., 2003). In addtion, it has been demonstrated that the cerebellum plays a crucial role in the fast learning system and that patients with cerebellar damage may show deficits in the fast component of motor learning (Morton and Bastian, 2004, 2006; Diedrichsen et al., 2005; Smith and Shadmehr, 2005; Tseng et al., 2007). Additionally, transcranial direct current stimulation of the cerebellum can increase the rate of adaptation to a sudden perturbation, and primary motor cortex stimulation can improve retention of the perturbation (Galea et al., 2011), verifying that the cortico-cerebellar loop is involved in the formation and retention of learned adaptation. There is also mounting evidence that the fast component of motor learning shares critical resources with declarative memory, and is subject to interference affects during dual-task paradigms (Anguera et al., 2010; Keisler and Shadmehr, 2010). Much less is known about the slow component of motor learning, aside from the fact that it is likely a distinct process and may be related to the same anatomical substrates as procedural memory (Keisler and Shadmehr, 2010). Future research would be well-served by attempting to further dissociate these two timescales of motor learning at the neuronal level.

# **REFERENCES**


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 28 April 2014; accepted: 08 July 2014; published online: 29 July 2014*.

*Citation: Fercho K and Baugh LA (2014) It's too quick to blame myself—the effects of fast and slow rates of change on credit assignment during object lifting. Front. Hum. Neurosci. 8:554. doi: 10.3389/fnhum.2014.00554*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Fercho and Baugh. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

# The influence of expertise on brain activation of the action observation network during anticipation of tennis and volleyball serves

*Nils Balser <sup>1</sup> \*†, Britta Lorey1,2†, Sebastian Pilgramm2, Tim Naumann1, Stefan Kindermann1, Rudolf Stark2, Karen Zentgraf 2,3, A. Mark Williams <sup>4</sup> and Jörn Munzert <sup>1</sup>*

*<sup>1</sup> Institute for Sport Science, University of Giessen, Giessen, Germany*

*<sup>3</sup> Institute of Sport and Exercise Sciences, Westfälische Wilhelms-University of Münster, Münster, Germany*

*<sup>4</sup> Centre for Sports Medicine and Human Performance, Brunel University London, London, UK*

#### *Edited by:*

*Simona Monaco, York University, Canada*

#### *Reviewed by:*

*Denise Henriques, York University, Canada Roger D. Newman-Norlund, University of South Carolina, USA Pietro Pietrini, Azienda Ospedaliero Universitaria Pisana, Italy*

#### *\*Correspondence:*

*Nils Balser, Institute for Sport Science, University of Giessen, Kugelberg 62, 35394 Giessen, Germany e-mail: nils.balser@ sport.uni-giessen.de*

*†These authors have contributed equally to this work.*

In many daily activities, and especially in sport, it is necessary to predict the effects of others' actions in order to initiate appropriate responses. Recently, researchers have suggested that the action–observation network (AON) including the cerebellum plays an essential role during such anticipation, particularly in sport expert performers. In the present study, we examined the influence of task-specific expertise on the AON by investigating differences between two expert groups trained in different sports while anticipating action effects. Altogether, 15 tennis and 16 volleyball experts anticipated the direction of observed tennis and volleyball serves while undergoing functional magnetic resonance imaging (fMRI). The expert group in each sport acted as novice controls in the other sport with which they had only little experience. When contrasting anticipation in both expertise conditions with the corresponding untrained sport, a stronger activation of AON areas (SPL, SMA), and particularly of cerebellar structures, was observed. Furthermore, the neural activation within the cerebellum and the SPL was linearly correlated with participant's anticipation performance, irrespective of the specific expertise. For the SPL, this relationship also holds when an expert performs a domain-specific anticipation task. Notably, the stronger activation of the cerebellum as well as of the SMA and the SPL in the expertise conditions suggests that experts rely on their more fine-tuned perceptual-motor representations that have improved during years of training when anticipating the effects of others' actions in their preferred sport. The association of activation within the SPL and the cerebellum with the task achievement suggests that these areas are the predominant brain sites involved in fast motor predictions. The SPL reflects the processing of domain-specific contextual information and the cerebellum the usage of a predictive internal model to solve the anticipation task.

**Keywords: sports-related anticipation, motor expertise, cerebellum, superior parietal lobe, functional magnetic resonance imaging**

# **INTRODUCTION**

One can think of many different situations where it is a crucial skill to anticipate what is going to happen next. For example, a car driver has to anticipate whether a person approaching a pedestrian crossing is going to cross the street or not, surgeons have to be aware of the upcoming actions of their colleagues in the operating theater, whereas a goalkeeper in soccer has to identify the shoot direction of a penalty taker as soon as possible. For the last example, researchers have shown that the ability to anticipate the effect of the observed actions is paramount to successful performance (Savelsbergh et al., 2002; Williams et al., 2011). Fast ball sports, like tennis or volleyball, provide perfect tasks to investigate the processes underlying the anticipation of action effects as well as the influence of the athlete's prior perceptual and motor experience. In these kinds of sports, one can find many situations where athletes are under enormous time pressure and have to decide on an appropriate response even before the opponent has finished his/her action, as can be seen during the tennis return of serves with above 200 km/h for example (Williams et al., 2011). Over the last few decades, numerous researchers have shown that experts outperform novices when anticipating their opponents' actions (e.g., Singer et al., 1996; Abernethy et al., 2001; Rowe and McKenna, 2001; Williams et al., 2002; Cañal-Bruland et al., 2011; for a review, see Williams et al., 2011). The results indicate that experts rely on information visually conveyed by the kinematics of their opponent's action ahead of a key event such as ball-racket or ball-foot contact, (Abernethy and Russell, 1987; Aglioti et al., 2008; Huys et al., 2008; Williams et al., 2009; Urgesi et al., 2011).

*<sup>2</sup> Bender Institute of Neuroimaging, University of Giessen, Giessen, Germany*

On the neural level, the action observation network (AON) is supposed to play a crucial role in the perception of another person's action. This network comprises all brain areas that are activated by the mere observation of actions (Cross et al., 2009). A meta-analysis of 104 studies revealed enhanced activation during the observation of hand movements in the inferior frontal gyrus (BA 44/45), the dorsal premotor cortex (dPMC), the inferior parietal cortex (IPL), the superior parietal cortex (SPL), the inferior parietal sulcus (IPS), the primary somatosensory cortex (SI), the posterior medial temporal gyrus (pMTG), the fusiform face/body area (FFA/FBA), and the visual area V5 (Caspers et al., 2010). Furthermore, an activation of the cerebellum during action observation has been reported by numerous researchers (Buccino et al., 2004; Gallagher and Frith, 2004; Gazzola et al., 2007; Gazzola and Keysers, 2009; Pilgramm et al., 2010; Molenberghs et al., 2012; Balser et al., 2014). These data indicate that the cerebellum is part of the AON as well (Calvo-Merino et al., 2006). The AON, however, seems to be a dynamic and experience-related system. In this regard, Calvo-Merino et al. (2006) examined male and female ballet dancers who were observing gender-specific dance videos. They found stronger activation in the cerebellum and other areas within the AON, namely the dPMC and the IPS, when dancers saw dancing steps from their own motor repertoire compared to moves of the other gender with whom they had only visual familiarity. These results indicate that motor expertise has an influence on the neural processes in the cerebellum and the whole AON.

One of the various functions that are discussed for the AON is the anticipation of the consequence of an action (Gazzola and Keysers, 2009; Zentgraf et al., 2011) which might be the next action step or the environmental effect of an action. Thus, activation within this network is associated with anticipation in everyday actions (Stadler et al., 2012; Avenanti et al., 2013) and in sports-related actions (Wright et al., 2010, 2011; Abreu et al., 2012; Bishop et al., 2013; Balser et al., 2014). The specific role of the SPL and the cerebellum during an anticipation task was reported in a study conducted in our laboratory (Balser et al., 2014). During anticipation, tennis experts showed an enhanced activation in IFG and SPL, as well as a strong activation increase in numerous parts of the cerebellum, more precisely in Crus I, Crus II, Lobule VII and Lobule VIII. Furthermore, the data revealed that the neural activation of the SPL and parts of the cerebellum co-varies linearly with anticipation performance. The latter results indicate that posterior parietal and cerebellar areas of the AON are actually involved in the anticipation of action effects, as the performance-related activation increase was specific to these areas (Balser et al., 2014). A potential role of the SPL during action prediction is the storage of internal models and perceptual-motor representations (Winstein et al., 1997; Wolpert et al., 1998a; Miall, 2003; Rizzolatti and Matelli, 2003). As posterior parietal areas, the cerebellum is described as a principal brain structure for the storage of internal forward models that predict the outcome of actions (Wolpert et al., 1998b; Imamizu et al., 2000; Bastian, 2006; Miall and King, 2008; Synofzik et al., 2008).

One of the shortcomings that apply for most of the previous studies examining expert-novice differences is related to the population investigated. In many studies experts are compared to novices that did not only fail to exhibit expertise in the particular sports that is investigated, but that also differ in principle with respect to any anticipation experience (Wright et al., 2010, 2011; Abreu et al., 2012; Balser et al., 2014). In this case, differences between experts and novices could be caused many different factors such as unfamiliarity with the task which requires attentional resource allocation, decision-making under time pressure etc. In the present study, these problems have been addressed by comparing expert athletes from two different sports that both imply anticipation expertise, but expertise only for a specific sport. Therefore, this study differs markedly from prior studies as we examine two different expert groups during the anticipation of an opponent's action in tennis and volleyball to better understand the role of the AON and of the cerebellum. This approach allows us to examine in a within-subject design whether anticipation of action effects recruits areas of this network depending on the very specific representations of the observed movement in the subject's personal motor repertoire.

We applied a 2 × 2 design with two different expertise groups (between-subject condition: tennis experts vs. volleyball experts) anticipating serves from two types of sports (within-subject condition: tennis serves vs. volleyball serves). All participants watched video clips of serves in their particular sport of expertise as well as in the sport with which they had only little experience. Thus, we compared two expert groups who both had exceptional anticipation skills in their specific domain of expertise but who were at the same time novices in the other sport. In both groups, the instruction was to anticipate the direction of the serves (left vs. right) that were occluded at the moment of ball–racket or ball-hand contact respectively. Based on prior studies on perceptual-motor representations, we expected stronger activation in areas of the AON and the cerebellum in athletes with high expertise compared with novices. Second, we expected a performance-dependent activation increase in motor experts which co-varies with the task performance within the AON that were suggested to contain welldefined perceptual-motor representations. Likely candidates are posterior parietal and cerebellar structures, as these areas are associated with the storage of internal models that support predictive motor control.

# **MATERIALS AND METHODS**

## **PARTICIPANTS**

All thirty one participants were right-handed according to the Edinburgh Handedness Inventory (Oldfield, 1971). They had normal or corrected to normal vision and had not reported any history of psychiatric or neurological disorders or current use of psychoactive medication. The sample consisted of 15 tennis experts (8 female, mean age = 23.87, *SD* = 5.26) and 16 volleyball experts (8 female, mean age = 25.69, *SD* = 4.19). All thirty one experts were playing in one of the four highest level leagues in Germany in their respective sport and had experience only at a recreational level in the sport in which they were not an expert. Tennis experts had played an average of 461 (*SD* = 222) tournament matches in a mean time period of 16.67 (*SD* = 5.94) years, volleyball experts had a mean experience of 12.69 (*SD* = 5.33) years and 343 (*SD* = 215) matches. Both groups did not differ significantly in any of the reported characteristics. Participants were paid and gave their informed written consent in accordance with the Declaration of Helsinki. The study was approved by the local ethics committee (LEK FB06, 2011–0026) at the lead institution.

# **STIMULI**

Participants observed 128 stimulus videos with a duration ranging from 2.9 to 4.6 s. Half of them showed tennis and volleyball serves performed by a male and a female right handed model from each sport that were playing on the same level as the corresponding expertise group in our study. The tennis as well as the volleyball serves were all stopped at ball-racket or ball-hand contact respectively. For the videos of the tennis serves, the camera was placed right before the baseline at a position that is typical for a player waiting to return the opponent's serve (cf. **Figure 1A**). To simulate the situation of a volleyball player waiting to receive an opponent's serve, for the volleyball serves the camera was positioned 6 m behind the net in the middle of the field (cf. **Figure 1B**). One half of the 32 video clips from each sport showed serves to the left-hand corner and one half showed serves to the right-hand corner of the volleyball field or to the right service box of the tennis court respectively. The remaining 64 video clips displayed the two models of both sports bouncing a tennis ball with their racket respectively a volleyball with their right hand standing at the baseline (cf. **Figures 1C,D**). All stimuli were recorded using a Basler avA 1600—50 gc (Basler AG, Ahrensburg, Germany) video camera with a sampling rate of 35 fps.

The 128 video clips were presented at a resolution of 1024 × 768 pixels with a PC running Presentation software (Version 12.9, Neurobehavioral Systems, Albany, USA) and projected onto a screen behind the scanner so that the participants could watch them via a mirror attached to the head coil (visual field 188 mm in the horizontal and 168 mm in the vertical plane, rectangular aperture; visual angle approximately 18◦ horizontal and 11◦ vertical).

# **TASK**

Participants had to respond to four different conditions. In the *Tennis Anticipation* condition, they watched tennis serves and were asked to anticipate the direction of the observed serve and subsequently indicate the perceived flight direction of the ball. In the *Volleyball Anticipation* condition, participants watched volleyball serves with the same instruction. In both anticipation conditions, the response was given by pressing the left or right button on a two-button response box. The left button indicated a

a volleyball serve (*Volleyball Anticipation* condition). All serve sequences

ball flying to the left-hand corner and the right button a ball flying to the right-hand corner. To control for effects due to visual stimulation and the observation of biological movements, we added a *Tennis Observation* and a *Volleyball Observation* conditions including the same two models in the same visual setting without any instruction for explicit anticipation. The task in these two observation only conditions was to observe the models bouncing the ball with their racket or their hand respectively and to press the left or right button immediately after the video. The instruction text indicated which button to press before each video. All responses in this study included motor reactions after the respective observation condition. The ratio of correct left and right reactions was balanced across all four conditions.

# **PROCEDURE**

Participants were given instructions for the experimental conditions illustrated with sample videos and figures. Before the start of the fMRI experiment, participants completed a short training session with two videos for each experimental condition to ensure their full understanding of the tasks. These videos were not used in the fMRI session. While lying in the scanner, participants had to complete 128 trials resulting in a total duration of 34 min for the whole experiment. The order of the trials was randomized for each participant. Every trial started with a black screen for 1 s, an instruction for 3 s and a fixation cross for another 5 s. The following presentation of the video sequence lasted 2.9–4.6 s. The screen turned blank instantaneously after the video presentation. The participants were instructed to give their response as quickly as possible by pressing the left or the right button on the response box with the index and middle finger of their right hand. When a button was pressed, the given response was displayed on the screen for the rest of the available response time (3 s). During the whole experiment, participants did not receive any feedback on their performance.

## **BEHAVIORAL DATA ACQUISITION AND ANALYSIS**

In each of the four experimental conditions, both correct answers and response times (defined as the time between the end of the video stimulation and the button press) were analyzed with SPSS (Version 19, IBM, Chicago, USA). To investigate the influence of expertise on the number of correct responses, a 2 × 2 mixed ANOVA with Anticipation task (*Tennis Anticipation* vs. *Volleyball Anticipation*) as repeated measures within-subject factor and Domain of expertise (tennis experts vs. volleyball experts)

condition).

as between-subject factor was performed. The same computation was employed for the response times. Additionally, *t*-tests within each group assessed whether the number of correct responses in the *Tennis* and the *Volleyball Anticipation* condition were significantly above chance level.

# **fMRI DATA ACQUISITION AND PREPROCESSING**

The fMRI data were acquired using a 1.5 Tesla whole body scanner (Siemens symphony, Erlangen, Germany) with a standard head coil. The structural images consisted of 160 T1-weighted sagittal images (slice thickness = 1 mm, *TR* = 1.99 s, *TE* = 4.18 ms, field of view = 250 × 250 mm, base resolution = 256 × 256, orientation = sagittal). During the experiment, a total of 816 T2∗ weighted images were collected using a gradient echo-planarimaging sequence (number of slices = 25, slice thickness = 5 mm, gap = 1 mm, *TA* = 100 ms per slice, *TR* = 2.5 s, *TE* = 55 ms, flip angle = 90◦, field of view = 192 × 192 mm, matrix size = 64 × 64). The axial slices recorded during the EPI sequence were oriented parallel to the AC–PC line. The onsets of the video clips were jittered within an interval between ± ½ TR to realize a better sampling of the HRF function.

Functional data were processed and analyzed using SPM8 (Wellcome Department of Cognitive Neurology, London, UK). The 816 volumes were realigned and unwarped, slice-time corrected, and normalized into Montreal Neurological Institute (MNI) space. Finally, data were smoothed with 9-mm Gaussian isotropic filter as recommended by Worsley (2007). Furthermore, a movement correction was employed to reduce the impact of rapid head movements by the usage of in-house software. The detection of outlier volumes was based on a comparison of each volume with its two neighbors in a motion-corrected time series. This procedure was done by calculating the mean squared differences to the previous and the next volume. The smaller difference was used as the outlier score for each volume. Scores were thresholded using Hubert and van der Veeken's (2008) method of calculating a skewness-corrected interquartile range. To threshold outlier scores, the range was multiplied by 1.5 and added to the 75th percentile. Later on, the correction of outlier volumes was done during the first-level analysis by the usage of an additional regressor for each odd volume.

For the cerebellar data, a specific normalization method was applied to allow a more accurate localization of activation within the small structures of the cerebellum. Because of the low contrast within the cerebellum in the 152 ICBM template (MNI space), a standard whole-brain normalization as used in SPM8 leads to a large spatial variance between participants (Diedrichsen, 2006). Therefore, we used the template of the SUIT toolbox for SPM8 (Version 2.5.3, Institute of Cognitive Neuroscience, London, UK), which is based on the average cerebellar anatomy of 20 participants. This procedure preserved the fine details of the cerebellum and improved the intersubject alignment compared to the standard normalization (Diedrichsen, 2006). In a first step, the automatic isolation algorithm provided by the toolbox was used to segregate the cerebellum and the brainstem. If necessary, the isolation maps were corrected manually based on anatomical information and were then normalized to the SUIT template via a nonlinear transformation. The resultant deformation maps were used to normalize the functional images of each participant. Contrary to the whole brain data, in which normalization and the ensuing smoothing were performed before the first-level analysis, in the SUIT normalization, these steps were conducted after the functional data had been analyzed on the single-subject level. On the second-level, the whole-brain and the cerebellar data were analyzed in exactly the same way.

# **DATA ANALYSIS**

The first-level analysis was computed for each participant separately on the basis of the general linear model (GLM). The signal was convoluted using the hemodynamic response function (HRF). The video observation of each trial in the four conditions was covered by this HRF matching the length of the video. Functional data were high-pass filtered with a cutoff of 128 s to remove slow signal changes. The correct and incorrect trials of the four different experimental conditions (*Tennis Anticipation*, *Volleyball Anticipation*, *Tennis Observation*, and *Volleyball Observation*) as well as the instructions and the responses were entered into the model. Furthermore, six parameters resulting from the movement correction were added to the GLM as covariates. Autoregressive processing was applied to account for serial correlations.

In the second-level analysis, one-sample and two-sample *t*tests were conducted. To identify brain activation correlated with the anticipation performance irrespective of the expertise of the participants, we introduced the parameter "percentages of correct responses in both anticipation conditions" as a parameter to the contrast *Tennis and Volleyball Anticipation* > *Tennis and Volleyball Observation* for all 31 participants. To investigate the role of expertise during effect anticipation, the contrast (*Expertise Anticipation* > *Expertise Observation*) > (*Novice Anticipation* > *Novice Observation*) was analyzed with a two-sample *t*-test in both groups. In this contrast the common activation of both groups during the anticipation of serves of the own expertise sport compared to the sport the participants had no experience with was identified, whereas differences due to different stimuli were controlled by considering the control conditions (*Expertise Observation* and *Novice Observation*). **For a comparison of the tennis experts anticipating tennis serves with the volleyball experts anticipating volleyball serves, please see the Supplementary Material**. Additionally, we fed the covariate "percentages of correct responses in the expertise anticipation condition" into the contrast (*Expertise Anticipation* > *Expertise Observation*) > (*Novice Anticipation* > *Novice Observation*) to eliminate the influence of the anticipation performance in the respective expertise sport on the activation in areas of the AON identified by this contrast. More precisely, this additional regressor in the design matrix specified the subject-specific information of correct responses made during the different tasks. The respective contrast then focuses on neural activation due to expertise during an anticipation task, partialing out activation due to the correct responses made. Furthermore, in a second parametric analysis, we introduced the percentages of correct responses in the expert anticipation condition as a further covariate to the contrast *Expertise Anticipation* > *Expertise Observation* for all 31 participants to investigate whether AON activation in the expertise sport is correlated with the anticipation performance. This analysis focuses on the specific effects of the covariate as the respective parameter estimate represents the magnitude of the correlation between anticipation-specific activation and the number of correct responses made.

With respect to our research questions, we were particularly interested in brain activation within the areas of the AON, and we expected to find activation differences within these areas depending on expertise. Therefore, we examined a small-volume correction with a priori defined search volumes in the AON for all contrasts comparing the respective expertise and novice anticipation conditions of the athletes. The selection of these regions of interest (ROIs) was based on the results of Caspers et al.'s (2010) meta-analysis and included the inferior parietal lobe (IPL), the superior parietal lobe (SPL), the dorsal and ventral premotor cortex (dPMC and vPMC), the supplementary motor area (SMA), the somatosensory cortex (S1), and the inferior frontal gyrus (IFG). Because Caspers et al.'s (2010) meta-analysis did not include the cerebellum, we chose ROIs in the cerebellum that had been reported to be activated during the execution (e.g., Dimitrova et al., 2006; Schmahmann et al., 2009), the observation (e.g., Sokolov et al., 2010) and the anticipation (Balser et al., 2014) of actions. These regions were Lobules I-IV, V, VI, VII, and VIII, as well as Crus I and Crus II. The cerebellar masks were based on the probabilistic atlas of the cerebellum provided by Diedrichsen et al. (2009), whereas the masks of the cerebral cortex were based on cytoarchitectonic data (Eickhoff et al., 2005). All masks for this ROI analysis were created using FSL software (Smith et al., 2004) and included voxels with an at least 50% probability of being part of the specific regions. The statistical threshold for the ROI analysis was set at *p* = 0.05 (FWE-corrected). To examine whether the expertise and the novice anticipation condition are associated with differential attention-related processes, for the contrast (*Expertise Anticipation* > *Expertise Observation*) > (*Novice Anticipation* > *Novice Observation*), we compared activation in the frontal eye field (FEF) in both anticipation conditions in a *post-hoc* analysis. Therefore, we used 10-mm spheres around the MNI coordinates suggested by Heinen et al. (2013) (MNI coordinates right FEF: 31, 1, 58; MNI coordinates left FEF: −31, −3, 57) with the same statistical threshold (*p* = 0.05, FWE-corrected).

## **RESULTS**

#### **BEHAVIORAL DATA**

In the tennis anticipation condition, tennis experts gave correct answers on an average of 65.42% (*SD* = 10.12) of trials, while volleyball experts reported correct responses on 61.14% (*SD* = 8.46) of trials. When anticipating volleyball serves, volleyball experts had a mean accuracy score of 74.19% (*SD* = 7.76), whereas tennis experts responded correctly on an average of 68.54% (*SD* = 8.05). In both groups the number of correct responses was significantly above chance level for the anticipation of the tennis [*t*tennis experts(14) = 5.90, *p* < 0.001; *t*volleyball experts(15) = 5.26, *p* < 0.001] as well as for the volleyball serves [*t*tennis experts(14) = 8.92, *p* < 0.001; *t*volleyball experts(15) = 12.59, *p* < 0.001]. A 2 (Domain of expertise) × 2 (Anticipation task) ANOVA with repeated measures for the last factor revealed a significant interaction between both factors, *F*(1, 29) = 5.66, *p* = <sup>0</sup>.024, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.163 (with higher scores for correct anticipation in each sport for the respective expert group compared to the less experienced group), as well as a significant main effect on the Anticipation task, *<sup>F</sup>*(1, 29) <sup>=</sup> <sup>14</sup>.76, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.337 (higher scores for correct anticipation in volleyball) (cf. **Figure 2**). No significant main effect was reported for the between-subject factor Domain of expertise, *F*(1, 29) < 1, ns.

Tennis experts had a mean response time of 513 ms (*SD* = 211) in the tennis anticipation condition and 574 ms (*SD* = 174) in the volleyball anticipation condition. For the volleyball experts the average response times were 641 ms (*SD* = 146) in the tennis anticipation condition and 608 ms (*SD* = 148) in the volleyball anticipation condition. A 2 (Domain of expertise) × 2 (Anticipation task) ANOVA with repeated measures for the last factor revealed a significant interaction between both factors, *<sup>F</sup>*(1, 29) <sup>=</sup> <sup>9</sup>.56, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.004, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.248 (faster response of both expertise groups in their respective expertise sport). Neither a significant main effect for the between-subject factor Domain of expertise, *<sup>F</sup>*(1, 29) <sup>=</sup> <sup>1</sup>.<sup>80</sup> *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.190, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.058, nor for the within-subject factor Anticipation task, *F*(1, 29) < 1, ns, was reported.

In the ball-bouncing conditions (*Tennis Observation* and *Volleyball Observation*), participants were asked to press either the left or right button depending on the instruction received before each video. In both groups 99% of the responses were correct, indicating that all participants had maintained attention in the *Tennis Observation* and the *Volleyball Observation* condition during the whole experiment.

#### **fMRI DATA**

The study was designed to identify the influence of motor expertise on the brain activation during the anticipation of action effects. Based on the results of our previous study (Balser et al., 2014), we expected stronger activation in areas of the AON when participants anticipated the effects of actions within their domain of expertise. Therefore, in all 31 participants the brain activation during the anticipation in the respective expertise condition was contrasted with the condition the participants had no experience with. To eliminate the influence of the anticipation performance in the respective expertise sport on the activation in areas of

**FIGURE 2 | Mean percentage of correct responses in the Tennis Anticipation and the Volleyball Anticipation condition of the tennis experts and the volleyball experts.** Bars represent *SD*.

the AON identified by this contrast, a covariate "percentages of correct responses in the expertise anticipation condition" was introduced. In a further step, we performed two parametric analyses to investigate whether activation in areas of the AON was correlated with anticipation performance irrespective of expertise or with the anticipation performance in the expertise sports.

# *Expertise-related differences in the activation of the AON during anticipation*

Based on the results of a previous study (Balser et al., 2014), we examined the hypothesis that anticipating the effect of actions, the observer has expertise for, is correlated with stronger activation of AON areas. To identify these differences, we compared brain activation during the anticipation of serves in the respective expertise sport with anticipation in the type of sport the participants were novices for. Each anticipation condition was contrasted first with the ball bouncing condition of the same sport resulting in the contrast (*Expertise Anticipation* > *Expertise Observation*) > (*Novice Anticipation* > *Novice Observation*) for all 31 participants. Because the ball-bouncing control conditions contained the observation of biological movements of the same players in the identical visual settings, the results of this contrast reflect brain activation due to expertise-related anticipation and not to the mere observation of biological motion or the button press. The within-subject ROI analysis revealed higher activation for anticipation in the experts for the superior parietal lobe (SPL), the presupplementary motor area (preSMA), as well as for broad sections of the cerebellum: Crus I, Crus II, Lobule I-IV, Lobule V, Lobule VI, Lobule VIIb, Lobule VIIIa and VIIIb, Lobule IX, and Lobule X (*p* < 0.05, FWE-corrected) (cf. **Figure 3**). The opposite contrast (*Novice Anticipation* > *Novice Observation*) > (*Expertise Anticipation* > *Expertise Observation*) did not reveal any significant brain activation for the novice anticipation condition compared to the expertise anticipation condition. When the influence of different anticipation performance scores in both sports was eliminated by introducing the covariate "percentage of correct responses in the expert anticipation condition" (*M* = 70.06%, *SD* = 9.94), the contrast (*Expertise Anticipation* > *Expertise Observation*) > (*Novice Anticipation* > *Novice Observation*) resulted in activation in the same activation sites, as well as in an additional activation within the IFG. All results are summarized in **Table 1**.

# *Performance-related differences in the activation of the AON during anticipation*

As we expected a performance-dependent activation increase irrespective of expertise sport within areas that are suggested to contain motor skill representations (e.g., posterior parietal

**FIGURE 3 | In the middle of the figure: Significant brain activation in all 31 participants for the contrast (***Expertise Anticipation > Expertise Observation***)** *>* **(***Novice Anticipation > Novice Observation***).** The blue vertical and horizontal lines indicate the slice positions. T maps were thresholded at *t* = 2.00 (*p* < 0.05, FWE-corrected). Activation is rendered on a high-resolution T1 template ("colin brain") as well as on the cerebellar SUIT

template (Diedrichsen, 2006). Upper and lower part of the figure: Mean percent signal changes and standard errors in the preSMA, the SPL, and in Lobule VI and VIIIa of the cerebellum for the contrasts *Tennis Anticipation* > *Tennis Observation* and *Volleyball Anticipation* > *Volleyball Observation*, separated for both expertise groups. The signal changes were calculated by means of the SPM toolbox rfxplot (Gläscher, 2009; http://rfxplot.sourceforge.net).


**Table 1 | Brain areas identified by the comparison of the respective expertise anticipation condition with the corresponding novice anticipation condition in all 31 participants.**

*Each anticipation condition was contrasted with the ball bouncing condition of the same sport (Expertise Anticipation* > *Expertise Observation)* > *(Novice Anticipation* > *Novice Observation). \*Same activation found when a covariate "percentages of correct responses in the expert anticipation condition" was introduced. MNI coordinates, p* < *0.05, FWE-corrected, ROI analysis, ROI masks thresholded at 50%, for all ROI masks used for this analysis see Section Data Analysis at page 4.*

areas, the cerebellum), in the current study, we introduced the parameter "percentages of correct responses in both anticipation conditions" (*M* = 67.39%, *SD* = 6.17) as a parameter to the contrast *Tennis and Volleyball Anticipation* > *Tennis and Volleyball Observation* for all 31 participants. The ROI analysis revealed that in all participants irrespective of the expertise sport a better anticipation performance in both anticipation conditions was correlated with stronger activation of the SPL (5 Ci, 7 P) and Lobule VIIIa and Crus I of the cerebellum (cf. **Figure 4A**, for a summary of the results, see **Table 2**).

To identify brain activation correlated with the anticipation performance in the expertise sport of the participants, we introduced the percentages of correct responses in the expert anticipation condition (*M* = 70.06%, *SD* = 9.94) as a parameter to the contrast *Expertise Anticipation* > *Expertise Observation* for all 31 participants. A ROI analysis of the influence of the parameter on this contrast resulted in a performance-related increase of activation in the SPL (5 Ci) (cf. **Figure 4B**, for a summary of the results, see **Table 3**).

# **DISCUSSION**

We hypothesized that the anticipation of action effects in sport experts is associated with an increased activation in areas of the AON and in the cerebellum as these areas are discussed to play a crucial role in action observation, anticipation and in motor control (Gazzola and Keysers, 2009; Wright et al., 2010, 2011; Zentgraf et al., 2011; Abreu et al., 2012; Stadler et al., 2012; Avenanti et al., 2013; Bishop et al., 2013; Diersch et al., 2013; Balser et al., 2014). Furthermore, we expected a linear performance-dependent and expertise-related activation increase in AON areas which are primarily suggested to contain perceptual-motor representations during the anticipation task. On the behavioral level, the present findings replicated previous research that both expert groups outperformed the respective novice groups with respect to the number of correct responses concerning the early anticipation of an opponent's action effects. Thus, our results are in line with numerous published reports that demonstrated an expertise effect for the anticipation performance on a behavioral level (see, for a review, Williams et al., 2011). Additionally, the analysis of the response times in both expertise groups revealed a faster response of the experts in their respective expertise sport. Such a result has already been shown by Williams et al. (2002) in a study with tennis experts and novices. Regarding to the authors, the faster anticipation of the experts in the expertise sport is a further indication for superior anticipatory abilities. We are therefore confident that we can interpret the current fMRI results as a result of specific expertise differences.

Regarding the neural level, three main findings of the present study provide support for our hypotheses. First, we show that experts across two different expertise groups in volleyball and tennis revealed an increased activation within broad areas of the AON, more precisely within the preSMA, the SPL, as well as within broad sections of the cerebellum during anticipation of action effects of an opponent in the sport in which they had

**FIGURE 4 | (A)** Brain areas showing significantly stronger activation as a function of the number of correct responses for the contrast *Tennis* and *Volleyball Anticipation* > *Tennis* and *Volleyball Observation* in all 31 participants (red marks). **(B)** Brain areas showing significantly stronger activation as a function of the number of correct responses in serve

**Table 2 | Brain areas showing stronger activation as a function of the number of correct responses in tennis and volleyball serve anticipation conditions when contrasting the anticipation of serves in both sports with the ball bouncing conditions in both sports in all 31 participants.**


*MNI coordinates, p* < *0.05, FWE-corrected, ROI analysis, ROI masks thresholded at 50%, for all ROI masks used for this analysis see Section Data Analysis at page 4.*

expertise. Second, we show that irrespective of expertise the percentage of correct responses in the anticipation conditions is associated with stronger activation in the SPL (Areas 5 Ci, 7 P) as well as in the Lobule VIIIa and Crus I of the cerebellum. Third and most important, particularly in motor experts, increasing activation of the superior parietal cortex (5 Ci) co-varies systematically with the anticipation performance during the task.

The present results underpin the notion that the AON, especially posterior parietal sites and the cerebellum are mandatory for the anticipation of action effects and were influenced by the acquired motor skills of the observer (Wright et al., 2010, 2011; Bishop et al., 2013; Balser et al., 2014). The new striking contributions to the literature are that neural activation within anticipation in the expertise sport for the contrast *Expertise Anticipation* > *Expertise Observation* in all 31 participants (blue marks). T maps were thresholded at *t* = 1.00 (*p* < 0.05, FWE-corrected). Activation is rendered on a high-resolution T1 template ("colin brain") as well as on the cerebellar SUIT template (Diedrichsen, 2006).

**Table 3 | Brain areas showing stronger activation as a function of the number of correct responses in serve anticipation in the expertise sport when contrasting the anticipation of serves in the respective expertise sport with the ball bouncing condition in the corresponding expertise sport in all 31 participants.**


*MNI coordinates, p* < *0.05, FWE-corrected, ROI analysis, ROI masks thresholded at 50%, for all ROI masks used for this analysis see Section Data Analysis at page 4.*

the cerebellum and the SPL is linearly correlated with an expert's anticipation performance and that these effects also occur when using a very conservative experimental condition as both expertise groups saw the same stimuli. Customarily, in the field of action anticipation, expertise studies compare the performance of experts in a specific domain with novices who do not exhibit any specific anticipation expertise. The present study differs markedly from prior studies. Here we compared two expert groups who both were defined by extraordinary anticipation skills in their specific domain of expertise but who were at the same time novices for the other sports. This comparison allows us to study very specific effects concerning the individual motor experience in a within-subject design. Therefore, these data conclusively support the notion that the AON as well as cerebellar areas responded to the stimuli in a way that depends on the observer's domainspecific motor expertise what suggests that anticipation of action effects recruits areas of this network depending on the very specific representations of the observed movement in the subject's personal motor repertoire. The following sections will discuss these findings and their implications in more detail.

# **PERCEPTUAL AND MOTOR EXCELLENCE IS LINKED TO ACTIVATION WITHIN THE AON DURING EFFECT ANTICIPATION**

The process of an appropriate reaction to an opponent's action outcome comprises several computations in the motor system. First, one is requested to accurately predict the consequence of the observed motor action. Second, one has to combine these predictions with the own body state. Third one has then to plan a reaction to the opponent's behavior. Especially the function of an accurate prediction corresponds well to activation within regions of the AON (Wright and Jackson, 2007; Gazzola and Keysers, 2009; Urgesi et al., 2010; Wright et al., 2010, 2011; Zentgraf et al., 2011; Abreu et al., 2012; Stadler et al., 2012; Avenanti et al., 2013; Bishop et al., 2013; Diersch et al., 2013; Balser et al., 2014). The present data revealed that both expert groups outperformed the respective novice groups with respect to the number of correct responses. These effects are accompanied on the neural level with an increased activation within the SMA, the SPL, as well as within sections of the cerebellum what is in line with broad body of literature (Stadler et al., 2011; Wright et al., 2011; Abreu et al., 2012; Bishop et al., 2013; Balser et al., 2014). For example, a recent study by Balser et al. (2014) demonstrated that tennis experts performed better than novices on different tennis anticipation tasks, with the experts showing stronger neural activation in areas of the AON, namely, the superior parietal lobe, the intraparietal sulcus, the inferior frontal gyrus, and the cerebellum. Similarly, Bishop et al. (2013) showed an expertise effect by demonstrating increased cerebellar, cingular and basal ganglia activation for experts during the prediction of the opponent's actions. The findings of Bishop and colleagues and the present results show that the perceptual, motor and cognitive superiority of an expert is clearly linked to increased activation within areas involved in action perception and motor control. On this background, a parametric analysis of the present data revealed that the activation within the Area 5 Ci of the superior parietal activation site and Lobule VIIIa and Crus I of the cerebellum are linearly associated with the anticipation performance irrespective of motor expertise. When comparing effect anticipation in the expertise sport with the observation condition, the parametric relationship between the performance and neural activation still holds for the superior parietal site (Area 5 Ci). This differential involvement of the SPL reflects the performance of motor experts in the expertise-related anticipation task: a better anticipation performance in the expertise sport is related to an increased activation within this region.

## **PARIETAL CONTRIBUTIONS TO THE ANTICIPATION OF ACTION EFFECTS**

Regarding the posterior parietal cortex, researchers have revealed over the last decade that this area is not only related to higherorder sensory analysis but also plays an important role in motor control (Fogassi and Luppino, 2005; Vesia et al., 2006). For example, it is crucial for visually guided actions. The activation of the SPL is related to the on-line control for reaching, grasping or pointing movements (Grafton et al., 1992, 1996; Culham and Valyear, 2006). In this regard, it was demonstrated that with the growing accuracy demands of an executed aiming task, neural activity within this area increases in line with increased visuomotor processing demands (Winstein et al., 1997; Fiehler et al., 2008), which suggests that the increased activation of posterior parietal sites like the SPL reflects the importance of the target representation when the planned movement comprises a target region. A further functional issue of the SPL is the storage of internal models and action representations which are mandatory for action prediction (Winstein et al., 1997; Wolpert et al., 1998a; Miall, 2003; Rizzolatti and Matelli, 2003).

In the present study, we found a performance-related activation increase in the medial section of the SPL irrespective of motor expertise as well as when comparing the anticipation of serves in the respective expertise sport with the ball bouncing condition in the corresponding expertise sport. Thus, the SPL activation is strongly related to anticipation performance in each participant and depends on the observer's domain-specific motor repertoire. It is likely that the activation within this area, which is functionally associated with visuomotor representations and motor prediction, accompanies the higher-order perceptual and anticipation skills seen in elite athletes, particularly in fast ball sports like tennis and volleyball where a precise coding of spatial information with respect to a target is required. It can be argued that in the present anticipation task, motor expertise seems to enhance the use of these specific internal perceptual-motor representations which are built up through years of training in a certain field of sports.

Another line of research demonstrated activation in the SPL when participants had to initiate movements based on prior expectations (Imamizu and Kawato, 2008). More precisely, it was concluded that the SPL associates contextual information with an appropriate internal model processed in the cerebellum to predict the consequences of an action. It can be argued that experts build up a very specific representation of the contextual framework, such as the opponent's position and its surrounding, which is strongly depending on the type of sports. Within this framework, several researchers have shown that experts improve their anticipation performance when they are provided with contextual, game-related information (Crognier and Féry, 2005; McPherson and MacMahon, 2008; McRobert et al., 2011). Thus, an alternative explanation for the SPL activation pattern within the present study could be that experts use such specific contextual information during the anticipation of their opponent's behavior what is particularly reflected by the expertise- and performance-related increase of the SPL activation.

# **CEREBELLAR CONTRIBUTIONS TO THE ANTICIPATION OF ACTION EFFECTS**

As for posterior parietal areas, neurophysiological and computational studies have demonstrated the cerebellum as a principal brain structure for the storage of internal forward models that predict action outcomes and therefore support predictive motor control (Wolpert et al., 1998b; Imamizu et al., 2000; Bastian, 2006; Miall and King, 2008; Synofzik et al., 2008). We found that besides the neural activation within the SPL the activation within the cerebellum co-varies systematically with the anticipation performance irrespective of the specific motor expertise. These results are nicely in line with our previous data which reported that activation of parts of the cerebellum co-varies with the anticipation performance irrespective of the motor expertise (Balser et al., 2014). However, the present study expands this finding as this relationship also holds for a within-subject design with two expert groups who both were defined by extraordinary anticipation skills in their specific domain of expertise but who were at the same time novices for the other sports.

It is argued that the cerebellum might house the so-called forward models (Wolpert et al., 1998b; Imamizu et al., 2000; Bastian, 2006; Miall and King, 2008; Synofzik et al., 2008) that are predictive on their part and, therefore, estimate the anticipated sensory outcome of an action (Miall and Wolpert, 1996; Wolpert and Flanagan, 2001). A recent study in cats, for example, showed that neuronal discharge in the lateral cerebellum predicts the motion of a moving external target (Cerminara et al., 2009). These data suggest a connection between a forward model, which predicts the sensory consequences of one's own actions, and a model that could predict the actions of others which has its neural substrate in the cerebellum. The authors reasoned that the measured neural discharge might be used in a predictive capacity for target interception. Extrapolating these data to the present results, it can be suggested that in both, volleyball and tennis, participants are required to predict the effect of an opponent's motion on ball trajectory (Yarrow et al., 2009) by using forward models that allow a rapid processing of incoming sensory stimuli. This offers the acting individual a clear advantage in producing a quick motor response which is mandatory in both sports.

# **DIFFERENTIAL INVOLVEMENT OF CEREBELLUM AND SPL DURING ACTION ANTICIPATION**

The present results demonstrate a differential involvement of cerebellar and superior parietal areas. Whereas the cerebellum shows a performance dependent activation increase irrespective from expertise, the superior parietal cortex shows a performance and expertise related activation increase. Thus, it seems reasonable to conceive a differential involvement of both structures in action anticipation. Imamizu and Kawato (2008) argued that the SPL associates contextual information with an appropriate internal model located in the cerebellum to predict the consequences of an action. Furthermore, it has been suggested that internal models are acquired in the cerebellum and top–down context information from the SPL to the cerebellum contributes to predictive switching between internal models (Imamizu and Kawato, 2008). We suggest that the expertise and performance dependent activation within the SPL reflects the processing of domain-specific contextual information (e.g., using a racket or not to hit the ball) and leads specifically to increased resonance in the expert's SPL. The activation of the cerebellum, however, reflects the usage of a predictive internal model to solve the anticipation task which is required for both anticipation tasks in the present setting.

## **POTENTIAL LIMITATIONS**

In the present study, we examined scenes from fast ball games that require quick responses under time pressure. The anticipation of respective action effects in tennis and volleyball include short time windows that are typical for fast ball sports but different to everyday anticipation problems. In our case, participants had to predict distal action effects of an opponent that were at the same time relevant for a selection of own motor responses. The present data, therefore, might not hold for all possible types of anticipation, like the anticipation of in-animated events (Schubotz, 2007) or the anticipation during serial prediction tasks and arbitrary stimulus-response mappings (Wolfensteller et al., 2004).

One possible flaw in the interpretation of the present data is related to the performance-related activation increase in the SPL we found when comparing the anticipation of serves with the ball bouncing condition within the respective expertise sport as well as when comparing both conditions irrespective of motor expertise.

To control for effects due to visual stimulation and the observation of biological movements, we contrasted the anticipation conditions with observation only conditions without an explicit instruction for anticipation. Although the anticipation and the observation only conditions were comparable concerning the depicted models, the sports hall background, the perspective of the camera and the fact that all conditions involved the observation of biological movements that included a ball, both conditions possibly resulted in differential attentional demands. Therefore, we cannot preclude that the posterior parietal activation is also associated with attention-related processes, as the posterior parietal areas has been shown to be involved in directing spatial attention and in disengaging and maintaining attention to visual and tactile stimuli (Posner et al., 1984; Pardo et al., 1991; Corbetta et al., 1993; Halligan et al., 2003; for a review, see Rushworth et al., 2003). However, the comparison of high expertise effect anticipation with low expertise effect anticipation [(*Expertise Anticipation* > *Expertise Observation*) > (*Novice Anticipation* > *Novice Observation*)] revealed activation in the SPL as well. In this contrast, before both anticipation conditions were compared, they were contrasted with the respective observation only condition in a first step. As prospective attention-related differences between the anticipation and the observation only conditions were supposed to be comparable in high and low expertise sport, the influence of the observation only condition concerning attention-related phenomena was minimized. Thus, activation differences in the SPL cannot be assigned to differences in the attention demand between the anticipation and the observation only conditions but to anticipation processes that are modulated by expertise. Furthermore, it has also been argued that the SPL is not the key structure in disengaging attention and further attention-related processes (Corbetta et al., 1995; Rizzolatti et al., 1997; Friedrich et al., 1998). In fact, Rizzolatti et al. (1997) state that the SPL plays a decisive role in the processing of sensory and motor signals in the context of somatosensory integration. Additionally, we examined the activation in the FEF for the comparison the expertise and the novice anticipation condition ((*Expertise Anticipation* > *Expertise Observation*) > (*Novice Anticipation* > *Novice Observation*)) in a *post-hoc* analysis. The FEF has been shown to be involved in attention-related eye movements (Bosch et al., 2013; Squire et al., 2013) and in the allocation of attention in a visual scene (Corbetta and Shulman, 2002; Heinen et al., 2013; Ronconi et al., 2014). The fact that we found no differences in the activation of the FEF between both anticipation conditions indicates that the stronger activation in the expertise sport is not due to differences in attention-related processes.

# **CONCLUSION**

We conclude that neural activation within several sections of the AON, especially within the superior parietal as well as within the cerebellar cortex, is associated with action anticipation performance in sport experts. The present data suggest that the AON, including cerebellar areas, responded to the stimuli in a way that depends on the domain-specific representation of the observed action in the subject's personal motor repertoire as well as on the achievement in this task. The present results extend the literature and findings from our previous work by using a very conservative design to show that especially neural activation within the SPL and the cerebellum is linearly associated with the task achievement, irrespective of the specific expertise. For the SPL, this relationship holds when an expert performs a domain-specific anticipation task. We consider that this activation pattern reflects that posterior parietal as well as cerebellar areas are the predominant brain sites that supposed to be involved in fast motor prediction. We suggest that the SPL reflects the processing of domain-specific contextual information (e.g., using a racket or not to hit the ball) and the activation of the cerebellum reflects the usage of a predictive internal model to solve the anticipation task.

# **ACKNOWLEDGMENTS**

The authors thank Fabian Helm, and Kristin Zimmermann for their help in stimuli recording and data acquisition; Bertram Walter for statistical support; and all participants for taking part in our study.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnhum. 2014.00568/abstract

# **REFERENCES**


Ashburner, S. Kiebel, T. Nichols, and W. Penny (Amsterdam: Academic Press), 232–236.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 25 April 2014; accepted: 11 July 2014; published online: 01 August 2014. Citation: Balser N, Lorey B, Pilgramm S, Naumann T, Kindermann S, Stark R, Zentgraf K, Williams AM and Munzert J (2014) The influence of expertise on brain activation of the action observation network during anticipation of tennis and volleyball serves. Front. Hum. Neurosci. 8:568. doi: 10.3389/fnhum.2014.00568 This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Balser, Lorey, Pilgramm, Naumann, Kindermann, Stark, Zentgraf, Williams and Munzert. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Integration of egocentric and allocentric information during memory-guided reaching to images of a natural environment

#### **Katja Fiehler <sup>1</sup>\*, Christian Wolf <sup>1</sup> , Mathias Klinghammer <sup>1</sup> and Gunnar Blohm<sup>2</sup>**

<sup>1</sup> Department of Experimental Psychology, Justus-Liebig-University, Giessen, Germany

<sup>2</sup> Canadian Action and Perception Network (CAPnet), Centre for Neuroscience Studies, Queen's University, Kingston, ON, Canada

#### **Edited by:**

Simona Monaco, York University, Canada

**Reviewed by:** Alexander Gail, German Primate Center, Germany Constanze Hesse, University of Aberdeen, UK

#### **\*Correspondence:**

Katja Fiehler, Department of Experimental Psychology, Justus-Liebig-University, Otto-Behaghel-Str. 10F, D-35394 Giessen, Germany e-mail: Katja.fiehler@ psychol.uni-giessen.de

When interacting with our environment we generally make use of egocentric and allocentric object information by coding object positions relative to the observer or relative to the environment, respectively. Bayesian theories suggest that the brain integrates both sources of information optimally for perception and action. However, experimental evidence for egocentric and allocentric integration is sparse and has only been studied using abstract stimuli lacking ecological relevance. Here, we investigated the use of egocentric and allocentric information during memory-guided reaching to images of naturalistic scenes. Participants encoded a breakfast scene containing six objects on a table (local objects) and three objects in the environment (global objects). After a 2 s delay, a visual test scene reappeared for 1 s in which 1 local object was missing (= target) and of the remaining, 1, 3 or 5 local objects or one of the global objects were shifted to the left or to the right. The offset of the test scene prompted participants to reach to the target as precisely as possible. Only local objects served as potential reach targets and thus were task-relevant. When shifting objects we predicted accurate reaching if participants only used egocentric coding of object position and systematic shifts of reach endpoints if allocentric information were used for movement planning. We found that reaching movements were largely affected by allocentric shifts showing an increase in endpoint errors in the direction of object shifts with the number of local objects shifted. No effect occurred when one local or one global object was shifted. Our findings suggest that allocentric cues are indeed used by the brain for memory-guided reaching towards targets in naturalistic visual scenes. Moreover, the integration of egocentric and allocentric object information seems to depend on the extent of changes in the scene.

**Keywords: reference frame, reaching, natural scene, allocentric information, egocentric information, human**

# **INTRODUCTION**

When reaching to a visual target in a naturalistic environment, the brain can make use of absolute or relative spatial information for reach planning. This can be formalized in terms of two broad classes of reference frames: an egocentric reference frame that represents the absolute position of an object with respect to the observer and an allocentric reference frame coding the position of an object relative to other objects in the environment (Colby, 1998). While egocentric reference frames depend on eye, head, body, etc. position and orientation, allocentric reference frames are relatively observer-invariant. It is well known that for goal-directed reaching movements, a gaze-dependent, egocentric reference frame is used preferentially as demonstrated by electrophysiological studies in monkeys (Batista et al., 1999; Buneo et al., 2002) and behavioral (Henriques et al., 1998; Medendorp and Crawford, 2002; Fiehler et al., 2011) and brain imaging studies (Medendorp et al., 2003; Bernier and Grafton, 2010) in humans.

Despite the dominance of gaze-dependent representations for reach planning, allocentric information also contributes to the encoding of reach target location. For example, visual landmarks provided during target presentation lead to an increase in accuracy and precision of reaching movements (Krigolson and Heath, 2004; Obhi and Goodale, 2005; Krigolson et al., 2007). The effect of reduced reach endpoint variability was even more pronounced when the landmarks were placed close to the reach target (Krigolson et al., 2007). If landmarks are present while participants reach to remembered targets updated in their visual periphery, the influence of gaze-dependent spatial coding has been found to decrease suggesting a combined use of egocentric and allocentric information (Schütz et al., 2013). Such combination of egocentric and allocentric reference frames is supposed to occur after the intervening saccade at the time of action (Byrne et al., 2010) and depends on heuristics for external cue stability as well as the reliability of egocentric and allocentric cues which determines the weighting in memory-guided reaching (McGuire and Sabes, 2009; Byrne and Crawford, 2010). In addition, the proximity of the landmarks and the target seems to affect reach endpoints showing systematic distortions toward the nearest landmark (Diedrichsen et al., 2004). However, this effect only occurred when landmarks were available during target encoding but not during reaching. Moreover, structured visual background placed close to the target led to more precise reaching movements than distal visual background presumably linked to the proximity of veridical target location (Krigolson et al., 2007). The use of allocentric cues in addition to egocentric representations has even been demonstrated for imagined landmarks which were not physically present during target encoding or reaching but represented a virtual straight line (Carrozzo et al., 2002). The authors argued for the use of concurrent and independent coexisting egocentric and allocentric target representations used for memory-guided reaching.

Here we set out to address a series of controversies and gaps in the literature: (1) so far, isolated visual targets together with abstract, task-irrelevant landmarks on an otherwise blank screen have been used to investigate the underlying reference frames for reaching movements. However, it is not a given that findings from such abstract studies will hold in natural situations, where we are surrounded by a vast number of visual features creating a complex visual scene; (2) moreover, previous studies (e.g., Schenk, 2006; Zaehle et al., 2007; Thaler and Goodale, 2011a,b) explicitly asked participants to use a predefined egocentric or allocentric reference to perform the task probably covering individual spatial coding strategies. Therefore, one aim of our study was to examine the contribution of egocentric and allocentric information to reaching to images of a natural scene without biasing subjects' behavior to use either one or the other reference frame; (3) it has been suggested that object proximity is an important factor biasing reach endpoint (Diedrichsen et al., 2004); we will challenge this view here; and (4) we will further test whether allocentric information influences reach trajectory planning (Burns and Blohm, 2010) vs. feedback-based control processes (Krigolson et al., 2007). Participants reached to a remembered location of an object on a breakfast table while we varied the location of the surrounding objects by applying a leftward or a rightward shift (allocentric cue). Spatial shifts were either applied to surrounding objects on the table which could be potential targets and were thus taskrelevant (local objects) or to objects in the environment which never served as a target (global objects). Since the position of gaze, head and body were kept constant, we expected no systematic reach errors if participants relied on an egocentric target representation alone. If participants represented the target with respect to other objects on the table and/or in the environment, i.e., they used an allocentric representation, we predicted reach errors which vary as a function of object shifts. We show that memoryguided reaches to images of naturalistic environments are planned using both egocentric and local allocentric information, but not global allocentric cues.

# **MATERIALS AND METHODS**

## **PARTICIPANTS**

Data were recorded from 14 participants with normal or corrected to normal vision. One subject was excluded from further analysis because of poor fixation behavior (<1% valid trials), another subject because of frequent movement onsets while the test scene was still displayed (29.2%). The final sample consisted of 12 participants (3 female; 3 left-handed, self-report) ranging in age from 20 to 37 years (mean 24 ± 4 years). All procedures were conducted in agreement with the ethical guidelines of the local ethics committee of the University of Giessen and were approved by the Queen's University Ethics Committee in compliance with the Declaration of Helsinki.

# **MATERIALS**

Participants viewed photographic color images showing a breakfast scene with six *local objects* (coffee mug, plate, espresso cooker, marmalade jar, butter dish, and egg cup) on a table that was placed in front of a white wall and three *global objects* (table [T], table cloth [C], and painting on the wall [P]) in the scene (see **Figure 1A**). The object properties are summarized in **Table 1**.

The six local objects were arranged in 18 different configurations on the table to minimize memory effects (*encoding image*). To this end, the objects were assigned to one of four possible locations in depth (8 cm, 28 cm, 48 cm, or 68 cm from the front table edge) and to a randomized horizontal position. Configurations were pseudo-randomized and fulfilled the following criteria: (i) at least one object was placed at every depth position; (ii) objects were placed with a minimum horizontal distance of 8 cm away from the edges of the table cloth in order to enable horizontal displacement on the table cloth; and (iii) <50% of each object was occluded. In addition to the encoding images, we created *test images* lacking one of the 6 local objects (= reach target). In 2/3 of the test images, local or global objects were physically displaced in the horizontal direction on the table by 8 cm either to the left or to the right (50% leftward displacement) prior to taking photographs. Due to the finite camera distance, this corresponds to different shifts on the image (and thus also on the screen), depending on the depth position of the object, i.e., whether it was located in the proximal, first medial, second medial or distal depth plane. Thus resulting visual shifts on the screen images could be 4.24◦ , 3.80◦ , 3.36◦ and 2.92◦ for proximal, first medial, second medial or distal object depth respectively. In the remaining 1/3 of the test images, the remaining objects in the scene were not shifted. In order to ensure precise and reproducible object placement in the images, a grid was projected from above on the table before the photographic image was taken with a resolution of 2048 × 1536 pixels.

In total, 342 photographic images were taken including 18 encoding images and 324 test images with 108 images without object displacement, 108 images with local object displacement and 108 images with global object displacement. Separate photographic images were taken for each target (6) in each configuration (18) and experimental condition (3; control, local and global).

# **APPARATUS**

Stimuli were presented on a 19<sup>00</sup> (40.64 cm × 30.48 cm) CRT monitor with a resolution of 1920 × 1200 pixels and a refresh rate of 60 Hz using the Psychtoolbox (Brainard, 1997) in Matlab (The Mathworks, Inc., Natick, MA, USA). Monitor/image edges

**Table 1 | Maximum height and width of objects in the scene in cm**.


were visible. Participants sat at a table with their head stabilized on a chin rest guaranteeing an eye-monitor distance of 47 cm. They performed the task in complete darkness but the use of a computer screen resulted in some limited illumination of the hand. Participants executed right arm reaches from an elevated start position placed 27 cm in front of the screen at the level of the lower screen edge. Reaches were recorded with an Optotrak Certus (NDI, Waterloo, ON, Canada) infrared marker-based motion tracking system with a sampling rate of 250 Hz using one marker attached to the fingertip. In order to control for correct fixation behavior, we also recorded eye movements using an EyeLink 1000 tracking system (SR Research, Osgoode, ON, Canada) with a sampling rate of 1000 Hz. Participants initiated the trials by a lefthand button press on a game controller located on the table in front of their left shoulder.

# **PROCEDURE**

The trial procedure is illustrated in **Figure 1**. Participants started each trial by a button press with their left hand. An encoding image containing all local and global objects was displayed on the screen until participants continued the trial with a button press on the controller. They were instructed to encode the location of the local objects in the scenes while freely moving the eyes. Participants had as much time as desired and were instructed to press the game controller with the left hand in order to pursue the trial. The encoding phase was followed by a central fixation cross that appeared on a uniform gray background for 2 s prompting participants to maintain fixation at this location until the end of the reach. Then, the test image without one of the six local objects was presented for 1 s, superimposed with a fixation cross. After the test image disappeared, the fixation cross was displayed on a uniform gray background and participants were asked to reach with their right hand to the remembered location of the missing object (= target) on the screen. Thus, reaches were performed while fixating at the center of the screen and without any visual information about the scene. Whenever participants were unsure about the location of the target, they were instructed not to reach but to continue with the next trial.

Participants performed three experimental conditions (**Figure 2**). In the *allo-local* condition, we manipulated the number of local objects shifted in the scene of the test image before reaching. In particular, 1, 3 or all 5 remaining local objects were horizontally misplaced by 8 cm (in physical space) to the left or to the right (loc1, loc3, loc5) without affecting the position of the global objects. Within one trial, objects were always shifted in the same direction. In the *allo-global* condition, one global object was shifted by 8 cm (in physical space) leftwards or rightwards by leaving the location of the local objects unchanged (gloT, gloC, gloP). In the control condition, no object shifts occurred.

Each participant completed 648 trials split up in 18 blocks consisting of 36 trials each. Before the start of the experiment, each participant completed a training block of 18 control trials. Data of each subject were recorded in three 1 h sessions on different days consisting of six blocks each.

## **DATA REDUCTION AND STATISTICAL ANALYSES**

Data preprocessing and analyses were done using MATLAB and final inferential statistics were computed in SPSS (Version 21.0). An α-level of 0.05 was used for evaluating all effects.

First, we analyzed eye tracking data in order to control for correct fixation. Trials were classified as invalid and excluded from further analyses if gaze deviated more than ±2.5◦ from the fixation location. This applied to 564 trials (7.25%). Second, reach endpoints were determined as the position where reach velocity and screen distance were minimal. Reaching endpoints in screen coordinates were then computed from camera coordinates using quaternion transformation (Leclercq et al., 2013). We excluded trials in which reach endpoints deviated more than 2.5 SD from the average reach endpoint per test image (**Figure 3B**). This resulted in removing 638 trials of the remaining trials (8.2%). In 187 trials (2.4%), subjects responded before the test image disappeared. To test memory-guided reaching without visual scene information, these trials were also removed for analysis. In total, 6387 out of 7776 trials remained for analysis.

## **Eye movement behavior**

To investigate eye movement behavior during the scene encoding phase, we computed the relative frequency of fixations (**Figure 3A**). To do so, we averaged fixation positions (excluding saccades) across all encoding phase time frames and convolved the result with a Gaussian filter of 1.5◦ width. The result was plotted as a heat map and overlaid onto an example encoding image.

## **Allocentric weights**

In order to investigate the influence of allocentric information in the scene on reach endpoints, we computed allocentric weights using linear regressions. In a first step, we calculated the groupmean reaching endpoint for every combination of object configuration and target identity in the control condition. These values served as subjective target location in the scene. In a second step, for every single reaching response in the allocentric conditions, its horizontal deviation from the subjective target location of the corresponding control image (same target and arrangement) was computed and compared to the expected allocentric deviation. Expected allocentric deviations were calculated for every test image as the average value by which the reference objects were shifted in the scene. For example, a visual leftward shift of three reference objects by 4.24◦ , 3.80◦ and 3.36◦ (loc3 condition; objects placed at different locations in depth) would result in an expected allocentric deviation of 3.80 cm (average of the three individual object shifts) if the target were solely represented in an allocentric reference frame, i.e., relative to other objects in the scene. In general, leftward deviations were coded as negative values and rightward deviations as positive values. In a third step, the observed horizontal deviation from the subjective target location for a leftward and for a rightward shift of the same target in the same arrangement were plotted against the expected allocentric deviations for each individual and each allocentric condition. Finally, a regression line was fitted to the

data and the slope of the regression line determined the allocentric weight.

We applied one-sampled *t*-tests to examine whether individual local and global allocentric weights significantly differed from 0. Since allocentric weights are computed on the basis of the results of the control condition, a test against zero corresponds to a statistical comparison to the control condition. To compare individual allocentric weights across conditions, we then computed one-way repeated measures ANOVAs with three levels for the local condition (loc1, loc3, loc5) and the global condition (gloT, gloC, gloP), separately. Significant results were followed-up with *post-hoc t*tests. Based on our hypotheses, *t*-tests were calculated one-sided and corrected for multiple comparisons using Bonferroni-Holm correction.

## **Response latency and movement time**

To test for differences in movement initiation and duration depending on the experimental conditions, we examined response latencies and movement times respectively. Response latencies were determined as the time from the disappearance of the test image until the start of the reaching movement which was defined as the point in time when the right index finger exceeded a velocity of 50 mm/s for 200 ms. Movement time was determined as the time from the start of the movement until its end defined as the time point when the velocity of the index finger fell below 50 mm/s for 100 ms and distance to the screen was minimal. Individual median response latencies and movement times were compared between the experimental conditions by computing separate one-way repeated measures ANOVAs with four levels for the local condition (loc1, loc3, loc5, control) and for the global condition (gloT, gloC, gloP, control). Two-sided *post-hoc t*-tests were calculated and corrected for multiple comparisons using Bonferroni-Holm correction.

## **Frequency of no-reach responses**

We instructed participants to perform no reach movement if they were uncertain about the location and/or identity of the target. Frequency of trials in which subjects did not respond was computed per condition and tested against the assumption that those trials are equally distributed across all conditions by using a Friedman's test.

# **Reach trajectories**

To determine whether allocentric influences were part of the overall movement plan or whether they emerged only during online corrections (cf. Krigolson et al., 2007; Burns and Blohm, 2010), we analyzed reaching trajectories using functional data analysis (FDA; Ramsay and Silverman, 2005). Some trials were excluded from the analysis due to the following reasons: (a) less than 50 data frames were collected per reaching movement due to Optotrak marker visibility problems; (b) moving velocities exceeded 600 cm/s during one reaching movement; and (c) trials lacked more than 20 consecutive data frames. Following these criteria only three trials (<0.1%) were discarded.

First, we shifted the movement onset (i.e., the first data frame) of each trajectory to the coordinate point 0/0/0 (*x*-, *y*-, *z*-direction in 3D Cartesian space) and aligned the subsequent data frames. Second, we spatially normalized the trajectories by fitting order 6 splines to each of the three dimensions (*x,y,z*) with a spline at every data frame. Third, we smoothed the data using a roughness penalty on the fourth derivative and λ = 1−<sup>10</sup> (within 0.008 of the generalized cross-validation estimate). Out of this mathematical definition we evaluated for each trajectory 1200 equally spaced data points. Then, 120 out of 1200 points were extracted resulting in spatially normalized trajectories. This procedure had also the advantage that missing data frames within one reaching movement were interpolated (for further details see also, Chapman and Goodale, 2010). As reaching endpoints differed between different stimulus' images (due to different target locations on the screen) within one condition, trajectories had to be rotated to one single reaching endpoint per condition to be able to average reach trajectories. Therefore each trajectory was transformed to the polar coordinate system. For every possible combination of object arrangements and targets, we calculated the mean angle of the last data point of the control conditions for every subject. This value was then subtracted from every angle value of the control condition and any other condition of the corresponding arrangement-target combination, resulting in a rotation of the trajectories of the control condition to the center of the display and a respective rotation of the trajectories from the other conditions. Consequently, the distances and proportions between control trajectories and the trajectories from other conditions remained unaffected. Afterwards the rotated trajectories were converted back to the Cartesian coordinate system. Finally we averaged trajectories over every condition for every subject.

For statistical analysis the preprocessed, normalized and averaged trajectories were entered into four functional-ANOVAs (Ramsay and Silverman, 2005), two for global and two for local conditions including one for right- and one for leftward object shifts. The functional-ANOVA models were single factor designs with four levels (control, loc1, loc2, loc3 and control, gloT, gloC, gloP). Functional pairwise comparisons (equivalence to a paired *t*-test) between the control condition (no object shift) and every experimental condition (with object shift) were conducted *posthoc* (one comparison for each shift direction).

# **RESULTS**

#### **EYE MOVEMENT BEHAVIOR**

In the present study we investigated whether or not allocentric coding schemes are used when people reach to remembered targets in a natural scene. We manipulated the location of the reference objects by shifting the objects to the left or to the right before reaching. Reference objects were either potential reach targets (local condition) or other objects in the scenes (global condition). First, we sought to quantify eye movement behavior during the encoding phase. **Figure 3A** illustrates the relative frequency of fixations overlaid on an example encoding image (see Section Materials and Methods for details). Clearly, participants visually explored relevant portions of the image, i.e., local object regions where potential reach targets were located. The screen center, the position of the future fixation cross, naturally resulted in the most frequent fixation location (red). **Figure 3B** depicts a typical example of individual reach endpoints for one participant and the applied exclusion criteria towards one target (egg). Clearly, only real outliers were removed.

# **ALLOCENTRIC WEIGHTS**

**Figure 4** represents the reach endpoints for all participants observed in the local and global conditions. As the overall pattern shows, reach endpoints were influenced by left- and rightward shifts of three or five reference objects in the local conditions (**Figure 4A**) but were hardly affected in the local and global conditions when only one reference object was shifted (**Figure 4B**). In particular, reach errors were distributed along the horizontal axis and increased with the number of local objects shifted in the scene.

**Figure 5** displays the observed horizontal reach errors as a function of the predicted allocentric reach errors for each test image. Reach errors varied within the expected direction of the shift of the reference objects in the loc5 and loc3 conditions where 5 or 3 local objects were shifted before the reach. The allocentric weights ranged between 1% to 43% in the local conditions and 1% to 4% in the global conditions. **Table 2** summarizes the mean (SD) reach errors for each individual participant and for loc3 and loc5 conditions separately. A leftward shift of the reference objects resulted in reach endpoints left of the target location and vice versa. This was confirmed by the allocentric weights (= slope of the regression line) which significantly differed from 0 in the loc5 (*t*(11) = 9.90, *p* < 0.001) and the loc3 (*t*(11) = 2.43, *p* = 0.017) conditions. We found a smaller but non-significant effect for the gloT condition where the table was shifted in the scene (*t*(11) = 2.36, *p* = 0.019; critical *p*-value = 0.0166).

for one test image across participants. Red colors indicate a leftward, green colors a rightward target shift.

**Table 2 | Mean (SD) reaching endpoints relative to control condition for every participant in the loc3 and loc5 condition, split up by the direction of the allocentric shift**.

specifies mean reach endpoints for one test image. Colored lines represent regression fits for each allocentric condition.


Each data point is based on 36 trials (minus disregarded trials). Negative values indicate a leftward shift relative to control condition. All values are reported in degree visual angle.

To compare the individual allocentric weights within the allo-local and the allo-global conditions, we computed oneway repeated measures ANOVAs which revealed a significant main effect of condition for allo-local (*F*(2,22) = 59.35, *p* < 0.001) but no effect for allo-global (*F*(2,22) = 2.438, *p* = 0.111). *Post-hoc t*-tests indicated that allocentric weights in the loc5 condition were significantly higher than in the loc3 (*t*(11) = 8.935, *p* < 0.001) and the loc1 (*t*(11) = 9.448, *p* < 0.001) conditions. In addition, allocentric weights in the loc3 condition were higher than in the loc1 condition (*t*(11) = 2.348, *p* = 0.019). Thus, allocentric weights increase with an increasing number of local reference objects shifted in the horizontal plane (**Figure 6**).

It has previously been shown that landmarks can influence reach trajectories and that this effect is distance dependent (Diedrichsen et al., 2004). Therefore, we also tested for the effect of proximity in the loc1 and loc3 conditions by correlating the observed reaching error with the mean distance of the shifted object/s with respect to the target. However, we could neither find a correlation for the loc1 (*r* = −0.09, *p* = 0.615) nor for the loc3 (*r* = −0.01, *p* = 0.962) conditions.

# **RESPONSE LATENCY AND MOVEMENT TIME**

Response latencies of reaches for the allo-local and the allo-global conditions are illustrated in **Figure 7A**. Response latencies did not significantly differ between the allo-global conditions (*F*(3,33) = 0.372, *p* = 0.774) but significantly differed between the allo-local conditions (*F*(3,33) = 14.54, *p* < 0.001). In comparison to the control condition, reaches were slower in the loc1 (*t*(11) = 5.643, *p* < 0.001) and the loc3 (*t*(11) = 6.64, *p* < 0.001) conditions. Moreover, reaches in the loc3 condition were also initiated more slowly than in the loc5 condition (*t*(11) = 3.616, *p* = 0.004). Movement times did neither vary between allo-local conditions (*F*(3,33) = 0.560, *p* = 0.645) nor between allo-global conditions (*F*(3,33) = 0.44, *p* = 0.726).

# **FREQUENCY OF NO-REACH RESPONSES**

To assess task difficulty, we tested whether the frequency of trials in which subjects did not respond differed across all conditions and thus violates the assumption of equal trial distribution across conditions. The results of the Friedman test rejected the assumption that those trials are equally distributed across all conditions (χ <sup>2</sup> = 46.6, *p* < 0.001). As depicted in **Figure 7B**, participants showed more frequent no reaching responses in the local compared to the global conditions with the highest frequency in the condition where three local objects were shifted (loc3).

# **REACH TRAJECTORIES**

To examine whether reaching errors due to allocentric object shifts emerged early during the reaching movement (due to different motor plans) or late during the reaching movement (due to error correction mechanisms), we used four functional ANOVAs (one for each experimental condition and shift direction) and functional pairwise comparisons to compare reaching trajectories of different allocentric conditions and the control condition. The functional ANOVAs revealed that trajectories of local object shifts differed in the horizontal plane (*x*-axis, parallel to the screen). Trajectories for both leftward and rightward shifts started to differ roughly at half-distance (≈48.75% = 11.7 cm) of the reach trajectory (**Figure 8A**, significant regions indicated by the gray vertical bars). Functional ANOVAs for global object shifts showed significant differences for leftward shifts starting from roughly the last third (68.3% = 16.4 cm) up to the end and for rightward shifts just for a small area right after half-distance (from 57.5% = 13.8 up to 68.3% = 16.4) of the reaching movement. Subsequent functional pairwise comparisons between every local condition and the control condition for the two shift directions showed that only trajectories in the loc5 condition differed significantly from the control condition. Loc5 trajectories for leftward and for rightward shifts started to differ slightly earlier than halfdistance of the reaching movement (leftward: 43.3% = 10.4 cm; rightward: 48.3% = 11.6 cm). Differences increased until the end of the movement (**Figure 8A**, indicated by the red vertical significance bars). Functional pairwise comparisons for global conditions revealed only a significant difference between gloT and the control condition for leftward object shifts for roughly the last third of the reaching movement (starting from 70.8% = 17 cm till the end; **Figure 8B**, indicated by the blue vertical significance bar).

# **DISCUSSION**

In this study, we investigated the use of egocentric and allocentric information during memory-guided goal-directed reaching using a naturalistic visual scene. Allocentric information was varied by shifting objects on the table (local objects) or objects in the environment (global objects) leftwards or rightwards after

scene encoding and before reaching. Memory-guided reaching movements were performed without visual information about the scene while gaze and body position remained fixed. We predicted accurate reaching movements if participants relied only on egocentric object coding, i.e., representing the target relative to gaze or body position, and systematic shifts of reach endpoints if they used allocentric cues (local or global) for goal-directed reaching. Our results demonstrated that reach endpoints varied as a function of objects shifted in the scene. The more local objects were horizontally misplaced the larger were the reach errors in the direction of the objects shifted. The present findings suggest that allocentric cues are indeed used during goal-directed reaching, but only if a substantial change of allocentric information is present in complex visual scenes.

Previous studies consistently reported that reach targets are represented relative to gaze direction, i.e., in an egocentric frame of reference (e.g., Henriques et al., 1998; Medendorp and Crawford, 2002). Beyond egocentric coding, allocentric cues also contribute to reaching movements as has been demonstrated in studies using visual landmarks (Obhi and Goodale, 2005; Byrne and Crawford, 2010; Byrne et al., 2010), imagined landmarks (Carrozzo et al., 2002) or structured visual backgrounds (Krigolson and Heath, 2004; Krigolson et al., 2007). While these studies examined reaching movements in rather unnatural tasks using isolated visual targets presented together with abstract, task-irrelevant landmarks, here we studied reaching behavior with more naturalistic stimuli by using photographic images of a breakfast scene. Despite the stable and reliable egocentric information of body and gaze position, we found large effects of allocentric cues on reach endpoints in line with the previous findings based on less ecologically valid experimental tasks (e.g., Byrne and Crawford, 2010). Since the target was defined as the missing local object in the shifted target scene, object shifts seem to be incorporated into the memory representation of the target established during scene encoding resulting in a combined representation which is used for calculating the reach plan. This is supported by the reaching trajectories in the object shift condition (loc5) which started to deviate from the no-shift condition early after reach onset. In sum, our results suggest that allocentric cues are even effective if they are provided after target encoding.

The present results demonstrated that the number of local objects shifted in the scene systematically affected reaching movements. We found larger distortions of reach endpoints with an increasing number of local objects shifted in the scene. Reach errors were most pronounced when all remaining local objects (loc5) were shifted, intermediate when three local objects (loc3) were shifted and absent for shifts of one local object (loc1). This result implies that substantial changes of allocentric cues in complex visual scenes are required to influence reaching movements. It is important to note that after object shifts the spatial relations between the objects in the loc5 condition remained constant while they completely changed in the loc3 condition. This resulted in a higher number of no-response trials and slower response latencies in the loc3 condition indicating higher task difficulty. Nevertheless, allocentric coding was still present in the loc3 condition, but the effect was diminished compared to the loc5 condition. Based on the present data, we cannot disentangle whether the reduced effect of allocentric coding is caused by larger task difficulty or fewer changes in the scene image. Previous findings on the Roelofs effect argue for the latter factor showing that the amount of a perceived target displacement when the whole frame around the target was shifted equaled the sum of a perceived target shift when only parts of the frame were shifted (Walter and Dassonville, 2006). Accordingly, we observed that allocentric weights were highest in conditions, when five local objects were shifted and lowest, when only one object was moved with the weights of three shifted local objects in between. We exclude a potential effect of proximity of target and allocentric cues on reach endpoints (c.f., Diedrichsen et al., 2004) because local object shifts appeared in the immediate vicinity of the target. Thus, we suggest that in a realistic visual environment it is the number of changed allocentric cues rather than distance that determines integration weight.

Local objects might also function as potential obstacles in real world situations which are especially important for movement programming. Obstacles constitute spatial constraints on movement execution and thus are not considered as distractors but rather as task-relevant non-target information (Tresilian, 1998) which is represented together with the target information in the attention system (Tipper et al., 1997; Baldauf and Deubel, 2010). As a consequence, the presence of obstacles requires additional anticipatory processing of movements leading to slower movement initiation (Saling et al., 1998; Biegstraaten et al., 2003). Accordingly, we observed longer response latencies when local objects (loc1 and loc3) were shifted, but not for global object shifts.

The absence of an influence of global allocentric cues on reach endpoints can be explained by multiple factors. First, the changes of global objects in the scene were undersized due to only one global object being shifted (instead of multiple as in the local conditions). Therefore, global conditions might be more similar to the loc1 condition. One can speculate that an increase in the number of shifted global objects might lead to similar results as we observed for the local object shifts. Second, it is also possible that it is the object displacement relative to object size that plays a role, in which case smaller objects should have larger influences on allocentric coding. Third, we cannot entirely rule out that the visibility of the frame of the presentation screen throughout the experiment has acted as a strong global allocentric cue. Since the screen never moved but the frame of the screen was a very salient visual feature (i.e., high contrast), it might have overridden more subtle global allocentric cues within the images. Fourth, local and global objects differed in task relevance, in the way that local objects represented potential reach targets in contrast to global objects which never served this function. This information was given by task instruction and thus may have influenced strategic behavior. Task relevance has been shown to affect overt attention in naturalistic tasks resulting in more fixations on taskrelevant than task-irrelevant objects (Land and Hayhoe, 2001; Ballard and Hayhoe, 2009). These findings are consistent with the fixation behavior we observed during the encoding phase which was spatially restricted to locations of the local objects. Fixations also frequently occurred at the table/table cloth placed right underneath the local objects; however, these global objects did not affect reaching behavior. In support of this finding, previous work demonstrated that object features which are taskirrelevant are not attended even if the respective object is fixated (Triesch et al., 2003). Together with the fact that working memory capacity for spatial information is limited to up to 4 items (Luck and Vogel, 1997) and retention of task-relevant objects is prioritized (Maxcey-Richard and Hollingworth, 2013), it is conceivable that participants encoded the location of local objects, i.e., task-relevant information, which were then incorporated into the reach plan while ignoring the location of the global objects, i.e., task-irrelevant information in the environment. Whether or not task relevance of allocentric information is a central factor in reach planning should be examined in future studies. Finally, the global allocentric cues lacked of a causal relationship to the reach target as discussed in the next paragraph.

We believe that our findings can be explained in the framework of causal Bayesian integration (Körding and Tenenbaum, 2007; Körding et al., 2007). The gradual increase of allocentric cue effects with the number of shifted local objects is consistent with more reliable allocentric cue information when more local objects are shifted. In that sense, the more local objects are shifted, the smaller the variance associated with allocentric information and thus the higher the allocentric weight in the integration of egocentric (probably body and gaze) and allocentric position. But how does this explain the absence of global allocentric cue effects? We believe that the concept of causality in Bayesian integration might be a key in understanding this. First, one can argue that there is no real causal link between the global objects and the local objects, as the picture frame is totally task-irrelevant and the exact position of the table and table cloth are not important, unless local objects had been positioned at the edge (and could thus fall off), which was not the case. Second, the spatial extend of the table and table cloth might have simply resulted in less precise positional information due to their large spatial extent. Third, and maybe more importantly, when the table cloth or table moved, local objects stayed fixed in space (i.e., did not move with the table and table cloth). Thus, the causal link between table/cloth and local objects on the table was broken, since normally objects would move with the table/cloth. In that case, causal Bayesian integration discounts any global allocentric cue effects due to a lack of a causal relationship between table/cloth movement and target location.

Our observations that movement endpoints are systematically shifted by local allocentric cues could result from two different sources: reach trajectory planning (Burns and Blohm, 2010) or feedback-based control processes (Krigolson et al., 2007). Indeed, allocentric information could be included in the reach plan right from the start as is the case in visual-proprioceptive integration (Sober and Sabes, 2003, 2005; Burns and Blohm, 2010), in which case one would expect manifestations of allocentric influences on the reach plan early on in the reach trajectory. Alternatively, allocentric information could only be incorporated during feedback corrective processes (i.e., later on in the movement), which would be consistent with observations of allocentric visual background influences on reaches (Krigolson et al., 2007). Our data on reaching trajectories is consistent with the former hypothesis and shows that local allocentric information might influence reach planning differently than allocentric background information.

In the present study we examined egocentric-allocentric cue integration for memory-guided (not visually-guided) reaches. Reaches were performed immediately after the presentation of the test scene; a condition which is usually defined as immediate reaching (cf. Bridgeman et al., 2000; Hay and Redon, 2006). However, here we asked participants to reach to the missing object in the test scene which required to build up representations of potential reach targets during the encoding scene which were then updated on the basis of the test scene after a 2 s delay. Delay is believed to have an important influence on spatial coding. For example, Hay and Redon (2006) found that delayed reaching accuracy declined in darkness but remained constant when a structured visual background was available. They explain their findings with a decaying egocentric representation and a more permanent allocentric representation of target location. This is also consistent with observations that visual landmarks increase space constancy (Deubel et al., 2010) and decrease egocentric, gaze-dependent coding of reach targets (Schütz et al., 2013). Furthermore, allocentric information has a stronger impact on delayed than immediate reaches showing increased reach errors in the direction of a shifted landmark with longer delays between stimulus offset and motor response (Bridgeman et al., 1997, 2000). An interesting prediction from these findings is that shorter (resp. longer) delays should lead to lower (resp. higher) allocentric weights because egocentric information is initially more accurate but decays faster than allocentric information.

Overall, we have shown that allocentric information is used by the brain to plan memory-guided reaches towards targets in naturalistic visual images. Our data is generally consistent with Bayesian causality principles and demonstrates that egocentricallocentric cue integration is highly flexible and task-dependent. It would be interesting to further examine the role of causality in egocentric-allocentric cue integration, in particular with respect to the causal relationship between visual landmarks.

# **ACKNOWLEDGMENTS**

This collaborative project was funded by the German-Canadian International research training group IRTG 1901 "The brain in action" from the German research foundation (DFG). Experiments were conducted in Gunnar Blohm's lab, funded by NSERC (Canada), ORF (Canada), CFI (Canada) and the Botterell Foundation (Queen's University, Kingston, ON, Canada).

# **REFERENCES**


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 09 May 2014; accepted: 30 July 2014; published online: 25 August 2014*. *Citation: Fiehler K, Wolf C, Klinghammer M and Blohm G (2014) Integration of egocentric and allocentric information during memory-guided reaching to images of a natural environment. Front. Hum. Neurosci. 8:636. doi: 10.3389/fnhum.2014.00636 This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Fiehler, Wolf, Klinghammer and Blohm. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

# Spatial task context makes short-latency reaches prone to induced Roelofs illusion

# **Bahareh Taghizadeh1,2 and Alexander Gail 1,2,3\***

<sup>1</sup> Sensorimotor Group, German Primate Center, Leibniz Institute for Primate Research, Göttingen, Germany

<sup>2</sup> Faculty of Biology and Psychology, Georg-August-Universität, Göttingen, Germany

<sup>3</sup> Bernstein Center for Computational Neuroscience, Göttingen, Germany

#### **Edited by:**

J. Douglas Crawford, York University, Canada

#### **Reviewed by:**

Lore Thaler, Durham University, UK Patrick A. Byrne, University of Toronto, Canada

#### **\*Correspondence:**

Alexander Gail, Sensorimotor Group, German Primate Center, Leibniz Institute for Primate Research, Kellnerweg 4, Göttingen 37077, Germany e-mail: agail@gwdg.de

The perceptual localization of an object is often more prone to illusions than an immediate visuomotor action towards that object. The induced Roelofs effect (IRE) probes the illusory influence of task-irrelevant visual contextual stimuli on the processing of task-relevant visuospatial instructions during movement preparation. In the IRE, the position of a taskirrelevant visual object induces a shift in the localization of a visual target when subjects indicate the position of the target by verbal response, key-presses or delayed pointing to the target ("perception" tasks), but not when immediately pointing or reaching towards it without instructed delay ("action" tasks). This discrepancy was taken as evidence for the dual-visual-stream or perception-action hypothesis, but was later explained by a phasic distortion of the egocentric spatial reference frame which is centered on subjective straight-ahead (SSA) and used for reach planning. Both explanations critically depend on delayed movements to explain the IRE for action tasks. Here we ask: first, if the IRE can be observed for short-latency reaches; second, if the IRE in fact depends on a distorted egocentric frame of reference. Human subjects were tested in new versions of the IRE task in which the reach goal had to be localized with respect to another object, i.e., in an allocentric reference frame. First, we found an IRE even for immediate reaches in our allocentric task, but not for an otherwise similar egocentric control task. Second, the IRE depended on the position of the task-irrelevant frame relative to the reference object, not relative to SSA. We conclude that the IRE for reaching does not mandatorily depend on prolonged response delays, nor does it depend on motor planning in an egocentric reference frame. Instead, allocentric encoding of a movement goal is sufficient to make immediate reaches susceptible to IRE, underlining the context dependence of visuomotor illusions.

**Keywords: reach movement, induced Roelofs effect, illusion, reference frame, allocentric, object-centered**

# **INTRODUCTION**

Goal-directed, object-oriented reach movements require accurate localization of the target object, yet object localization can be prone to visual illusions. The fact that in many cases visual perceptual localization is more prone to illusions than immediate visuomotor responses (Smeets and Brenner, 2001) is typically taken as strong evidence for two functionally independent visual processing streams, a ventral "vision-for-perception" pathway, and a dorsal "vision-for-action" pathway (Goodale and Milner, 1992; see Schenk et al., 2011 and Westwood and Goodale, 2011 for recent reviews). Understanding the circumstances under which perceptual illusions do or do not affect motor performance can be highly informative about the nature of the two putative visual streams, and, more specifically, about the nature of visuospatial processing during sensorimotor transformations (Smeets et al., 2002). Here we re-investigate the induced Roelofs effect (IRE) in reach movements. In the IRE, the position of a task-irrelevant visual object induces a shift in localization of the target object. The IRE depends on the mode of the subjects' behavioral response to indicate this position, e.g., key-presses vs. immediate reaches towards the target (see details below). This observation was originally taken as evidence for the dual-visual-stream or perceptionaction hypothesis (Bridgeman et al., 1997), attributing the IRE to ventral stream perceptual processing. A later, opposing view explained the IRE by a phasically distorted egocentric (object-toself) reference frame—i.e., changes in space defined relative to the own body—attributing the IRE to dorsal stream processing along the vision-to-action pathway (Dassonville and Bala, 2004b). Here we expand on these findings by revisiting the IRE in a short-latency reach task. In the first experiment, different to previous studies, we varied the spatial task context in which reaches had to be performed. We distinguished reaches in an allocentric (object-to-object) reference frame, i.e., a task in which the reach goal location is defined relative to another object, from otherwise identical reaches in an egocentric reference frame, i.e., reach goals relative to the own body. We thereby test if the IRE can also be induced for immediate reaches to the target (typically considered an "action" task) if the spatial context of the task is modified. In a second experiment, we test if the IRE critically depends on a phasic distortion of an egocentric frame of reference or if it can also be induced by allocentric encoding.

The IRE probes the illusory influence of task-irrelevant visual context stimuli on the processing of task-relevant visuospatial instructions during movement preparation. Note that taskrelevance here refers to whether a stimulus was instructive for subjects, independent of its effect on behavior. In a series of studies Bridgeman et al. (1997, 2000) showed that the position of a task-irrelevant visual object (frame) can induce a systematic shift in localization of visual targets. When the frame was laterally off-center relative to subjects' mid-sagittal plane, i.e., the frame was shifted to the left or right with respect to the subjects' body midline, subjects misjudged the position of targets presented inside the rectangular frame (Bridgeman et al., 1997). The mislocalization was in the opposite direction of the frame shift, i.e., if the frame was left of the midline then targets were mislocalized to the right, and vice versa. Target mislocalization occurred in two conditions. First, when subjects had to indicate target position by pressing response keys assigned to different targets. The keyboard was placed on the table in front of the subjects, and hence the keys were spatially incongruent to the actual target positions. Second, when subjects pointed to the memorized position of the target after an instructed delay (Bridgeman et al., 1997). Importantly, when subjects in the same task indicated the target position without instructed delay by either pointing to it (Bridgeman et al., 1997) or by directly reaching to jab at the target (Bridgeman et al., 2000), no IRE was observed. This discrepancy was originally interpreted as an indication of separate visuospatial representations for direct sensorimotor processing (immediate reaching or pointing without instructed delay) in the dorsal visual stream, compared to spatial cognitive or perceptual processing (verbal response, using response keys, or pointing with instructed delay) in the ventral visual stream. This dual-visualstream or perception-action hypothesis of the IRE was based on two assumptions. First, only the perceptual "cognitive" ventral stream is prone to the IRE illusion. Second, only the immediate and directly target-aimed manual responses can be performed by direct egocentric sensorimotor processing in the dorsal stream. Symbolic responses (verbal response or pressing of response keys) and delayed memory-guided reaching and pointing, on the other hand, require ventral stream processing (Bridgeman et al., 1997, 2000). In case of visually instructed delayed reaching and pointing, the need for ventral stream processing arises from the assumption that the dorsal vision-to-action pathway is incapable of even medium-term (several seconds) memory storage of the required reach parameters, while immediate reaches can be processed by the dorsal stream alone, as further discussed below.

Behavioral and imaging studies challenged this interpretation of the IRE in favor of an alternative biased-midline hypothesis (Dassonville and Bala, 2004b; Dassonville et al., 2004) in which the IRE is explained by a temporary distortion of the egocentric spatial frame of reference which is used for reach planning and which is centered on the direction of the subjective straight-ahead (SSA; see **Figure 1**). Dassonville and colleagues showed that the IRE can be accounted for by an observed mislocalization of the memorized array of reference positions, relative to which the target position had to be indicated with a saccade. Since the mislocalization of the memorized reference positions occurred in the same direction as the off-centered visual frame it explained the observed target localization error opposite to the off-centered frame. This finding was interpreted as indication of a phasic translational shift in an egocentric reference frame which is used for movement planning, and which is centered on the direction of SSA (Dassonville and

opposite to the direction of the frame offset.

Bala, 2004b; Dassonville et al., 2004). According to this biasedmidline hypothesis, in an immediate motor response task (nondelayed pointing, reaching, or saccade) the target location and the corresponding movement plan will both be encoded in the phasically shifted egocentric frame of reference, and the movement plan will be executed while the reference frame is still shifted. No obvious movement error occurs, since movement planning and execution are both affected by the shift, and hence the shift is compensated (**Figure 1B**). In a delayed-response task, the movement will be executed after relaxation of the shifted SSA back to the mid-sagittal plane. This induces a target error to the direction opposite to the off-set visual frame, since the movement was planned relative to the SSA but executed relative to the original un-biased frame of reference after relaxation (**Figure 1C**).

An fMRI study of the IRE revealed differential activity in the dorsal visual stream but not in the ventral stream (Walter and Dassonville, 2008).The dual-visual-stream hypothesis would have pointed to a main contribution from the ventral stream for IRE-prone behavioral conditions. In contrast, the biasedmidline hypothesis implies that the IRE is based on a single egocentric visuospatial reference frame, likely in the dorsal visual stream, which would be relevant for both the IRE-resistant "sensorimotor" or "action" tasks (immediate target-directed manual or ocular response) and the IRE-prone "cognitive" or "perceptual" versions of the task (delayed pointing and looking or symbolic responses). However, the localization of IRErelated neural activity in the dorsal stream does not answer the questions of which spatial reference frame and which temporal dynamics determine the behavioral consequences of the IRE. The previously suggested dual-visual-stream model for the IRE is tied to the perception-action model (Goodale and Westwood, 2004; Goodale et al., 2004), according to which the ventral and dorsal visual streams are preferentially associated with allocentric and egocentric processing, respectively. On the other hand, there is growing evidence for parallel existence of both spatial reference frames within the dorsal visual pathway (Burgess, 2006; Milner and Goodale, 2008) and it is clear that the brain uses both types of information for localization of spatial targets in many tasks (Byrne and Crawford, 2010). Accordingly, spatial locations are not purely encoded in egocentric frames of reference in the posterior parietal cortex. The fMRI-active areas in the Dassonville IRE study (Walter and Dassonville, 2008) overlapped not only with areas shown in previous experiments to be involved in egocentric spatial localization (Vallar et al., 1999), but also with areas involved in allocentric localization relative to immediate visual objects (Galati et al., 2000; Thaler and Goodale, 2011a) or the enduring spatial features of a familiar environment (Galati et al., 2010). In addition, Fink et al. (2003) showed that egocentric and allocentric (object-centered) reference frames can interact in the human parieto-frontal network. Although there are not many studies directly comparing egocentric and allocentric reference frame in monkeys, there is evidence that neurons in parietal area 7a can encode the spatial location of objects in an eye-centered (i.e., egocentric) reference frame (Andersen et al., 1985) as well as relative to other task-relevant objects (Chafee et al., 2007; Crowe et al., 2008). Neurons in the same area are gain-modulated by the position of the subject's body in the surrounding environment (i.e., in world-centered reference frame) (Snyder et al., 1998). The original dual-visual-stream hypothesis for the IRE argued that the dorsal stream, which dominates immediate egocentric "action" tasks, makes use of the ventral stream information only in case of memory-guided tasks. This explains the susceptibility of reaches to the IRE when they are substantially delayed by several seconds (Bridgeman et al., 1997, 2000; Dassonville and Bala, 2004b).

In summary, both existing interpretations of the IRE, namely the dual-visual-stream and the biased-midline hypothesis, critically depend on the following observation: in tasks in which subjects are required to directly point to, look at, or touch the perceived target position, and in which they can do so in an egocentric reference frame, the IRE can be observed if the manual or ocular response is purposefully delayed by several seconds, but not if an immediate response is required (Bridgeman et al., 1997, 2000; Dassonville and Bala, 2004b). Since the biased-midline hypothesis assumes a distortion of an egocentric reference frame (a shifted SSA) which is only phasic, it predicts that immediate reaches should be resistant to IRE because visual encoding of the reference positions and the reach target are affected in the same way. The dual-visual-stream hypothesis, on the other hand, assumes that dorsal stream processing utilizes ventral stream information only in memory-guided action, hence, it predicts resistance to the IRE for immediate reaches in an egocentric reference frame, but makes no prediction about immediate targetaiming reaches in other reference frames. In experiment I we test if immediate reaches, independent of a prolonged reach delay, can become prone to IRE if the task context prevents egocentric reach planning. To dissociate egocentric from allocentric reach planning, we introduced a spatially incongruent object-centered reach task. In contrast to previous IRE reaching experiments, we also introduced ocular fixation constraints. Furthermore, the fact that the dorsal stream areas which are active during target localization in IRE tasks cover areas of egocentric as well as allocentric spatial encoding brings up the second and related question of whether the IRE is really restricted to phasic distortion effects on egocentric frames of reference induced by the relative position of an object to the body. If not, mislocalization effects like the IRE might also be induced by the relative (allocentric) position of an object relative to another object. Previous IRE experiments including allocentric task constraints were nevertheless still explained by egocentric causes (Dassonville and Bala, 2004b; Lester and Dassonville, 2011). In Experiment II we tested whether the IRE can interfere with allocentric reach planning and can thus be explained independently of an egocentric reference frame distortion.

# **MATERIALS AND METHODS APPARATUS**

Subjects were seated in a dimly lit room in front of a frontoparallel touch screen (43 cm distance from eye, screen center at eye level) so that their mid-sagittal plane was aligned to the center of the screen. Visual stimuli were presented on an LCD screen (19" ViewSonic VX922) mounted behind a touch-sensitive screen (IntelliTouch, ELO Systems, CA, USA). Custom-written display software (C++) was controlled via a real-time program running on a PXI computer (LabView, National Instruments). Stimulus display was synchronized with vertical synchronization of the screen to avoid latency jitter. Visual display latencies were recorded with a photo diode and corrected for during data analysis. All visual stimuli had a low intensity gray tone (9.0 cd/m<sup>2</sup> on a 1.2 cd/m<sup>2</sup> background) to minimize retinal afterimages. Hand position was registered using the touch screen. Gaze positions were registered using an infrared eye tracker at 500 Hz (SMI, Teltow, Germany, in experiment I and EyeLink 1000, Kanata, Canada, in experiment II). Subjects rested their head on a chinrest for stability. Behavioral parameters were monitored using the real-time control software.

# **SUBJECTS**

All subjects had normal or corrected-to-normal vision and were naïve with regard to the purpose of the study. Detailed written instructions were given to the subjects before each experiment. Experiments were in accordance with institutional guidelines for experiments with humans and adhered to the principles of the Declaration of Helsinki. All subjects gave their informed written consent prior to their inclusion in the study.

Eleven right-handed subjects (20 to 27 years, four females) participated in experiment I and control experiment Ia. Nine different right-handed subjects (22 to 39 years, five female) participated in control experiment Ib. A disjunct group of subjects was necessary for this control task to avoid possible task interference with experiment I. Ten different right-handed subjects (16 to 27 years, five females) participated in experiment II and control experiment IIa.

# **EXPERIMENTAL PARADIGM**

The following procedures for implementing the IRE were common to both experiments. Details specific for the individual experiments, especially the spatial positioning of stimuli, will be described in experiments I and II below.

Each trial started with an eye-fixation target, presented to the subject at the vertical midline (mid-sagittal plane), and 5 cm (7◦ visual angle) above the horizontal midline of the screen (**Figure 2A**). Subjects were required to fixate the spot throughout each trial within an invisible window of 3 cm (4◦ ) radius (ocular fixation). To start a trial, subjects had to push a "home" button, placed on subject's mid-sagittal plane on the desk 40 cm below the screen center, and keep it pressed with their index finger until a "go" signal occurred later in the trial (manual fixation). Whenever subjects failed to keep ocular or manual fixation, the trial was aborted and repeated at a random later time during the experiment. After valid eye and hand fixation of 500–700 ms, a reference array (RA) of five boxes, horizontally connected with a line, appeared for 200 ms. Boxes were 0.35 cm (0.5◦ ) squares, and centered 1.5 cm (2◦ ) apart from each other. The position of the boxes indicated the potential positions of the pending cue. Subjects were required to keep these positions in mind for proper task performance, as will become clear below. Reference array presentation was followed by a memory period of 3 s. After the memory period a visual cue was presented for 200 ms. The cue consisted of a small dot of 0.27 cm (0.35◦ ) diameter at the randomly chosen position of one of the five RA

boxes, indicating the target box to which subjects should later reach. The cue was surrounded by a simultaneously presented frame, which was 16.9 cm (21◦ ) wide and 6.6 cm (9◦ ) high, but which was task-irrelevant. Cue and frame were succeeded by a decision array (DA), which was graphically identical to the RA, but was not necessarily spatially congruent (see below). Stimulus-onset asynchrony (SOA) between "cue + frame" and the subsequent DA was 200–300 ms. Simultaneously to the appearance of the DA, an acoustic signal was presented for 50 ms as the go-signal, indicating to the subject to lift their finger from the home button and touch the target position on the screen within 1000 ms after the go signal. After a correctly executed trial, subjects received acoustic feedback (high-pitched tone).

One constraint common to both experiments was that the frame could randomly take one of three possible positions relative to the RA: centered, or shifted by 3.85 cm (5◦ ) to the left or right. Another constraint common to all experiments was that the cue appeared at one of the five RA positions. Subjects were instructed to hit the one of the five DA boxes which corresponded to the RA box at which they had perceived the cue, e.g., for a cue perceived at RA box #2 subjects should reach to DA box #2, irrespective of the absolute position of the DA. If the reach endpoint was within 4.5 cm (6◦ ) distance from the target box the trial was counted as "successful". By tolerating off-sets up to three boxes distant from the physically cued target box, we could analyze localization errors without inducing behavioral biases from error feedback. In the following sections, for each trial of the task the term "cue" refers to the dot stimulus presented simultaneously with the frame (**Figure 2A**, spatial cue + frame) and "target" refers to the position of the relevant box of the DA (i.e., the box of the DA that corresponds to the cued box of the RA).

Before entering the experiment, all subjects completed a training session and were encouraged to ask any questions which were not answered by the written instructions. The aim of the behavioral training was to familiarize subjects with object-based (allocentric) reach planning. More details on the training task will be elaborated for each experiment separately in the following sections.

# **EXPERIMENT I**

The main conclusion of this study will result from Experiment II. But since Experiment II differs in multiple respects from previous implementations of IRE tasks, we first wanted to establish some basic findings in our type of experimental setting to make the data more comparable to previous experiments. In experiment I, we asked what determines the "immediacy" of the reaches which previously did not show an IRE. Is it only the time between the presentation of the cue that instructs the target and the reach onset which determines whether the IRE occurs or not, or can the spatial frame of reference in which the cue and target have to be encoded cause an IRE even when other spatial and temporal reach parameters are matched? Experiment I and control experiment Ib aim to distinguish between these two alternatives by introducing a task in which reaches can be conducted without instructed delay ("temporally immediate") but might be associated with a spatially non-congruent target

position ("spatially not immediate", Experiment I), or only with congruent target position as in previous experiments (Experiment Ib). It is important to note that the positions of the frame stimulus relative to the body are still at the straight-ahead direction as in the original Roelofs experiment and in previous IRE experiments. To be able to later dissociate the frame position from any egocentric frame of reference, body-centered or eye-centered, we also tested for the effect of ocular fixation in our task (control

plane. **(B)** Experiment I: In order to test the IRE in an allocentric reference

experiment Ia), which previous reach experiments did not do. Note also, the term "temporally immediate" refers to the fact that the visual stimuli necessary to determine the reach target were available to subjects earlier than typical reach responses would occur. This means that there was no major response delay imposed by the sequence of stimulus events. Although spatial stimulus-response incongruencies and the need for allocentric spatial encoding are known to induce reach response delays in

location).

the order of a few 10 ms (Gail and Andersen, 2006; Westendorff et al., 2010; Thaler and Goodale, 2011b; Westendorff and Gail, 2011), such short additional latencies are about two orders of magnitude less than the instructed delays necessary to evoke an IRE in previous experiments (Bridgeman et al., 1997; Dassonville and Bala, 2004b; Dassonville et al., 2004; Walter and Dassonville, 2006, 2008; Bridgeman and Hoover, 2008).

# **Methods of experiment I**

In experiment I subjects were required to reach-to-touch the target location. The important difference of our design compared to previous IRE studies was that the physical positions of cue and target were spatially congruent in only 1/3 of the trials (**Figure 2B**). In the other trials the reference and DA were (partially) incongruent in their position, but otherwise identical. In experiment I the RA position was constant across trials and always at the center of the screen. The DA randomly took one of three possible positions relative to the RA: identical (congruent condition), shifted by 1.5 cm (2◦ ) to the left (partly congruent), or shifted by 2.12 cm (2.8◦ ) to the right and 2.12 cm up (incongruent). Only in the congruent condition were cue (one of the RA boxes) and target (the corresponding DA box) physically identical, as in previous IRE experiments using egocentric reaching or pointing tasks (Bridgeman et al., 1997, 2000; de Grave et al., 2002; Dassonville and Bala, 2004a,b; Dassonville et al., 2004; Walter and Dassonville, 2006, 2008; Bridgeman and Hoover, 2008; Lester and Dassonville, 2011). This task design resulted in 45 possible combinations of cue (target), frame and DA positions (5 × 3 × 3), which were randomly presented. Since subjects could not predict whether a trial will be congruent or not, they always had to encode cue position with respect to the RA in order to be able to perform the task correctly. Subjects needed to perform 200 hit trials, resulting in at least four repetitions per condition. In case subjects' errors might not be balanced across conditions, we decided against using "pseudo-random" trial orders to avoid changing probabilities of individual task conditions. Instead, we presented more than 4 × 45 trials to yield a minimum of four repetitions per conditions. Analysis of exactly four trials per condition instead of 4–5 trials per condition did not change the results.

Training was identical to the experimental task, except that the frame was not presented. Training was terminated after 20 hit trials.

## **Methods of control experiment Ia**

In a control experiment Ia, we tested whether the presentation of the ocular fixation target has an impact on the IRE. Since previous studies on IRE purposefully tried to avoid any possibility of allocentric spatial coding, no ocular fixation stimulus was shown to subjects during the trial (Dassonville and Bala, 2004b). Hence, in our control experiment Ia, we omitted the ocular fixation stimulus and did not impose any constraints on eye movements. This control was run for all subjects of experiment I on a separate day.

## **Methods of control experiment Ib**

In control experiment Ib, we reproduced the original IRE paradigm (Bridgeman et al., 1997) in order to establish that our setup and task layout allows us to reproduce previous findings of no IRE in immediate reaches. We used an independent group of subjects to avoid a possible transfer of response strategy between the two experimental designs. Each trial started with ocular and manual fixation. After valid fixation, cue and frame were simultaneously presented. Following the offset of cue and frame, an acoustic go signal indicated to the subjects to lift their finger from the starting home button and reach-to-touch the perceived location of the cue. Subjects had 1000 ms to finish the reach and they were required to hold ocular fixation until the end of the trial. There were no reference or decision objects shown in control experiment Ib. Importantly, the spatial layout and timing of the stimuli was otherwise identical to experiment I, i.e., the same cue, target and frame positions, sizes and presentation times were used. The 15 different possible combination of cue and frame (5 cue locations (0, ±2 ◦ and ±4 ◦ relative to the midsagittal plane) and 3 frame locations (0 and ±5 ◦ relative to the mid-sagittal plane)) were randomly presented to the subjects. For six out of the nine subjects stimuli had 23.5 cd/m<sup>2</sup> luminance on a 1.2 cd/m<sup>2</sup> background, for the other three the contrast was identical to experiment I. The results were independent of stimulus contrast, hence will be presented jointly.

# **EXPERIMENT II**

In experiment II, we tested whether the IRE in experiment I can be explained by a biased perception of the SSA. After we established with experiment I that incongruent reference and DA positions encourage allocentric reach planning and allow an IRE for short latency reaches to the target, we now additionally dissociated the position of the RA from the straight-ahead direction to test explicitly whether the IRE is determined by frame position relative to straight-ahead or relative to the RA.

During the training session for experiment II subjects performed the identical task to the incongruent condition of experiment I, but without the frame stimulus. The goal was to familiarize subjects with the setup and the allocentric reach task. Training was terminated after 20 hit trials.

# **Methods of control experiment IIa**

Trials in experiment IIa were identical to the incongruent condition of experiment I. Subjects conducted 75 correctly performed trials to test whether they were prone to IRE in the allocentric reach task. This served as baseline for the expected effect size in the experiment II for this group of subjects.

## **Methods of experiment II**

In Experiment II, we dissociated the position of the RA from the objective straight-ahead (see **Figure 2C**). Except for the positions of decision and RA, the procedure was the same as in the experiment I. The RA was randomly shifted by 5.8◦ (4.5 cm) either to the left or to the right of the objective straight-ahead with equal probability. As an example, consider the case when the RA was shifted to the right by 5.8◦ . Even if the frame was shifted by the maximum value of 5◦ to the left relative to the center of the RA (leftward allocentric shift of the frame), the center-of-mass of the frame still remained in the same hemi-field relative to objective straight-ahead (rightward egocentric shift of the frame, see **Figure 2C**). Although the frame could take three different positions relative to each of the two RA positions, it always stayed to the right of the body's midline if the RA was on the right side, and it stayed left of the body's midline when the RA was on the left side. Subjects were asked to maintain ocular fixation on the fixation target at the objective straight-ahead direction to align the body-centered reference frame with the gaze-centered reference frame. The DA was always located at the center of the screen, i.e., at the objective straight-ahead direction in all trials. According to the biased midline hypothesis, an offcentered frame relative to the body midline will cause target mislocalization to the direction opposite the frame shift. Therefore, one would expect when the RA and the frame were placed in the left or right hemi-field, they would cause a shift of the SSA to the same direction as the egocentric shift of the frame, thereby causing mislocalization of cue or target to the opposite side (**Figure 5A**). The 30 possible combinations of target, frame and RA positions (5 × 3 × 2) were presented in random order. The experiment included 160 hit trials to achieve 4–5 repetitions per condition.

# **DATA ANALYSIS**

For each combination of target, frame and DA position, the horizontal reach endpoint relative to the center of the decision array (HRDA) was taken as the subject's response (averaged across 4–5 identical trials). A HRDA of 2◦ (1.5 cm) means that the subject in this condition on average reached 1.5 cm to the right of the center of the DA. If the central box was cued, a HRDA of 2 ◦ (1.5 cm) corresponds to the nearest right neighboring box. A two-factor analysis of variance with cue position (5 levels) and frame position (3 levels) as factors was applied to HRDA for the population of all subjects (repeated measures ANOVA). A significant main effect of the factor "frame" indicated IRE. Additionally, for each position of the DA, the HRDA in the frameright conditions was subtracted from the frame-left conditions for each target position and the mean difference was computed. This average localization error was used to compare effect sizes between different task conditions.

# **RESULTS**

## **RESULTS OF EXPERIMENT I**

**Figures 3A–C** shows the average target localization error, quantified by the mean HRDA (see Section Methods), across all 11 subjects. The three panels show separately the three different DA positions. All three DA conditions showed highly significant main effects of the factors "cue" (incongruent/partlycongruent/congruent: *F*cue(4,40) = 134/124/142, all *p*cue < 0.0001) and "frame" (incongruent/partly-congruent/congruent:

decision array. Error bars represent S.E.M. For all three congruency conditions

planning, i.e., when no task-relevant RA existed.

*F*frame(2,20) = 22.6/26.5/26.7, all *p*frame < 0.0001), qualified by significant interactions (incongruent/partly-congruent/congruent: *F*cue <sup>×</sup> frame(8,80) = 5.80/6.02/5.55, all *p*cue <sup>×</sup> frame < 0.0001).

The significant factor "frame" in all three DA conditions demonstrates that the IRE occurred independently of the trialby-trial level of congruency between the reference and DA. The average localization error for individual subjects shows that the IRE was characterized by varying effect strength with most but not all subjects showing an IRE at the single subject level (average localization error for individual subjects: 3.79◦ , 4.38◦ , 0.52◦ ,1.19◦ , 5.02◦ , 0.91◦ , 4.20◦ , 2.03◦ , 4.65◦ , 0.29◦ , 3.50◦ ). The congruency condition did not affect the size of the localization error (*p* > 0.10, *F*congruency = 2.57, two-factor repeated measure ANOVA on localization error for population of 11 subjects with factors "congruency" and "target relative to DA"). At the population level, the localization error was 2.77◦ (S.E.M. across subjects: 0.54◦ , S.E.M across all subjects and task conditions: 0.15◦ ; **Figure 4A**).

This means that even in the congruent condition, which was identical to previous experiments in terms of spatial congruency of cue and reach target, a significant IRE was induced for immediate reaches. This was not the case in previous studies (Bridgeman et al., 1997, 2000; Dassonville and Bala, 2004b; Dassonville et al., 2004; Lester and Dassonville, 2011) where only congruent trials were presented (see also Section Results of control experiment Ib). None of the subjects showed a significant effect of congruency condition on reaction times (0.20 < *p* < 0.97, one-way ANOVA on per-subject trial-by-trial reaction times with factor "congruency"). From experiment I we can conclude that object-centered allocentric planning of immediate reaches is subject to the IRE.

In control experiment Ia we tested the effect of ocular fixation on the IRE by releasing the eye movement constraints but otherwise keeping everything identical to experiment I. The main result of this control was the same as for experiment I. The three congruency conditions in experiment Ia showed significant main effects of factors "cue" (incongruent/partlycongruent/congruent: *F*cue(4,40) = 98.4/97.0/99.7, all *p*cue < 0.0001) and "frame" (incongruent/partly-congruent/congruent: *F*frame(2,20) = 38.2/32.2/34.4, all *p*frame < 0.0001), qualified by significant interactions (incongruent/partly-congruent/congruent: *F*cue <sup>×</sup> frame(8,80) = 6.53/5.94/3.10, *p*cue <sup>×</sup> frame <0.0001/<0.0001/ <0.005). Mean localization errors for individual subjects were 5.45◦ , 6.22◦ , 0.40◦ , 2.77◦ , 6.24◦ ,1.48◦ , 5.51◦ , 2.57◦ , 5.80◦ , 1.70◦ and 5.52◦ . Across the population of subjects, the localization error was 2.99◦ (S.E.M. 0.50◦ , **Figure 4A**). A two-tailed paired *t*-test between experiments I and Ia did not show a significant difference in localization error with and without ocular fixation (*p* > 0.14). From experiment Ia we can thus conclude that in our allocentric reach task the introduction of an ocular fixation constraint to align body- and gaze-centered reference frames does not affect the IRE.

In control experiment Ib we replicated the original finding of Bridgeman et al. (2000) for immediate reaches by asking subjects to reach and touch the perceived location of spatial cues which were presented within a frame (**Figure 3D**). The two-factor repeated measure ANOVA on the population of nine subjects showed a significant main effect of the factor "cue" (*F*cue(4,32) = 435, *p*cue < 0.0001), but no significant effect of "frame" (*F*frame(2,16) = 2.15, *p*frame > 0.14), qualified by a significant interaction (*F*cue <sup>×</sup> frame(8,64) = 2.27, *p*cue <sup>×</sup> frame < 0.04). This means that the subjects correctly directed their reaches to the cue position (main effect of cue), but were unaffected by the frame stimulus (no main effect of frame). Correspondingly, mean localization errors for individual subjects were close to zero: 0.23◦ , −0.04◦ , 0.17◦ , 0.56◦ , 0.33◦ , 0.12◦ , −0.06◦ , −0.19◦ , 0.07◦ . The lack of an IRE for immediate egocentric reaches is comparable with the original finding (Bridgeman et al., 2000).

#### **RESULTS OF EXPERIMENT II**

In experiment I the sustained presence of a visual landmark at the direction of the objective straight-ahead, namely the fixation spot on which subjects kept ocular fixation, did not diminish the IRE. We consider it unlikely that despite continued ocular fixation at the true straight-ahead direction subjects would undergo a substantial shift in SSA. This allowed us to question the previous hypothesis that IRE is due to a temporarily perturbed perception of the SSA direction, an assumption of the biased-midline hypothesis that we want to test in Experiment II.

For both left and right peripheral positions of the RA, experiment II showed a significant effect of the factors "cue" (left/right: *F*cue(4,36) = 111/87.3, all *p*cue < 0.0001) and "frame" (left/right: *F*frame(2,18) = 54.0/58.4, all *p*frame < 0.0001), with no significant interaction (left/right: *F*cue <sup>×</sup> frame(8,72) = 1.63/1.69, *p*cue <sup>×</sup> frame >0.13/ >0.11; **Figure 5B**). Individual subjects had mean localization errors of 5.08◦ , 4.20◦ , 2.54◦ , 6.09◦ , 5.75◦ , 5.91◦ , 6.25◦ , 6.73◦ and 0.34◦ in experiment II and 5.47◦ , 4.43◦ , 0.57◦ , 5.66◦ , 6.52◦ , 5.75◦ , 5.79◦ , 4.46◦ and 0.73◦ in control experiment IIa. The average localization error across subjects for peripheral RA in experiment II was 3.71◦ (S.E.M. 0.48◦ ), and 3.43◦ (S.E.M. 0.52◦ ) for the central RA in control experiment IIa (**Figure 4B**). A paired two-tailed *t*-test between test and control experiment did not reveal a significant difference (*p* > 0.28).

The result of experiment II shows that the main source of IRE in our data is the relative position of the frame with respect to the reference object (allocentric shift of the frame) rather than with respect to the straight-ahead direction (egocentric shift of the frame).

# **DISCUSSION**

We conducted two experiments to study the effect of visual contextual information on reach planning. Our two experiments were designed such that subjects were required to encode first the cue and then the reach target relative to the position of a reference object, i.e., in an allocentric reference frame. In this case, subjects reliably showed an IRE (i) even for short-latency reaches to the target; (ii) with and without ocular fixation; and (iii) with mislocalizations being dependent on the allocentric position of the context stimulus (frame) relative to the center of the reference object, not the egocentric position relative to the SSA. Our results are not consistent with a previously suggested biased-midline hypothesis (Dassonville and Bala, 2004b). Instead, we suggest that the IRE can be induced by egocentric or allocentric spatial information, depending on which reference frame the task requires for the behavioral response.

## **IRE FOR ALLOCENTRIC REACH PLANNING**

In our study we show that IRE can be observed in an allocentric reference frame for reach planning, while previous studies emphasized the role of egocentric reference frames as an explanation.

Our findings argue against the idea that the IRE in our data can be explained by a phasic shift of the SSA direction (egocentric reference frame), as suggested previously (Dassonville and Bala, 2004b). First, we observed IRE with short-latency reaches. According to the biased-midline hypothesis short-latency reaches should not be subject to IRE since the assumed shift of the SSA is only phasic and affects target localization and reach planning likewise, such that the effect cancels out after relaxation of the SSA perturbation. Second, the fact that in our experiment I the IRE was also present when subjects were required to keep ocular fixation at a visual spot in the objective straight-ahead direction provided an additional hint that a shift in SSA might not be the cause of our observed results. We consider it rather unlikely that the SSA shifts in response to presentation of an off-center visual frame while subjects are fixating at a stable landmark in the true straight-ahead direction. Third, our experiment II provided direct evidence against the biased-midline hypothesis. For task conditions which should all have induced a SSA shift in the same direction, we found IRE in opposite directions (**Figure 5**). We therefore argue that in our data the Roelofs effect was not induced by an effect of the contextual visual frame stimulus on the SSA.

Ruling out a shifted SSA as explanation of the IRE in our experiment brings up the question which other egocentric or allocentric spatial encoding might be responsible for the observed IRE. Previous results do not contradict the idea of an allocentric IRE, since egocentric and allocentric reference frames were typically not dissociated. In previous IRE experiments (Bridgeman et al., 1997, 2000; de Grave et al., 2002; Dassonville and Bala, 2004a,b; Dassonville et al., 2004; Walter and Dassonville, 2006, 2008; Bridgeman and Hoover, 2008; Lester and Dassonville, 2011) subjects memorized the potential cue positions during a primary training period or behavioral calibration (i.e., equivalent to presentation of the RA in the present experiment). Later in the experiment or later in the trial subjects were asked to conduct reaches or saccades in which the egocentric encoding of the cue location was sufficient to solve the task. When subjects did not need to use the memorized positions to determine the target, no IRE was observed for immediate responses. But in such a task design, egocentric and allocentric references are aligned and the task-irrelevant visual frame is off-set equally in both reference frames. Therefore, even previous IRE task designs which required subjects to conduct a movement directly aiming at the target position, can in principle be consistent with an allocentric cause. Egocentric and allocentric representations of space are present in parallel and both types of information are usually used for more accurate behavior (Burgess, 2006; Byrne and Crawford, 2010). It has been shown that egocentric spatial memory is short lasting, putatively because it has to provide mainly real-time representation of the environment for direct interaction with objects (Hay and Redon, 2006; Chen et al., 2011). The fact that in previous task designs IRE was observed after a certain delay could be attributed to the interaction of shortlasting egocentric and long-lasting allocentric spatial representations. When the same subjects were exposed to a symbolic version of the task in which they had to use the memorized reference positions for a verbal response (to compare the position of visual cue with the memorized array of reference positions and report which one was cued), then the IRE was present even in immediate responses (Bridgeman et al., 1997; de Grave et al., 2002; Dassonville and Bala, 2004b). We argue that the verbal report of cue position required subjects to encode the cue relative to the RA hence mandated the use of an allocentric reference frame. It is therefore possible that even in previous IRE experiments the allocentric offset of the frame was the source of the illusion.

We suggest that the IRE in our reach task at least partially depended on allocentric encoding of space. Our present experimental design required subjects to follow an object-centered, hence allocentric, movement planning strategy. For proper interpretation of the IRE it is necessary to distinguish different phases of the trial when discussing reference frames. In the context of our task, at least the following spatial parameters are of interest: (i) the ego- or allocentric position of the (memorized) RA; (ii) the egocentric position of the frame relative to the bodymidline; (iii) the allocentric position of the frame relative to the RA; (iv) the allocentric position of the cue relative to the memorized RA; and (v) the allocentric target position relative to the DA. Experiment II showed that the IRE was determined by the allocentric frame position relative to the RA, not the frame's egocentric position. Thus, the IRE had an allocentric cause in our case. The consequence of this original allocentric cause needs to survive or be inherited by subsequent spatial encoding steps in order to affect the final motor behavior. The question is, which spatial encoding mediates the originally allocentric effect to finally become apparent in allocentric reach behavior? We ruled out a shifted SSA above. Previous studies showed that the memorized location of the reference object is shifted by the frame stimulus (Dassonville and Bala, 2004b). In our case this subjective shift of the RA would be sufficient to explain the results. The subjects need to encode the cue relative to the RA and later compute the target as the corresponding position on the relocated reference object (DA). Hence, a shifted RA translates into an erroneous allocentric cue position, and this in turn translates into an erroneous allocentric target position, and finally into an erroneous reach. Whether the memorized RA itself is encoded in an egocentric reference frame (e.g., relative to direction of gaze or body midline) or in an allocentric reference frame (e.g., relative to the surrounding screen frame) does not matter for the outcome. Both are possible and our experiment did not dissociate these alternatives.

## **EXPANSION OF MEMORIZED VISUAL SPACE**

In all previous IRE studies, an underestimation of target eccentricity was reported along with a significant systematic mislocalization of the target. This can be seen by the fact that movement endpoint position as a function of cue position has a slope smaller than unity. The present results (**Figures 3** and **5**) also show underestimation of target eccentricity (*p*experiment I < 0.0001 and *p*experiment II < 0.0001, one tail *t*-test on the slope of the nine regression lines fitted separately to the population response for different DA and frame positions in experiments I and six regression lines fitted separately to the population response for different RA and frame positions in experiments II). In contrast to previous reports, subjects underestimated the object-centered eccentricity of the cue or target (i.e., laterality of the cue/target relative to the center of the reference/DA). Underestimated eccentricity can be viewed as an apparent compression of the movement space. Yet, when in a previous study subjects were asked to make saccadic eye movements to memorized reference locations, the apparent compression turned out to be a result of expansion of the spatial memory of potential target positions (Dassonville and Bala, 2004a). Our observed underestimation of eccentricity adds to previous findings by showing that expansion of memorized visual space occurs in the behaviorally relevant reference frame, here centered on the object.

# **PERCEPTION VS. ACTION**

We do not argue that the behavioral report via an allocentric reach is necessarily substantially different from IRE tasks with symbolic encoding of the target, e.g., by key-presses or verbal report. The underlying mechanism of the IRE for this class of tasks, which previously were labeled "perceptual", might be identical or at least overlap. Accordingly, previous lines of argumentation based on a perception-action model might also account for our data (see also Section Discussion on ventral and dorsal stream processing below). In this case, we would have to assume that the memorized RA underwent a "perceptual" shift due to the context stimulus (Dassonville and Bala, 2004b) with the consequences discussed in the above paragraph. Whether allocentric reach planning and perceptual encoding in the context of such tasks can at all be meaningfully distinguished, remains an open question. We find it noteworthy, though, that the congruent condition of the allocentric task (Experiment I, congruent trials) and the egocentric control condition (Experiment Ib and previous studies), were equivalent in terms of spatial layout, timing of stimuli, and manual response mode, and only differed in the task context requiring allocentric reach target selection. In terms of spatial layout, the equivalency refers to the fact that in congruent trials, the allocentric and egocentric spatial location of the cue (the dot which is presented with the frame) and the target (final reach goal) are identical. In terms of timing, equivalency refers to the fact that in both experiments subjects receive the acoustic gosignal soon after the presentation of the cue plus frame stimulus and faster than typical manual response times would require. In this sense the immediacy of the movement is given in both experiments. The task context was not provided by the congruent trials themselves but rather by the interspersed incongruent trials which requested subjects to encode the cue relative to the RA rather than according to their liking. If the congruent condition would have been predictable, the congruent trials could have been solved with egocentric encoding of the cue and target. This rendered the allocentric congruent trials, which showed an IRE, quite similar to the egocentric trials, which did not show an IRE. This means that spatial task context was enough to make shortlatency reaches, which share many properties of typical "action" tasks, prone to IRE.

The results of experiment I differ from a recent study on IRE with an allocentric task in which the stimuli defining the allocentric reference frame were shown simultaneously with the off-center context stimulus, and no visual cue was shown to instruct the target (Dassonville and Bala, 2004b). The reach target was inferred in an allocentric reference frame as the fourth corner of a rectangle, while the other three corners were presented within a visual frame stimulus shifted laterally relative to the subject's mid-sagittal plane. The pattern of observed target errors was similar to previous IRE experiments, with no effect for immediate responses and a significant effect for delayed responses. When the reference stimuli were shown together with an off-center frame, one had to expect that they will be subject to an IRE themselves (Lester and Dassonville, 2011) and the mislocalization of the target, which has to be inferred from the affected reference objects, is then a secondary effect without an IRE on the allocentric space representation itself. These results were used to argue against separate cognitive and sensorimotor visuospatial representations, and were instead explained with the biased-midline hypothesis, i.e., by an egocentric cause, an explanation that does not work for our data.

Taken together, we conclude that in our task, which required reach planning in an allocentric reference frame, the IRE was caused by an allocentric space representation and mediated via a distorted visual memory of the reference object. This may also have been the case in previous Roelofs experiments. It cannot be ruled out that an egocentric mislocalization of the memorized RA gave rise to the allocentric mislocalization of the visual cue, but it can be ruled out that the original cause for the mislocalization was a shift of the SSA direction or any other egocentric reference frame.

# **VENTRAL VS. DORSAL VISUAL STREAMS**

According to the perception-action model (Goodale and Westwood, 2004; Goodale et al., 2004), egocentric references support visually guided actions through the dorsal sensorimotor stream in the posterior parietal cortex, while allocentric encoding of spatial locations can be predominantly found in the ventral stream supporting perception. According to this view, the dorsal stream is required and capable of making use of allocentric information from the ventral stream in the case of memory guided movements, e.g., IRE pointing tasks with long delays (Bridgeman et al., 1997; Goodale and Westwood, 2004; Milner and Goodale, 2008). In terms of the short-latency manual interaction with the visual target stimulus, our task would have to be considered a typical "action" task, hence should be attributed to dorsal stream processing. But according to the perception-action model, the allocentric spatial task constraints in our task also require ventral stream input. The model does not provide threshold values of how quickly ventral stream information can become accessible to dorsal stream processing. But in previous experiments, the required delays in target-aiming pointing, reaching, or saccade tasks ranged in the order of seconds before an IRE became apparent, suggesting a very slow transfer of information between ventral and dorsal stream in IRE tasks. If the model does account for our data, then our results suggest that the use of allocentric ventral-stream information by dorsal stream visuomotor processing can occur much faster than thought from previous IRE experiments. Such fast transfer is also suggested by a recent behavioral study (Thaler and Goodale, 2011b) which showed that reaction times in allocentric movements are 30–40 ms slower than egocentric movements, a finding that is reminiscent of behavioral and neural response delays in posterior parietal cortex during stimulus-response incongruent reach tasks (Gail and Andersen, 2006; Westendorff et al., 2010; Westendorff and Gail, 2011).

Slow brain imaging techniques cannot resolve the issue of whether such short-latency ventral-to-dorsal information transfer occurs, but experimental results have repeatedly pointed to overlapping structures for egocentric and allocentric encoding in the dorsal stream (Galati et al., 2000; Zaehle et al., 2007; Thaler and Goodale, 2011a; Gallivan et al., 2013). From our own previous neurophysiology work, we know that posterior parietal cortex encodes memory-guided anti-reach goals, which are independent of immediate visual input and independent of visual memory, with a delay of roughly 200 ms relative to visual cue onset, and roughly 100 ms relative to the visually selective neural response onset in the same area (Gail and Andersen, 2006; Gail et al., 2009; Westendorff et al., 2010). From the above discussion, we expect similar latencies for allocentric encoding in the posterior parietal cortex in the context of the current task.

The extent to which the perception-action model is valid is an ongoing debate in visual and visuomotor neuroscience. Growing evidence from behavioral and neurophysiology studies challenges the strictly separated vision-for-perception and visionfor-action theory (see Schenk et al., 2011 for review). The most compelling evidence for this model was patient D.F., who has bilateral damage to the ventral stream. D.F. failed in visual perceptual tasks while her visuomotor performance was not impaired (Milner et al., 1991). A recent study (Schenk, 2006) revealed that the discrepancy in her behavior was not due to different response modes, but rather due to deficits in different spatial representations (Himmelbach et al., 2012). The study showed that her perceptual performance was as good as her visuomotor performance when the perceptual task demanded egocentric spatial encoding whereas she failed in perceptual tasks which required object-based (allocentric) spatial encoding. Further behavioral support for the perception-action model was provided by a substantial body of research exploring visual illusions in perceptual and motor tasks where unlike perceptual responses, immediate motor responses seemed to be robust to the erroneous effects of spatial contextual information (for recent reviews see Schenk et al., 2011; Westwood and Goodale, 2011). However, in more controlled experimental conditions, contextual information can similarly affect perceptual and motor responses (Glover, 2004; Coello et al., 2007; Neely et al., 2008; Schenk et al., 2011). Therefore, based on our IRE for short-latency reaches, we suggest that the differential effect of spatial contextual information on sensorimotor behavior as explained by the perception-action model might not primarily be a question of perceptual vs. actionlike behavioral response mode, but rather a question of the spatial task demands.

#### **ACKNOWLEDGMENTS**

We thank Diemut Regel and Katharina Wörgötter for help in data collection, Sina Plümer and Klaus Heisig for administrative and technical support. This work was supported by the Federal Ministry for Education and Research (BMBF, Germany, grants, 01GQ0814 and 01GQ 1005C), the German Research Foundation (DFG, grant SFB-889), and the State of Lower Saxony (grant VWZN2563).

# **REFERENCES**


predict upcoming motor actions. *Eur. J. Neurosci.* 38, 2408–2424. doi: 10.1111/ ejn.12215


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 08 April 2014; accepted: 12 August 2014; published online: 29 August 2014*. *Citation: Taghizadeh B and Gail A (2014) Spatial task context makes shortlatency reaches prone to induced Roelofs illusion. Front. Hum. Neurosci. 8:673. doi: 10.3389/fnhum.2014.00673*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Taghizadeh and Gail. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

# An investigation of the neural circuits underlying reaching and reach-to-grasp movements: from planning to execution

# *Chiara Begliomini 1\*, Teresa De Sanctis1, Mattia Marangon1, Vincenza Tarantino1, Luisa Sartori 1, Diego Miotto2 , Raffaella Motta2 , Roberto Stramare2 and Umberto Castiello1*

<sup>1</sup> Department of General Psychology and Center for Cognitive Neuroscience, University of Padova, Padova, Italy <sup>2</sup> Department of Medicine, University of Padova, Padova, Italy

*Edited by:* Simona Monaco, York University, Canada

#### *Reviewed by:*

Chiara Bozzacchi, Istituto Italiano di Tecnologia, Italy Jason P. Gallivan, Queen's University, Canada

#### *\*Correspondence:*

Chiara Begliomini, Department of General Psychology and Center for Cognitive Neuroscience, University of Padova, Via Venezia 8, 35131 Padova, Italy

e-mail: chiara.begliomini@unipd.it

Experimental evidence suggests the existence of a sophisticated brain circuit specifically dedicated to reach-to-grasp planning and execution, both in human and non-human primates (Castiello, 2005). Studies accomplished by means of neuroimaging techniques suggest the hypothesis of a dichotomy between a "reach-to-grasp" circuit, involving the anterior intraparietal area, the dorsal and ventral premotor cortices (PMd and PMv – Castiello and Begliomini, 2008; Filimon, 2010) and a "reaching" circuit involving the medial intraparietal area and the superior parieto-occipital cortex (Culham et al., 2006). However, the time course characterizing the involvement of these regions during the planning and execution of these two types of movements has yet to be delineated. A functional magnetic resonance imaging study has been conducted, including reach-to-grasp and reaching only movements, performed toward either a small or a large stimulus, and Finite Impulse Response model (Henson, 2003) was adopted to monitor activation patterns from stimulus onset for a time window of 10 s duration. Data analysis focused on brain regions belonging either to the reaching or to the grasping network, as suggested by Castiello and Begliomini (2008). Results suggest that reaching and grasping movements planning and execution might share a common brain network, providing further confirmation to the idea that the neural underpinnings of reaching and grasping may overlap in both spatial and temporal terms (Verhagen et al., 2013). But, although responsive for both actions, they show a significant predominance for either one of the two actions and such a preference is evident on a temporal scale.

**Keywords: reach-to-grasp, reaching, functional magnetic resonance imaging, motor planning, motor execution**

# **INTRODUCTION**

The reach-to-grasp movement has been investigated from many perspectives and through different approaches given that it represents an ideal experimental windowto elucidate action–perception interactions. Studies centered on motion analysis of grasping have shown that the final posture of hand and fingers in contact with the object represents the end result of a motor sequence starting well ahead of the action of grasping itself (Jeannerod, 1984; Gentilucci et al., 1991; Jakobson et al., 1991; Chieffi and Gentilucci, 1993). The progressive shaping of hand and fingers is accomplished through a progressive opening of the grip with straightening of the fingers, followed by a closure of the grip until the size of the object is perfectly matched. The point in time where grip size is the largest (maximum grip size) is a clearly identifiable landmark that occurs well before the fingers come into contact with the object (Jeannerod, 1984). Many studies have showed that even very subtle changes in object properties can result in a significant change in grasping kinematic parameters (see Smeets and Brenner, 1999, for a review). The susceptibility of kinematic parameters demonstrates how sensitive and sophisticated are the processes responsible for the "translation" of object properties into the motor program implemented during the "hand preshaping" stage are.

In neural terms, neurophysiological studies in non-human primates have demonstrated that reaching and grasping movements, even if embedded in the same act, are coded by different neural systems. Computations regarding the grasp component seem to occur within a lateral parietofrontal circuit involving mainly the anterior intraparietal area (AIP) and both the dorsal (PMd) and the ventral (PMv) regions of premotor areas (Moll and Kuypers, 1977; Godschalk, 1991; Raos et al., 2004). Computations regarding the reaching component, instead, seem to occur within a more medial parieto-frontal circuit including the medial intraparietal area (mIP) at the boundaries with area V6A (Andersen and Cui, 2009), and the PMd (Sakata and Taira, 1994).

Neuroimaging and transcranial magnetic stimulation (TMS) studies in humans go in the same direction (for review see Castiello, 2005; Culham et al., 2006; Olivier et al., 2007; Begliomini et al., 2008). Several studies agreed on the key role played by the human AIP (hAIP) in grasping behavior (Grafton et al., 1996; Dohle et al., 2000; Culham et al., 2003, 2006; Frey and Gerry, 2006; Begliomini et al., 2007a, 2008; Filimon, 2010) and it has also been proposed the inferior frontal gyrus (IFG) and the dorsal part of the middle frontal gyrus (MFG) at the boundaries with the precentral gyrus (PreCG) as the human homologs of monkey F2 and

F5 (Davare et al., 2006; Begliomini et al., 2007b, 2008). Rather, a pathway including the superior part of the parieto-occipital cortex (SPOC), the medial intraparietal area (mIP) and the PMd has been suggested as the neural substrate for planning and execution of reaching movements (Connolly et al., 2003; Prado et al., 2005; Culham et al., 2006; Cavina-Pratesi et al., 2010; Filimon, 2010; Vesia and Crawford, 2012).

The dichotomy between a lateral fronto-parietal network supporting grip formation and a medial fronto-parietal network being the neural underpinning of reaching has recently been put into question. Evidence from single-cell data (Raos et al., 2004; Fattori et al., 2009, 2010) and lesion studies (Battaglini et al., 2002) suggests that the parieto occipital area V6a and dorsal premotor area F2 are also involved in managing specific aspects of grasping behavior such as grip posture and wrist orientation. For example, reaching-related neurons in macaque area V6A appear to be sensitive not only to reach direction (Fattori et al., 2004), but also to target orientation (Galletti et al., 1999; Fattori et al., 2009), target shape (Fattori et al., 2012), and grasp configuration (Fattori et al., 2010). Similarly, functional magnetic resonance imaging (fMRI) investigations in humans reported grasping-related parieto-occipital and dorsal premotor cortex activations (Chapman et al., 2002; Begliomini et al., 2007a,b, 2008; Gallivan et al., 2011), which might be considered the possible human homolog for monkey areas V6A and F2, respectively. Moreover, a recent neuroimaging study, based on the effective parieto-frontal connectivity, argues against the existence of dedicated circuits for reaching and grasping (Grol et al., 2007). The results of this study show that while grasping large objects increases connectivity among areas belonging to the dorso-medial circuit, grasping small objects increases inter-regional couplings mainly within the dorsolateral circuit: however, a certain degree of overlap between the two circuits was observed. Along the same line, a recent multi-voxel pattern analysis (MVPA) study provides further evidence against a segregation of reaching and grasping circuits, showing that both grip types and reach direction are coded within the inferior portion of the dorsal premotor cortex (iPMd), PMv, AIP, primary motor (M1), somatosensory (S1) cortices, and the anterior superior parietal lobe (SPLa – Fabbri et al., 2014).

Overall, these findings indicate that in humans, like in monkeys, reach-to-grasp movements involve a large network of interconnected structures in the parietal and frontal lobes (Rizzolatti and Luppino, 2001; Brochier and Umiltà, 2007; Castiello and Begliomini, 2008). However, how these areas interact has yet to be fully clarified. The majority of studies has focused on the question of "if" or "how" these areas interact during grasping or reaching execution, neglecting the possibility that interaction patterns could change across time, according to action stages (Verhagen et al., 2013).

In this respect, the functional distinction between the premovement planning and the control stages of action has been the subject of much investigation (e.g., Woodworth, 1899; Vince, 1948; Fitts, 1954; Keele, 1968; Beggs and Howarth, 1970, 1972; Carlton, 1981; Meyer et al., 1988; see Glover, 2004 for a review). And the existence of these two stages has generally become accepted as an underlying principle of human motor

behavior (Jeannerod, 1988; Rosenbaum et al., 1990; Rosenbaum, 1991).

In neural terms, the functional distinction between planning and execution has been investigated in a variety of studies (e.g., Grol et al., 2007; Bozzacchi et al., 2012; Glover et al., 2012). Grol and colleagues used Dynamic Causal Modeling (Friston et al., 2003) on fMRI timeseries acquired during planning and execution of visually guided reaching-to-grasp movements toward objects of different size to explore the interregional couplings between regions of the dorsolateral (AIP and PMv) and the dorsomedial (V6A and PMd) circuits. By assessing how different hand–object interactions modulated the effective connectivity within these networks, they demonstrated that the involvement of the dorsolateral and dorsomedial parieto-frontal circuits is largely related to the degree of online control required by the prehension movement (Grol et al., 2007).

Another study provides an attempt to contrast activity related to planning and online control in the human brain during simple reaching and grasping movements (Glover et al., 2012). These findings provide evidence that the planning and control of even simple reaching and grasping actions use different brain regions, including different parts of the frontal and parietal lobes. Movement planning determined activity within the superior temporal sulcus (STS), the pre-supplementary motor area (pre-SMA), the mIP, the SPOC, the PM, and the insula. Movement execution, instead, seems to be supported mainly by the sensorimotor cortex, the cerebellum, the SMA, the supramarginal gyrus (SMG), and the superior parietal lobe (SPL).

Pre-movement cortical activity related to reaching and grasping tasks has also been studied by means of motor-related cortical potentials (Bozzacchi et al., 2012). In this study, different activity patterns in terms of onset, amplitude, duration, and sources were recorded in the preparation phase according to the specific action. The results indicate the presence of parietal activity, well before the action begins, for goal-oriented actions such as grasping an object but not in reaching. This activity starts about two seconds prior to the action and is maximal about one second later in the areas contralateral to the used hand. Moreover, the type of action to be performed also modulates motor preparation in terms of timing and intensity of the different brain activity.

Along these lines, we hypothesized that (i) action planning might be characterized by a prominent contribution of decisionrelated areas, in charge of choosing the grasping schema to be implemented according to object properties, position, and action goals. Differently, action execution might be characterized by a larger contribution of motor-related areas. In addition (ii) we aimed to disentangle interactions between dorsolateral and dorsomedial circuits not only during the actual execution of reaching and grasping movements, but also during their planning. Finally (iii) concerning grasping, we hypothesized that different grasping schemata (e.g., precision grip and whole hand grasp) could be characterized by different neural underpinnings during both movement planning and execution.

To test these hypotheses, we instructed participants to perform reaching or grasping movements, toward a small or a large spherical object, while lying in a magnetic resonance (MR) scanner. Action stages (planning and execution) were distinguished and segregated by acoustic cues presented through headphones. To monitor temporal dynamics of interaction patterns within the fronto-parietal network a Finite Impulse Response (FIR – Henson, 2003) model was adopted for BOLD signal modeling. Importantly, with respect to previous studies we subdivided the time course of activation to determine brain activity related to the pre-movement planning and online control of reaching and grasping in humans. Prior to movement initiation, planning is entirely responsible for the initial determination of all movement parameters, and continues to be highly influential early in the movement. As movements progress, however, the influence of control on the spatial parameters of the action increases. Can such a gradual crossover between planning and control systems being evident through the temporal unfolding of neural activity?

# **MATERIALS AND METHODS**

## **PARTICIPANTS**

Eighteen volunteers (six men, 12 women, range 20–31 years old) participated in the study. All participants fulfilled the inclusion criteria suggested by the Italian Society of Medical Radiology, none had a history of neurological, major medical, or psychiatric disorders. They were all right-handed according to the Edinburgh Handedness Inventory (Oldfield, 1971). Experimental procedures and scanning protocols were approved by the University of Padua Ethics Committee and conducted in accordance with the Declaration of Helsinki (Sixth revision, 2008). All participants gave their informed written consent to participate in the study.

## **TASK AND STIMULI**

Three dimensional (3D) stimuli were presented by means of anMR compatible motorized circular rotating table (ABRAM1; **Figure 1**). The experimental stimuli consisted of two wooden spheres of different dimensions (a small wooden sphere of 3 cm diameter and a large wooden sphere of 7 cm). Participants were requested to perform two different kinds of movement: (i) reach toward and grasp the stimulus; (ii) reach the stimulus with the hand in a fist posture. All participants naturally adopted a precision grip, the opposition between the index finger and thumb to grasp the small stimulus, and whole hand prehension in which all fingers were opposed to the thumb to grasp the large stimulus. During movement execution, participants were requested to keep the eyes on the stimulus. To facilitate direct viewing of the stimulus the head was tilted (10–15◦) by means of foam MRI compatible cushions. Given that participants performed the actions with the right hand, a further cushion was placed under the upper right arm, in order to minimize discomfort during the movement. Trials structure was the following: (i) an acoustic cue delivered through MR-compatible headphones indicated the type of movement to perform. A single tone indicated a reach to grasp movement (duration 300 ms; frequency 1600 Hz); a double pulse tone indicated a reaching only movement (each pulse lasted 70 ms with a frequency of 400 Hz). The interval between the two pulses was of 60 ms and the total

duration of the tone was 200 ms; (ii) the acoustic cue was followed by a 2 s delay; and (iii) a "go" signal was presented (a whistle of 200 ms duration; frequency 440 Hz). Participants were requested to wait for the "go" signal to begin the movement indicated by the acoustic cue. Participants were trained to familiarize with the acoustic instructions during a training session before scanning. They were requested to perform the movement at a natural speed.

## **EXPERIMENTAL DESIGN**

The entire experiment consisted of four runs of 45 trials each, in which stimulus size (small, large) was randomized across runs and type of movement (grasping, reaching) was randomized within runs. Therefore the design (factorial 2 × 2) included four experimental conditions: reach to grasp toward a small stimulus (GS), reach to grasp toward a large stimulus (GL), reaching only toward a small stimulus (RS), reaching only toward a large stimulus (RL). Since stimulus size was randomized across runs, for each run two movements had to be performed, either grasping or reaching. A mixed design was adopted, grouping trials belonging to the same type (grasping or reaching) in short sequences of different numerosity (varying from 3 to 5 trials of the same type). This approach has been adopted on one hand in order to control for predictability phenomena, possibly induced by trials sequences of constant length. On the other hand, continuous changes in task request (e.g., RS-GS-RS-GS-RS and so on) could have resulted in task-switching related activity. Variable interstimulus interval (ISI) was considered, including durations from 3 to 6 s according to a long exponential probability distribution (Dale, 1999; Hagberg et al., 2001). ISI duration was independently randomized within each single experimental run.

## **DATA ACQUISITION**

The experiment was carried out on a whole body 1.5 T scanner (Siemens Avanto) equipped with a standard Siemens

<sup>1</sup>http://www.ab-acus.com/products.html

eight channels coil. Functional images were acquired with a gradient-echo, echo-planar (EPI) T2∗-weighted sequence in order to measure blood oxygenation level-dependent (BOLD) contrast throughout the whole brain (37 contiguous axial slices acquired with descending interleaved sequence, 56 × 64 voxels, 3.5 mm × 3.5 mm × 4.0 mm resolution, FOV = 196 mm × 224 mm, flip angle = 90◦, TE = 49 ms). Volumes were acquired continuously for each run with a repetition time (TR) of 3 s; 102 volumes were collected in each single scanning run, resulting in functional runs of 5 min and 25 s duration (21 min and 40 s of acquisition time in all). High-resolution T1-weighted image were acquired for each subject (3D MP-RAGE, 176 axial slices, no interslice gap, data matrix 256 × 256, 1 mm isotropic voxels, TR = 1900 ms, TE = 2.91 ms, flip angle=15◦).

#### **DATA ANALYSIS**

Data preprocessing and analysis were performed using SPM8 (Statistical Parametric Mapping, Wellcome Institute of Cognitive Neurology, London, UK) implemented in MATLAB 7.5.0 environment (MathWorks, Natick, MA, USA). For each participant, the first two volumes of each fMRI run were discarded because of the non-equilibrium state of the magnetization in order to allow for stabilization. ArtRepair toolbox (ArtRepair software Package, for SPM2) was adopted in order to correct for possible images corruption due to signal spikes induced by head motion. Motion correction was carried out by realigning and unwarping data. Structural images were segmented and subsequently the image of gray matter was co-registered with all the functional images. Structural and functional images were then normalized adopting the template provided by the Montréal Neurological Institute (MNI) implemented in SPM8. Finally, functional images were spatially smoothed using a 7 mm × 7 mm × 8 mm full-width-at half-maximum (FWHM) Gaussian Kernel. At the end Artrepair toolbox was applied in order to identify and correct large scanto-scan head motion, which may result in large global intensity changes. First-level analysis was carried out by adopting an FIR (Henson, 2003), in order to characterize the temporal evolution of the hemodynamic response (HR) without *a priori* hypothesis on its shape The peculiarity of the FIR model is the absence of assumptions about the shape of the HR: this feature allows for the splitting of a selected post-stimulus time window into different temporal segments (a number of successive Time Bins (TB) by providing a set of basis functions within the framework of a general linear model (GLM). These basis functions can be considered as separate parameters (Dale and Buckner, 1997) and can be entered into the GLM model with time as a factor (Henson, 2003). According to this model, task-related BOLD variations were monitored from stimulus onset (cue signal), in order to capture BOLD variations related to both action planning (cue) and execution (go). In this respect, a simple canonical HRF model would have been not appropriate to capture signal variations related to all action stages: the structure of the trial includes different action stages lasting for a prolonged time (cue-go interval of 2 s plus action occurring thereafter). From this perspective the FIR model provides a more sensitive and detailed signal modeling, allowing

for a monitoring of BOLD variations related to all trial stages. A post-stimulus time window of 10 s length was considered, starting from cue onset, and divided into 10 TB of 1 s each. TB width was lower than the TR used during data acquisition (3 s) because we attempted to specifically characterize differences at the subsequent stages of action planning and execution. In addition, it has also been shown that it is possible to sample the impulse response at post-stimulus intervals shorter than TR by jittering event onsets with respect to scan onsets (Josephs et al., 1997; Schilbach et al., 2008). In our study interstimulus interval varied from 3 to 6 s and had a jittered distribution. Reaching (RS and RL) and grasping (GS and GL) movements were modeled as separate events for each participant. Errors in action execution or missed trials were modeled as separate regressors of no interest. T-contrasts were computed for each condition (RS, RL, GS, and GL), in order to capture condition-specific HR variations for each condition in single TB (10 images per condition in all). Image analyzes were carried out after high-pass filtering (154 s) to remove subject-specific, low-frequency signal drifts and global intensity scaling. Following the estimation of a GLM for each single participant, effects for each experimental condition were tested by applying appropriate linear contrasts to the parameter estimates, resulting in a *t*-statistic for each voxel (SPMt). Images for each experimental regressor/condition were entered in a second level random effect analysis (RFX) allowing for inference to the general population, with type of movement (reaching, grasping) and stimulus dimension (small, large) as factors across the considered TBs (2 × 2 × 10; 40 images in all for each participant). With the purpose of clearly localize the neural substrates underlining the proposed reach to grasp or reaching only tasks, the analysis was conducted by adopting a searching mask built by several regions of interest, on the basis of available literature (for review see Castiello and Begliomini, 2008), suggesting a primary distinction between planning and execution-related areas. According to this distinction, the dorsolateral region of the prefrontal cortex (Rizzolatti and Luppino, 2001) and the anterior cingulate area (Matelli et al., 1991) would be mainly involved in movement planning, while the primary motor (Glover et al., 2005; Tunik et al., 2005; Rice et al., 2006) and premotor cortices (Culham et al., 2003; Frey et al., 2005; Begliomini et al., 2007b), as well as the parietal cortex (Binkofski et al., 1998; Culham et al., 2006; Begliomini et al., 2007a) would play a substantial role in action execution. In addition, also the SPL was included, as a brain region known to be involved in reaching control (Culham et al., 2003). The toolbox WFU PickAtlas (Wake Forest University3) was adopted to build the mask involving all the mentioned areas.

# **RESULTS**

## **GLOBAL ANOVA**

The interaction between type of movement, stimulus dimension and TBs was significant for several portions of the considered mask (see **Table 1**). Results are 0.001 uncorrected for multiple comparisons (*k* ≥ 20). This analysis underlined that the PreCG (Brodmann Area, BA 6) in the right hemisphere, and the inferior

<sup>2</sup>http://www.fil.ion.ucl.ac.uk/spm/ext/#ArtRepair

<sup>3</sup>www.ansir.wfubmc.edu


**Table 1 | Brain regions showing interaction effects between type of movement (grasping, reaching) and stimulus dimension (small, large) across all 10 Time Bins.**

Results are obtained by means of a RFX analysis. Coordinates refer to the Montréal Neurological Institute (MNI) stereotaxic space. p values are uncorrected for multiple comparisons (0.001, k ≥ 20). BA, Brodmann area.

parietal lobule (IPL, BA 40) together with the anterior cingulate cortex (aCC) exhibited significant effects.

To better characterize our results, and in order to elucidate the possible evolution of interaction patterns across the whole poststimulus window (10 s), separate ANOVA were conducted for each TB, considering type of movement (grasping; reaching) and stimulus dimension (small; large) as factors.

#### **SINGLE BIN ANOVA**

Statistical threshold was set to *p* < 0.001, uncorrected for multiple comparisons and the adopted cluster extension was set to *k* ≥ 12.

### *TB 1–3*

Random effect analysis performed on TB 1, 2, and 3 did not bring to any significant result – neither main nor interaction effects.

#### *TB 4*

The interaction between type of movement and stimulus dimension was significant for the IPL bilaterally (BA 40), within both anterior and posterior sector of the right intraparietal sulcus [aIPS: *F*(1,68) = 21.36, and pIPS: *F*(1,68) = 15.72, respectively], and the left aIPS [*F*(1,68) = 21.52] and the aCC bilaterally [BA 32; left side: *F*(1,68) = 19.92; right side: *F*(1,68) = 16.09]. A close inspection of the interaction effects revealed a similar pattern of results for all the considered regions, that is RL determined a higher level of activation than RS; vice versa, GS seems to be associated with a higher level of activity than GL. *Post hoc* contrasts revealed that only the comparisons GS > RS and GS > GL were significant. In detail, the contrast GS > GL became significant only within the left aIPS, whereas the comparison GS > RS reached significance in all areas showing interaction effects (see **Table 2** and **Figure 2**). No further significant effects were observed.

## *TB 5*

The interaction between type of movement and stimulus dimension was significant for the right MFG [MFG, BA 6; *F*(1,68) =20.72] and the aCC (BA 32) bilaterally [right: *F*(1,68) = 18.57; left: *F*(1,68) = 17.70]. In both circumstances RL determined a higher level of activity than RS. Conversely, activity for GS was higher than that for GL. *Post hoc* comparisons revealed that only the contrast GS > RS and GL > GS reached significance. The contrast GS > RS led to significant differences in both the MFG and

the aCC, both in the left and the right hemisphere. The contrast GL > GS showed significant effects only within the left aCC (see **Table 2** and **Figure 3**). No further significant effects were observed.

#### *TB 6*

The interaction between type of movement and stimulus dimension was significant for the left aIPS [BA 40; *F*(1,68) = 16.31], the right pIPS [BA 40; *F*(1,68) = 19.84], and within the left middle cingulate cortex [mCC, BA 24; *F*(1,68) = 20.68]. For these regions, the level of activity was higher for RL than for RS. Conversely, GS was associated with a level of activity higher than that observed for GL. More in detail the difference between RL and GL became significant in all regions showing interaction effects (see **Table 2** and **Figure 4**). No further significant effects were observed.

## *TB 7*

The interaction between type of movement and stimulus dimension did reach significance within the left PreCG [BA 4: *F*(1,68) = 22.98; and 6: *F*(1,68) = 19.30] and the left MFG [BA 6; *F*(1,68) = 18.78]. Inspection of the interaction indicated that RS and RL were associated with a similar level of activation, while GL showed a signal level which was higher than that observed in GS. The contrast GL > GS was significant in both regions of the PreCG, while the comparison GL > RL underlined significant differences within the PreCG (BA 4) and the MFG. (see **Table 2** and **Figure 5**). No further significant effects were observed.

## *TB 8*

The interaction between type of movement and stimulus dimension was significant within three different sectors of the IPL corresponding to the lateral surface of the left IPL [*F*(1,68) =31.13], the left aIPS [*F*(1,68) = 28.91], and the left pIPS [*F*(1,68) = 23.80]. Inspection of the interaction patterns revealed a similar pattern of results for all regions, that is RL was associated with a higher level of activity than RS, and the level of activity for GS was higher than that found for GL. The contrast RL > RS was significant within both aIPS and pIPS, while the comparisons GS > GL underlined significant effects within IPL and aIPS. In addition, the contrasts GS > RS and RL > GL were significant in both sectors of the IPS (aIPS and pIPS – see **Table 2** and **Figure 6**). No further significant effects were observed.

**Table 2 | Brain regions showing interaction effects between type of movement (grasping, reaching) and stimulus dimension (small, large) distinguished in single Time Bins.**


Results are obtained by means of a RFX analysis. Coordinates refer to the Montréal Neurological Institute (MNI) stereotaxic space. p values are uncorrected for multiple comparisons (0.001, k ≥ 13). BA, Brodmann area.

## *TB 9 and 10*

Random effect analysis performed on TBs 9 and 10 did not bring any significant result.

# **DISCUSSION**

Neuroimaging investigations on grasping in humans have revealed similarities between human and non-human primates (Grefkes and Fink,2005). Both domains agree on the idea that both reaching and grasping, even if belonging to the same act, are supported by the recruitment of different brain regions. A more dorsomedial network, involving the SPOC, the mIP and the dorsal premotor cortex would mainly subserve the reaching component, while a more dorsolateral circuit, including the anterior intraparietal area (AIP) and the ventral premotor cortex would support visuomotor transformation and grip formation. However, this scenario has been put into question by some recent findings (Grol et al., 2007;

**type of movement and stimulus dimension in TB 4.** The p value is set to 0.001, uncorrected for multiple comparisons, cluster size k ≥ 13. White asterisks indicate significant effects for the contrast GS > RS; red asterisks

cortex; aIPS, anterior intraparietal sulcus; pIPS, posterior intraparietal sulcus; LH, left hemisphere; RH: right hemisphere; medial, medial view; lateral, lateral view).

Cavina-Pratesi et al., 2010; Fabbri et al., 2014) suggesting that both components could be supported by the same circuit, and that the distinction could take place more in temporal rather in qualitative terms. In other words, a common network would supply coding for both aspects, but at different time points. Our results seem to add further support to this view by demonstrating that several key areas belonging to the fronto-parietal network can play a different role according to the stage of the action.

# **ANTERIOR INTRAPARIETAL AREA**

Intraparietal area is considered the human homolog of monkey aIPS, a brain region involved in visuomotor transformation: this view has been supported by many neuroimaging findings (Frey et al., 2005; Shmuelof and Zohary, 2005; Begliomini et al., 2007a; Stark and Zohary, 2008; Filimon, 2010). The present findings confirm the role played by this area during the visuomotor processes underlying reach-to-grasp movement, but importantly they outline that the kind of computations ascribed to AIP varies as time unfolds. In fact, AIP seems to code for type of movement in TB 4 and 6, and for stimulus dimension in TB 8. TB 4 and 6 refer to 4

and 6 s after *stimulus* onset, and 2 and 4 s after *movement* onset, respectively. Therefore the observed effects might refer to planning rather execution processes, since we know that the maximum peak of the hemodynamic response is reached around 6 s. Differently, TB 8 (that is 8 s before *stimulus* onset and 6 s after *movement* onset) refers to a time point at which the hemodynamic response mainly reflects brain activity related to the execution rather than the planning phase. This scenario invites to make several considerations. Firstly, AIP begins to differentiate between movements rather early. Even if the hemodynamic response around the 4th second is still far from reaching its maximum, AIP already discriminates among conditions with different accuracy requirements. Along these lines, a recent evoked related potentials (ERPs) reachto-grasp study showed that processing occurring in AIP starts at the very early stages of action planning, when the translation of object representation into a motor program occurs (Bozzacchi et al., 2012; Verhagen et al., 2013; see also Tarantino et al., 2014). Secondly, during the planning and the execution stages, the role played by AIP seems to change: at the very beginning of action planning (TB4), AIP activity seems to be devoted to

**between type of movement and stimulus dimension in TB 5.** The p value is set to 0.001, uncorrected for multiple comparisons, cluster size k ≥ 13. White asterisks indicate significant effects for the contrast

GS > RS. (aCC, anterior cingulate cortex; MFG, middle frontal gyrus; LH, left hemisphere; RH, right hemisphere; medial, medial view; lateral, lateral view).

computations related to accuracy as witnessed by the fact that AIP activity is significantly higher for the GS than for the RS condition. In a later stage (TB6) RL activity is significantly stronger with respect to the GL condition; finally, during execution (TB8) effects observed in the left parietal region seem to suggest that AIP is coding for both accuracy in grip formation and spatial computing, necessary to approach the object with the right trajectory. A recent study on macaques (Lehmann and Scherberger, 2013) has indeed demonstrated that AIP contains different neuronal populations dedicated to either grip formation or spatial encoding. While neurons devoted to grip formation appear to be more active during action execution, neurons coding for spatial computing are active during both action planning and execution. Therefore we could expect that during planning AIP activity could reflect spatial processing rather than grip formation. Along this line, RL might require "more" spatial computing than GL, since the hand cannot count on palm and fingers to reach the goal, but just on hand knuckles. Therefore in this condition the spatial analysis necessary to support RL might require additional resources, as shown by the RL > GL effect in both planning and execution stages. Why we observe this effect for the large but not the small object might be due to the fact that GL and RL are physically distinct movement (GL involves palm and fingers, RL only the back of the hand). In comparison, GS and RS are more "similar" from a spatial point of view (GS involves only two fingers). We are aware that this hypothesis stems from neurophysiological data and would need further investigations in humans to be fully confirmed. However, at TB 8, that is during action execution, both GS and RL appear to be associated with significantly higher levels of activity than those noticed for RS and GL, respectively. Thirdly, stimulus dimension seems to play a significant role only at later stages, corresponding to action execution: the small stimulus seems to be associated with significantly stronger activity with respect to the large stimulus, but only for reach-to-grasp movements. This may suggest that during action executionAIP might be chiefly devoted to matters concerned with accuracy requirements related to the on-line control of a sophisticated grasping movement like GS. It is known that prehension of objects with small surfaces (relative to finger size) requires a larger degree of visual feedback (Bootsma et al., 1994), and that the kinematic profile of

**type of movement and stimulus dimension in TB 6.** The p value is set to 0.001, uncorrected for multiple comparisons, cluster size k ≥ 13. White

cingulate cortex; aIPS, anterior intraparietal sulcus; LH, left hemisphere; RH, right hemisphere; medial, medial view; lateral, lateral view).

the hand is disproportionally altered when grasping small objects without visual guidance (Chieffi and Gentilucci, 1993). Berthier et al. (1996) also showed that as visual information and object size decreased, subjects had longer movement times, slower speeds, and more asymmetrical hand-speed profiles. In line with previous evidence we suggest that, during the prehension of small objects, AIP activity could increase in order to transform object-centered target representations into motor space on the basis of incoming visual information of the moving arm (Grol et al., 2007). The emphasis here is on control, as the modulatory influences of object size on the dorsolateral circuit are related to the execution phase of the prehension movement.

Differently, during the execution of reaching movements AIP activity was higher for movements performed toward the large than the small object. This is a puzzling result given that evidence in humans indicates that the kinematical organization of reaching is affected by the precision requirements related to intrinsic features of objects such as size (MacKenzie et al., 1987; Gentilucci et al., 1991; Castiello, 2001). In this perspective we would have expected increased AIP activity as a reflection of the need for more on-line control required by reaching small objects. Although we do not have a firm explanation regarding this specific aspect of our results, it is worth clarifying that previous experiments in humans have employed a variety of tasks to investigate the neural correlates of reaching. They include reach-to-touch (Levy et al., 2007; Cavina-Pratesi et al., 2010), pointing (DeSouza et al., 2000; Astafiev et al., 2003; Connolly et al., 2003; Fernandez-Ruiz et al.,2007; Hagler et al., 2007), and joystick manipulation (Grefkes et al., 2004). These tasks differ widely in the extent of arm movement, purpose, and cortical recruitment (Culham and Valyear, 2006; Culham et al., 2006; Filimon et al., 2009). Therefore, we cannot exclude that adopting a different task might have brought to different outcomes.

Another aspect of the present findings worth mentioning is that in TBs 4 and 6 AIP involvement is bilateral. This might be due to a bidirectional crosstalk between the two homologous areas or, more simply, to the fact that in TB 4 and 6 the action has yet to be executed, participants could theoretically grasp or reach the object with either the left or the right hand (Binkofski et al., 1999; James et al., 2003; Culham and Valyear, 2006; Culham et al.,

**between type of movement and stimulus dimension in TB 7.** The p value is set to 0.001, uncorrected for multiple comparisons, cluster size k ≥ 13. White asterisks indicate significant effects for the contrast

GS > RS. (PreCG, precentral gyrus; MFG, middle frontal gyrus; BA, Brodmann area; LH, left hemisphere; RH, right hemisphere; medial, medial view; lateral, lateral view).

2006). The need for bilateral AIP contribution for hand shaping has been demonstrated by some previous findings (Culham et al., 2003; Ehrsson et al., 2003).

# **CINGULATE CORTICES**

Cingulate cortices (aCC and mCC) are known to play a fundamental role in decision making processes. This aspect is of particular interest since each object we interact with can be grasped in several ways (Fagg and Arbib, 1998; Rizzolatti and Luppino, 2001). The chosen grip depends on object visual properties, but also on object meaning and on what the agent wants to do with the object. In this perspective, the selection of one amongst the possible ways of grasping an object does not only rely on the visual intrinsic properties of the object, but also on action goals (Cohen and Rosenbaum, 2004; Ansuini et al., 2007, 2008). Therefore decisions regarding which motor program has to be implemented should occur before movement execution, that is during action planning. Accordingly, here we found that aCC (bilaterally) and mCC (left hemisphere) show a significant interaction effect between type of movement and stimulus dimension during movement planning (TBs 4, 5, and 6). To elaborate, in

TB 4 the aCC distinguishes among movements performed toward the small object, with higher levels of activity for GS than for RS. In TB 5 such difference persists, but also stimulus dimension appears to play a role for grasping movements. The level of activity for GS was significantly different from GL. At TB 6, the mCC shows higher levels of activity for RL rather than GL. These results agree with previous evidence indicating the aCC and mCC are regions involved in action selection (Lau et al., 2004; Cavina-Pratesi et al., 2010; Rowe et al., 2010). According to these studies, the aCC and mCC play a fundamental role in the selection among competing responses (movement schemata, in this case) together with other areas of the fronto-parietal network. In addition, as for AIP, activity within the CC seems to change during different stages of action planning: at a very early stage (TB 4 and 5) aCC seems to be responsible for choosing the most appropriate motor program on a more accuracy-based criteria: we know from previous studies that GS is usually associated with stronger activity in visuomotor related areas as well as longer reaction time suggestive of a more demanding planning phase. In TB 6 the mCC seems to be more engaged for the coding of type of movement as far as the large object is

**type of movement and stimulus dimension in TB 8.** The p value is set to 0.001, uncorrected for multiple comparisons, cluster size k ≥ 13. White asterisks indicate significant effects for the contrast GS > RS and RL > GL;

asterisks indicate significant effects for the contrast RL > GL. (IPL, inferior parietal lobule; aIPS, anterior intraparietal sulcus; LH, left hemisphere; RH, right hemisphere; medial, medial view; lateral, lateral view).

concerned (RL > GL). Although this result can sound a bit counterintuitive since the general agreement considers grasping more "demanding" than reaching, it is also known that the mCC is involved in the integration between the effector and the target during reaching planning (Beurze et al., 2007): the fact that the stronger activity is associated with the large object can be due to the larger amount of visuospatial information processing necessary when the target of the action is a large object (Tarantino et al., 2014).

Overall, the results concerned with CC activity seems to indicate that a more anterior sector of this regions is engaged in the processing of "accuracy" at the very early stages of action planning (TB 4 and 5), whereas a more posterior sector (mCC) seems to be more devoted at a later stage (TB 6) to spatial coding and the matching between effector and target.

## **MIDDLE FRONTAL GYRUS**

Interaction effects became significant in the MFG within the *right* hemisphere at TB 5 and within the *left* hemisphere in TB 7. The MFG belongs to the dorsal sector of the premotor cortices (PMd) and it is known to be involved in motor planning (Davare et al., 2007; Begliomini et al., 2007b, 2008; Fabbri et al., 2014; Tarantino et al., 2014). Interaction effects in TB 5 show greater activity for GS in respect to RS, while no effects are evident for movements performed toward the large object. This finding may reflect the need of higher levels of accuracy required by the planning and the subsequent execution of a precise grasping movement. The fact that only the ipsilateral MFG shows significant effects it is not surprising: several studies have advanced the role of the right PMd in online monitoring of action planning and execution, regardless of the side of the effector (Davare et al., 2006; Begliomini et al., 2008). At TB 7, when the action is about to start, it is the left MFG to show interaction effects between type of movement and stimulus dimension. This region of the MFG seems to be sensitive to stimulus dimension while grasping (GL > GS) but not while reaching objects. This pattern of results becomes significant when the target of the action is the large object, with higher levels of activity for GL with respect to RL. This seems to indicate that, while switching from planning to execution, the left MFG is significantly more alerted for the GL condition. Previous

results indicate that the MFG, together with aCC, could represent a part of the neural circuit supporting the selection for action (Lau et al., 2004). The fact that GL is associated with the strongest level of activity seems to suggest that grasping a large object might require additional control since all the fingers have to act in concert to achieve a hand posture suitable for grasping the large object.

A particular worth mentioning is the discrepancy in anatomical coordinates of the MFG in the right and the left hemisphere. More precisely, the MFG region showing significant effects in the right hemisphere (TB 5) appears to be more ventrally and anteriorly located in respect to the MFG regions showing significant effects in the left hemisphere (TB 7). However, when movements with the right hand are performed, MFG activity typically reflecting on-line monitoring is usually detected in regions more anteriorly located in respect to their homologous in the left hemisphere. However, the right MFG shows significant effects at TB 5 (action planning) while the left MFG appears to be significantly engaged only at TB 7 (action execution). The different stages of the action and the consequent different contribution of MFG to the ongoing process, together with the laterality of the effector used to perform the action might explain this anatomical discrepancy. Further studies are needed to confirm this result, especially in light of some very recent neurophysiological findings investigating the role of premotor cortices during the execution of a specific task and the refraining from performing it (Bonini et al., 2014). The study indicates that MFG seems to be involved in both situations, suggesting that this region encodes action representations also when the actions is not performed or delayed, which is actually the case of our paradigm (remember the 2 s delay before action initiation).

#### **PRECENTRAL GYRUS**

The PreCG hosts the primary motor cortex, the brain region controlling the execution of proximal and distal motor acts of our body. Here, we find significant interaction results within the left PreCG at TB 7, that is during motor execution. The pattern of activity within this region indicates that while reaching small or large objects does not lead to any difference, the act of grasping a large object leads to significant increases with respect to both reaching for the large object and grasping the small object. Similar findings have been reported in several previous studies (Begliomini et al., 2007a,b) and it is suggestive of a need for additional motor control to coordinate palm and fingers: in fact GL is the only condition in which fingers and palm have to be perfectly coordinated in order to acquire the right configuration as to hold the object.

#### **CONCLUSION**

We examined interaction effects between grasping and reaching movements performed toward small and large spherical objects within areas belonging to both the "reaching" and the "grasping" circuit. We observed that similar areas seem to be sensitive to both types of movements, providing further confirmation to the idea that the neural underpinnings of reaching and grasping may overlap in both spatial and temporal terms (Verhagen et al., 2013). However, from the results, it also emerges the possibility that, although responsive for both actions, they show a significant predominance for either one of the two actions and such a preference is evident on a temporal scale. Further studies are needed to better disentangle the temporal dynamics of medial and lateral pathways interactions, exploring patterns of functional and effective connectivity among these regions.

# **ACKNOWLEDGMENTS**

The work was supported by: FP7 Grant No. 287713; Ministero dell'Istruzione, dell'Università e della Ricerca (PRIN 2010–2011) Grant No. 2010XPMFW4 and Strategic Grant – University of Padova Grant No. STPD11B8HM\_003.

# **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 01 May 2014; accepted: 13 August 2014; published online: 02 September 2014.*

*Citation: Begliomini C, De Sanctis T, Marangon M, Tarantino V, Sartori L, Miotto D, Motta R, Stramare R and Castiello U (2014) An investigation of the neural circuits underlying reaching and reach-to-grasp movements: from planning to execution. Front. Hum. Neurosci. 8:676. doi: 10.3389/fnhum.2014.00676*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Begliomini, De Sanctis, Marangon, Tarantino, Sartori, Miotto, Motta, Stramare and Castiello. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Reach adaptation and proprioceptive recalibration following terminal visual feedback of the hand

# *Victoria Barkley1, Danielle Salomonczyk1, Erin K. Cressman2 and Denise Y. P. Henriques 1,3 \**

<sup>1</sup> Sensorimotor Control Lab, Centre for Vision Research, Department of Psychology, York University, Toronto, ON, Canada

<sup>2</sup> Sensorimotor Control Lab, School of Human Kinetics, University of Ottawa, Ottawa, ON, Canada

<sup>3</sup> School of Kinesiology and Health Science, York University, Toronto, ON, Canada

#### *Edited by:*

Gavin Buckingham, Heriot-Watt University, UK

#### *Reviewed by:*

Adrian Mark Haith, Johns Hopkins University School of Medicine, USA Krista E. Overvliet, VU University Amsterdam, Netherlands

#### *\*Correspondence:*

Denise Y. P. Henriques, Sensorimotor Control Lab, School of Kinesiology and Health Science, York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada e-mail: deniseh@yorku.ca

We have shown that when subjects reach with continuous, misaligned visual feedback of their hand, their reaches are adapted and proprioceptive sense of hand position is recalibrated to partially match the visual feedback (Salomonczyk et al., 2011). It is unclear if similar changes arise after reaching with visual feedback that is provided only at the end of the reach (i.e., terminal feedback), when there are shorter temporal intervals for subjects to experience concurrent visual and proprioceptive feedback. Subjects reached to targets with an aligned hand-cursor that provided visual feedback at the end of each reach movement across a 99-trial training block, and with a rotated cursor over three successive blocks of 99 trials each. After each block, no cursor reaches, to measure aftereffects, and felt hand positions were measured. Felt hand position was determined by having subjects indicate the position of their unseen hand relative to a reference marker. We found that subjects adapted their reaches following training with rotated terminal visual feedback, yet slightly less (i.e., reach aftereffects were smaller), than subjects from a previous study who experienced continuous visual feedback. Nonetheless, current subjects recalibrated their sense of felt hand position in the direction of the altered visual feedback, but this proprioceptive change increased incrementally over the three rotated training blocks. Final proprioceptive recalibration levels were comparable to our previous studies in which subjects performed the same task with continuous visual feedback. Thus, compared to reach training with continuous, but altered visual feedback, subjects who received terminal altered visual feedback of the hand produced significant but smaller reach aftereffects and similar changes in hand proprioception when given extra training. Taken together, results suggest that terminal feedback of the hand is sufficient to drive motor adaptation, and also proprioceptive recalibration.

**Keywords: visuomotor rotation, terminal feedback, motor adaptation, proprioceptive recalibration, vision**

# **INTRODUCTION**

Numerous studies have shown that people can rapidly adapt their reaches when provided with altered visual feedback of their hand, such as a misaligned hand cursor. Moreover, people continue to produce deviated reaches even after the cursor misalignment, or even cursor itself, is removed; such changes in reach direction that are in the opposite direction of the misalignment are known as reach aftereffects (Izawa and Shadmehr, 2011; Taylor and Ivry, 2012; Taylor et al., 2014). Our lab has recently shown that training to reach with distorted visual feedback of the hand also leads to changes in proprioceptive estimates of hand position (Cressman and Henriques, 2009, 2010; Cressman et al., 2010; Salomonczyk et al., 2011; Clayton et al., 2013; Mostafa et al., 2014), such that one perceives their felt hand location to be shifted in the direction consistent with the visuomotor distortion. Moreover, we have shown that it is the discrepancy between vision and proprioception (rather than motor error signals) that drives this change in felt hand position, or what we refer to as proprioceptive recalibration (Henriques and Cressman, 2012). Our aim in this study was to test whether reducing the duration of

this discrepancy to only the very end of the reaching movement is sufficient to also lead to changes in felt hand position. That is, we asked whether adapting reaches to terminal feedback of the hand (i.e., to a hand-cursor that appears only at the end of the reach), and hence limiting subjects exposure to a visual-proprioceptive discrepancy, would lead to proprioceptive recalibration.

Reach adaptation following reach training with terminal versus continuous feedback has shown mixed results, with several studies showing poorer reach adaptation following terminal feedback versus continuous feedback (Hinder et al., 2008; Shabbott and Sainburg, 2010; van der Kooij et al., 2013). On the other hand, other studies have found that differences in learning rate or aftereffects produced following training with terminal feedback versus continuous feedback are rather small (Izawa and Shadmehr, 2011; Taylor et al., 2014). In some cases, learning rates have been shown to be comparable, but reach aftereffects are substantially diminished when training feedback is terminal compared to continuous (Hinder et al., 2008; Shabbott and Sainburg, 2010). Inconsistencies across the studies mentioned above may have to do with the size or

difficulty of the distortion introduced. In particular, it seems that for more difficult visuomotor rotations (e.g., abruptly introduced or large distortions), aftereffects following training with terminal feedback are either smaller than those with continuous feedback (Shabbott and Sainburg, 2010; van der Kooij et al., 2013; Taylor et al., 2014) or non-existent (Hinder et al., 2008), while introducing the distortion gradually can remove this difference such that aftereffects are even larger (Bernier et al., 2005) or nearly equivalent (Izawa and Shadmehr, 2011) compared to those following continuous feedback training.

Previous results from our lab and others have shown that adaptation to a visuomotor distortion leads to changes not only in hand movement, but also to one's sense of hand position, or hand motion estimates, when the hand-cursor is visible for most of the reach (Synofzik et al., 2008; Cressman and Henriques, 2009, 2010; Cressman et al., 2010; Izawa and Shadmehr, 2011; Salomonczyk et al., 2011; Izawa et al., 2012; Clayton et al., 2013; Mostafa et al., 2014). Specifically, we have found that after training with a visuomotor distortion, subjects adapt their no cursor reaches (i.e., post-training reaches without visual feedback used to assess reach adaptation), and shift their estimates of the felt position of the reaching hand in the direction consistent with the visual perturbation (Cressman and Henriques, 2009; Cressman et al., 2010; Salomonczyk et al., 2011; Henriques and Cressman, 2012). In accordance with these findings, other studies have also shown that subjects recalibrate their reaches to visual and proprioceptive targets following reach training with laterally displacing prisms (Hay and Pick, 1966; Redding and Wallace, 1996; van Beers et al., 1999; Redding and Wallace, 2000) or with altered visual feedback of the hand in a virtual reality environment (van Beers et al., 2002; Simani et al., 2007). However, it is unclear if these changes in reaches to proprioceptive targets were due to motor adaptation and/or proprioceptive recalibration, as proprioceptive changes were evaluated with goal-directed reaches. Given that changes in goal-directed reaches can be driven by motor adaptation, motor changes may have influenced proprioceptive target localization. To avoid this potential confound between motor adaptation and proprioceptive recalibration, we use a task designed to assess proprioceptive changes independent of motor changes. Specifically, we measure estimates of felt hand position by having a two-joint robot manipulandum precisely place or guide the subject's hand to a specified location in the workspace, and then ask subjects to judge whether their unseen hand is located to the left or right of either a visual reference marker or the body midline. The extent of change in felt hand position is typically about 20% of the visuomotor distortion, and occurs regardless of whether the distortion is introduced gradually, as in most of our studies (Cressman and Henriques, 2009, 2010; Cressman et al., 2010; Salomonczyk et al., 2011), or abruptly (Salomonczyk et al., 2012). Moreover, this proportional change in felt hand position is evident even when the cursor rotation gradually increases to a maximum of 70◦ (Salomonczyk et al., 2011). Surprisingly, these changes in perceived hand position are not restricted to changes following reach training with a visuomotor distortion, but have also been found following adaptation to a force-field perturbation (Ostry et al., 2010; Mattar et al., 2013).

Taken together, these results suggest that somatosensory plasticity is an integral part of motor learning, at least when subjects reach with continuous visual feedback of their hand. We have suggested that it is the discrepancy between vision and proprioception that drives perceptual changes of felt hand position (and likely a small portion of the reach aftereffects (Cressman and Henriques, 2010; Henriques and Cressman, 2012). In the current study our goal was to test whether training with gradually introduced terminal feedback, and hence limiting subject's exposure to a visual-proprioceptive conflict, was sufficient to lead to proprioceptive recalibration. We also wanted to determine how much terminal feedback training was required for both proprioceptive recalibration and reach aftereffects to saturate and potentially achieve levels similar to those seen after continuous feedback training. To do this, we measured open-loop reaching errors and proprioceptive estimates following each of three sets of 99 reach training trials. In the reach training trials, we used a relatively small cursor rotation (30◦) that was gradually introduced over 40 trials, as using this type of perturbation should lead to significant reach aftereffects following training with terminal feedback, although possibly smaller than those following continuous feedback. In addition, we had subjects perform several sets of reach training trials in order to investigate whether additional reach training may compensate for possibly slower changes in reach adaptation and/or proprioceptive recalibration. We hypothesize that terminal feedback – although perhaps sufficient to drive reach adaptation – may not induce sizeable proprioceptive recalibration, since subjects see their rotated hand-cursor only at the reach's endpoint and thus do not have as much exposure to the visualproprioceptive conflict. Specifically, we predict that any change in felt hand position should be substantially smaller than those produced following training with continuous visual feedback, or at least require additional training to obtain a comparable level of recalibration.

# **MATERIALS AND METHODS SUBJECTS**

Eleven healthy, right-handed adults (mean age = 20.73, SD = 4.45, 7 females) were recruited from York University and volunteered to participate in the current experiment. Prior to participation, subjects were prescreened for self-reported handedness and history of visual, neurological and/or motor dysfunction or injury. In addition to these subjects, the results of ten subjects (mean age = 21.5, SD = 2.62, 5 females) from a previous study (Salomonczyk et al., 2011) were included to serve as a control for comparing the quality of visual feedback on reach adaptation and proprioceptive recalibration. All subjects provided informed consent prior to participating in accordance with the ethical guidelines of York University Human Participants Review Sub-committee.

## **GENERAL EXPERIMENTAL SET-UP**

**Figure 1A** provides a side view of the experimental set-up for the current and previous study. Subjects were seated in a heightadjustable chair in order that they could comfortably view and reach to all targets and reference markers presented on an opaque, reflective surface while grasping the vertical handle of a two-joint robot manipulandum (Interactive Motion Technologies) with

their right hand. The position of the robot handle was recorded at a sampling rate of 50 Hz and had a spatial accuracy of 0.1 mm.

Installed 17 cm above the robot arm was a reflective surface onto which visual stimuli were projected from an LCD monitor (Samsung 510N, refresh rate 72 Hz). The reflective surface was positioned so that targets and reference markers projected onto the surface appeared to lie in the same horizontal plane as the unseen robot manipulandum. All natural light was blocked from the room, the room lights were dimmed, and subject's view of their right hand and the manipulandum was occluded by the reflective surface and a black cloth that covered their right shoulder to the reflective surface.

## **GENERAL PROCEDURE**

To determine the effect of visual feedback quality on reach adaptation and changes in proprioceptive sense of hand position, we had subjects reach to targets with terminal visual feedback of their hand position, and compared their performance with subjects who had previously participated in a similar study in which continuous visual feedback of the hand was provided (Salomonczyk et al., 2011). For the terminal feedback group, during reach training trials, subjects were only shown the hand-cursor at the end of their ballistic reach movements, while subjects in the continuous feedback group were first shown the hand-cursor after the hand had traveled 4 cm from the home position toward the target (located 10 cm from the home position), up until the cursor acquired the visible target. Following the reach training tasks, both groups then reached to the same targets without any hand-cursor feedback and performed a proprioceptive estimation task. Both groups performed two different testing sessions on two separate days (**Figure 2**). For session one, reach training trials involved a cursor that was aligned with the unseen reaching hand to provide baseline measures of performance (**Figure 2**, top row). For session two, the cursor was rotated during reach training trials, and the reach training, no cursor reaches and proprioceptive estimate tasks were repeated three times in succession (**Figure 2**, bottom row).

## *Task 1: Reach training*

In the reach training task (**Figures 1B** and **2**, Boxes 1, 3, and 5), subjects reached to a visual target (yellow circle, 1 cm in diameter) from the home position using the robot manipulandum. Four reach targets were radially located 10 cm from the home position: 30◦ counterclockwise (CCW), 30◦ clockwise (CW), 5◦ CCW, and 5◦ CW of the body's midline (**Figure 1B**). Visual feedback was provided in the form of a hand-cursor (green circle, 1 cm in diameter) that indicated the reach end position (terminal feedback). The cursor was aligned with the actual hand position in the first testing session (**Figure 2**, top row) and gradually rotated to 30◦ CW relative to hand position during the first block of the second testing session (bottom row). Subjects began their reaches from a home position that was approximately 40 cm in front of them and aligned with their body midline. The home position was not illuminated during reach training trials. At the end of each reach trial, visual feedback was eliminated, and subjects returned their hand to the home position along a robot-established linear route (similar to Salomonczyk et al., 2011). If subjects attempted to move outside this linear route or grooved wall, a resistance force was generated [proportional to the depth of penetration with a stiffness of 2 N/mm and a visual damping of 5 N/(mm/s)] perpendicular to the grooved wall (also in (Henriques and Soechting, 2003; Cressman and Henriques, 2009, 2010; Cressman et al., 2010; Jones et al., 2010). Trial order was pseudo-randomized such that subjects reached to each of the two peripheral targets and one of two of the peri-central targets prior to any target repeating. Subjects completed one set of 99 reach trials with the aligned-cursor in the first testing session (**Figure 2**, Box 1, top row) and three sets of 99 reach trials with the rotated-cursor in the second training session (Box 1, bottom row). In the first set of the rotated reach training trials, the cursor rotation was gradually introduced by rotating the cursor 0.75◦ CW relative to the hand each trial, until the maximum rotation of 30◦ CW was achieved on the 41st trial. This 30◦ CW rotation was maintained for all subsequent reach training.

proprioceptive estimate trials were interleaved with further reach training trials. This sequence was repeated a total of 10 times (Boxes 4 and 5). The session ended with another set of no cursor reach trials (Box 6). Session 1 subsequent reach training trials (Boxes 3 and 5) and the additional training sets. These tasks (Boxes 1–6) constitute one block, and were repeated twice more for a total of three blocks.

During reach training trials with terminal feedback, the handcursor was not illuminated until the initial reach movement was complete, i.e., when the velocity of the hand was less than or equal to 3 mm/s for 0.5 s. At this point, the hand-cursor appeared in order to provide subjects with a visual representation of their hand location relative to the target at the end of their initial ballistic motion. After the hand-cursor appeared, subjects were told to move the illuminated hand-cursor to the visible target, and the trial ended when the hand-cursor's center and the target's center were within 0.5 cm of each other. We do not expect that the post-reach motion to target had a significant impact on no cursor reaches; Tseng et al. (2007) compared continuous feedback reach adaptation and aftereffects between subjects who were either permitted to make online corrective movements or not, and no differences were found between groups. On average, subjects moved approximately 2.4 cm while seeing the hand-cursor across all reach training trials. In the infrequent case when subjects managed to obtain the target in the first ballistic motion, the trial ended immediately. At the end of the trial, no visual feedback was provided from the hand-cursor, the target disappeared, and subjects returned their hand to the position along a robot-generated, linear route. In contrast, for subjects training with continuous feedback, the hand-cursor was first displayed once the hand had moved 4 cm from the home position. The hand-cursor then remained visible until subjects acquired the target (Salomonczyk et al., 2011). Thus, subjects who experienced continuous visual feedback experienced real-time feedback about their unseen hand's position in the workspace during their first ballistic motion.

Prior to the reach training task in the first testing session, subjects in the terminal feedback group were given a practice session

of 20 reach training trials with the aligned hand-cursor visible during the entire reach so that subjects could become accustomed to the apparatus and reach task prior to introducing terminal visual feedback. In the continuous feedback condition, there were no preceding practice trials.

# *Task 2: No cursor reaching*

In the no cursor reaching task (**Figure 2**, Boxes 2 and 6), subjects reached to the same visible targets but without visual feedback of the hand-cursor. After subjects held their end position for 0.5 s, the target disappeared, and subjects' hands were again guided back to the home position by a linear grooved path. We calculated reach aftereffects, by subtracting reach endpoints made without a cursor after aligned-cursor training (top row) from those produced after rotated-cursor training (bottom row). Subjects reached to four visual targets three times (Box 2), and to the same four targets plus one additional target at 0◦ (i.e., body midline or center) following proprioceptive estimate trials with interleaved reach training (Box 6). This second set of no cursor trials was to assess whether the aftereffects, and thus, reach adaptation decreased or decayed during the proprioceptive estimate test described below.

# *Task 3: Proprioceptive estimates*

Proprioceptive estimate trials (**Figure 2**, Box 4) began with subjects holding their hand at the home position. The home position, indicated by a green, 1 cm diameter circle, was illuminated for 0.5 s. After the home position disappeared, subjects were instructed to push their hand outward along a robot-constrained, 10 cm long, linear path (**Figure 1C**, elongated rectangle). When a subject's hand arrived at the end of the path, a reference marker (yellow, 1 cm-diameter circle) appeared. Subjects were instructed to make a two-alternative forced-choice decision regarding whether they felt that their unseen hand was to the left or right of this reference marker. Following their response, subjects returned their hand to the start position using the same robot-generated, linear path and began the next trial. The reference markers were located 30◦ CCW, 30◦ CW or 0◦ relative to the body midline (**Figure 1C**, white and open circles). Subjects' hand position relative to each reference marker was adjusted over the course of 50 trials using an adaptive staircase algorithm (Kestin, 1958; Treutwein, 1995), as previously described in our other studies (Cressman and Henriques, 2009, 2010; Jones et al., 2010; Salomonczyk et al., 2011). As in Salomonczyk et al. (2011), there were two staircases per reference marker, each starting at 20◦ either left (CCW) or right (CW) of the reference marker (**Figure 3A**). As outlined by Cressman and Henriques (2009), the two staircases were adjusted individually and randomly interleaved.

Proprioceptive estimate trials were interleaved with reach training trials (**Figure 2**, Boxes 4 and 5). Fifteen proprioceptive estimate trials and six reach training trials (with either an aligned or rotated cursor) immediately followed these initial reach training trials (**Figure 2**, Boxes 4 and 5). A set of 15 proprioceptive estimate trials and 6 reach training trials was completed 10 times, and then subjects performed 15 no cursor reaching trials. Thus, there were a total of 150 proprioceptive estimate trials per block.

## **TESTING SESSIONS: ALIGNED AND MISALIGNED BLOCKS**

The three aforementioned tasks were arranged in blocks within testing sessions that were completed on two days, between 24 h and 30 days apart. Each block consisted of 99 trials of reach training (**Figure 2**, Box 1), no cursor reaches (Box 2), proprioceptive estimate trials intermixed with further reach training trials (Boxes 3–5), and ended with a second set of no cursor (aftereffect) reaches (Box 6). Only one block was completed in the first testing session, where the cursor was aligned with the hand in reach training trials, and the no cursor reach errors and proprioceptive estimates served as a baseline for future rotated cursor blocks. The second day of

testing consisted of three blocks which were performed in succession, as it was unknown whether reach adaptation or shifts in felt hand position following training with terminal feedback would be evident after only one block, (as was the case for continuous feedback) or would require a second or third block of training. Moreover, it was unclear if these changes would increase in size with each set of reach training. The testing sessions were identical to those in the continuous visual feedback study (Salomonczyk et al., 2011).

# **DATA ANALYSIS**

# *Reaches: motor adaptation*

Our main analysis was to determine if open-loop reach errors (i.e., aftereffects) following rotated-cursor training differed from those following aligned-cursor training and if aftereffects following each set of 99 trials with the rotated cursor differed from one another. We also compared these differences or aftereffects across the two sets of no cursor reaches within each block (epoch 1 and epoch 2) to determine if the aftereffects decayed following proprioceptive estimates interleaved with reach training. To examine reach errors, we analyzed the endpoint angle errors and the angle of the hand at peak velocity (PV) in the no cursor reach trials. Endpoint errors were defined as the angular difference between a movement vector (the linear path from the home position to movement endpoint) and the reference vector (the linear path joining the home position to the target). PV angle was defined as the difference in angle between a movement vector, which joined the home position to the point at which the hand reached PV, and the reference vector. For both endpoint errors and angle at PV, we conducted a 4 block (aligned 1 vs. rotated 1 vs. rotated 2 vs. rotated 3) by 2 epoch (post-reach training vs. post-proprioceptive estimates with interleaved reaching) by 4 target location (30◦ left vs. 30◦ right vs. 5◦ left vs. 5◦ right) RM-ANOVA for the terminal feedback group. In order to determine if additional training with rotated terminal feedback yielded any increase in aftereffects over successive blocks, we calculated reach aftereffects by subtracting the no cursor reaches for the aligned block from each

with a subject's hand placed 20◦ from either side of the reference

of responding left 50% of the time.

of those in the three rotated blocks, and then ran another threeway ANOVA but this time with only three blocks (rotated 1–3). Likewise, we used reach aftereffects to compare these changes in movements for the terminal feedback and continuous feedback group, using a mixed ANOVA with visual feedback type (terminal versus continuous) as a between-subjects factor and block (rotated 1 vs. rotated 2 vs. rotated 3) and epoch (post-reach training vs. post-proprioceptive estimates with interleaved reaching) as within subjects factors.

#### *Proprioceptive estimates of hand position*

We examined the influence of training with terminal hand-cursor visual feedback on proprioceptive estimates of hand position. For each subject, we fit a logistic function to his or her responses for each reference marker (**Figure 3B**). From the logistic function we determined the subject's bias, which is an estimate of the subject's accuracy of their sense of felt hand position (Cressman and Henriques, 2009, 2010). Bias was represented by the point at which subjects responded "left" (and "right") 50% of the time (Cressman and Henriques, 2009, 2010; Jones et al., 2010; Salomonczyk et al., 2011). We compared these estimates of felt hand position relative to reference markers after alignedcursor training (baseline) with those after misaligned-cursor training.

Bias was analyzed in a 4 block (aligned 1 vs. rotated block 1 vs. rotated block 2 vs. rotated block 3) by 3 reference marker location (30◦ CCW, 0◦, 30◦ CW) RM-ANOVA. This was followed by another ANOVA where we compared the changes in sense of felt hand position across additional rotated-training blocks by subtracting biases from the aligned session from those biases measured following each rotated set, so that the number of training blocks was reduced to three. These changes were then compared to changes in sense of felt hand position following reach training with continuous visual feedback of the hand in a 2 by 3 mixed ANOVA with visual feedback type (terminal and continuous) as a between-subjects factor and block as a within subjects factor.

For all ANOVAs, differences with a probability of less than 0.05 were considered significant and pairwise comparisons were Bonferroni corrected. We report Greenhouse–Geisser corrected *p*-values when required.

# **RESULTS**

#### **MOTOR ADAPTATION**

Subjects reached to targets with an average movement time of 1.18 ± 0.34 s (SD) and an average PV of 15.85 ± 9.52 cm/s (SD) in the no cursor reaches. In Salomonczyk et al. (2011), the average movement time was 1.78 ± 0.8 s (SD) and the average PV was 16.4 ± 5.9 cm/s (SD). Mean reach endpoint errors for trials performed after aligned-cursor training were 3.73◦ to the right of the target, as illustrated by the first two sets of reach endpoints plotted in **Figure 4A** (labeled session 1). These open-loop reaching errors (prior to adaptation) indicate that subjects were moderately accurate with their reaches to targets even when they lacked visual feedback pertaining to their hand position. These reach errors were a bit more shifted than those observed in the continuous feedback study: in our previous study, these errors were 0.75◦ to the right of the target (Salomonczyk et al., 2011).

We compared these open-loop reaches following training with an aligned cursor with those following rotated-cursor training, as illustrated in **Figure 4A**, which plots these reaches across trials for the aligned block and the three rotated blocks for the terminal feedback group. We found a substantial shift in the direction that subjects reached after training with both terminal and continuous feedback, the extent of which is shown by the black and gray circles in **Figure 4B**. For terminal feedback training, the no cursor reaches deviated significantly leftwards compared to the reaches following the aligned-cursor training block, *F*(3,30) = 36.97, *p* < 0.001, and this was true following all three blocks of rotated-cursor training: aligned cursor block-rotated cursor block: rotated block 1=14.1◦ (*p*<0.001); rotated block 2=12.06◦ (*p*<0.001); rotated block 3 = 11.84◦ (*p* = 0.001). The no cursor reaches relative to baseline (i.e., reach aftereffects) for the terminal feedback group (**Figure 4B**, black circles) were slightly smaller, by roughly 5.8◦ across rotated blocks than those found for the continuous feedback group (gray circles), *F*(1,19) = 4.5, *p* = 0.047. As reported in Salomonczyk et al. (2011), the no cursor reaches were also significantly different between the aligned block and the three rotated blocks when subjects used continuous feedback. We also found that further rotated training with terminal feedback (the additional two blocks) did not lead to substantially larger aftereffects, *F*(2,20) = 2.21, *p* = 0.136. The same was true for subjects receiving continuous feedback (Salomonczyk et al., 2011; **Figure 4B**, gray squares).

When we compared the terminal feedback aftereffects (i.e., change in no cursor reaches relative to baseline performance) made soon after reach training (epoch 1) with the aftereffects completed after proprioceptive estimates (epoch 2), we found no significant difference across the three blocks, *F*(1,10) = 1.67, *p* = 0.22. Likewise, no changes in epoch were found for the continuous feedback group (Salomonczyk et al., 2011). Thus, subjects reached with similar errors before and after completing the proprioceptive estimate trials.

We found a similar pattern of results for changes in the angular reach deviation at PV, as we did for the angular endpoint errors described above for the terminal feedback group. Directional errors at PV were significantly more leftward following all rotated reach training blocks compared to the aligned training block, [*F*(1.461,14.609) = 19.16, *p* < 0.001], in that all comparisons of these no cursor reaches between the aligned training block and each of the three rotated blocks were significantly different (*p* < 0.01). When comparing reach aftereffects, for the most part, the angular deviations at PV closely resembled those of the endpoints (within 2◦) for the terminal feedback group. This was different than the continuous feedback group, where the angle at PV deviated from the endpoint error by 5◦, suggesting that these open–loop reaches were much straighter in the terminal feedback group than in the continuous feedback group. Overall, there was no change over rotated training blocks, thus additional rotated training had no significant impact on PV angle.

#### **BIAS**

Next, we wanted to determine if adapting to a rotated cursor with terminal feedback also led to similar changes in felt hand position, i.e., proprioceptive recalibration, as has been

seen after training with continuous visual feedback of the handcursor. **Figure 5A** displays the three reference marker locations (circles), average biases following aligned-cursor training (diamonds) and rotated-cursor training (triangles) when terminal feedback was provided. Each successively darker triangle represents subject's estimates of felt hand position relative to the reference marker for rotated blocks 1, 2, and 3. **Figure 5B** uses the same schematic to illustrate the results under continuous feedback conditions (Salomonczyk et al., 2011). In the terminal feedback condition, for the aligned block, felt hand positions were slightly left of the reference markers, specifically 7.27◦ left of the reference marker. This leftward bias has been previously observed in our lab and is due to a hand bias (Jones et al., 2010); this hand bias was also observed in the continuous feedback condition (Salomonczyk et al., 2011), where the average bias across subjects and reference markers for the aligned block was 5.1◦ leftward.

For terminal feedback, we see that each rotated block yielded estimates of felt hand positions that were successively further left of the reference markers and the estimates after training with an aligned hand-cursor, consistent with the direction of the visuomotor distortion (**Figure 4B**), black triangles). There was a main effect of training block among the aligned and three rotated training blocks, *F*(3,30) = 8.62, *p* < 0.001. Thus, we

next assessed whether biases after each rotated set were significantly shifted relative to those following the aligned-cursor training. We found that although biases were not significantly shifted for the first rotated block, they were for the second and third rotated blocks relative to the aligned block: rotated block 1 – aligned, 3.39◦ (*p* = 0.404); rotated block 2 – aligned, 5.12◦ (*p* = 0.035); rotated block 3 – aligned, 7.41◦ (*p* = 0.008). Additionally, the change was much larger, by 4.01◦, in the last rotated block compared to the first rotated block (*p* = 0.029), suggesting that more practice with terminal feedback led to greater proprioceptive recalibration (illustrated by the increasing height of the black triangles across blocks in **Figure 4B**). This was not the case for the continuous feedback group (Salomonczyk et al., 2011), where the significant change in bias saturated after the first set of rotated training (**Figure 4B**, gray triangles).

Interestingly, we found that the overall size of the change in felt hand position was similar across the terminal and continuous feedback groups, in that there was no significant difference in changes in bias for the terminal feedback and continuous feedback groups, *F*(1,19) = 0.56, *p* = 0.46. Although Salomonczyk et al. (2011) did not find a significant difference across the three blocks of rotated training, when we looked at the change in bias across the three rotated blocks for the terminal feedback group, we found that they did significantly differ as explained above. Thus, both feedback groups reached a similar level of change in felt hand position by the end of the three training blocks.

## **MOTOR ADAPTATION AND PROPRIOCEPTIVE RECALIBRATION**

To better compare changes in reaches (aftereffects) to changes in felt hand position, we ran a linear regression to see whether changes in felt hand position depended on changes in reach aftereffects. As consistent with our previous studies (Cressman and Henriques, 2009; Salomonczyk et al., 2011, 2012, 2013), we found no significant relationship between the changes (*p* = 0.17, *<sup>R</sup>*<sup>2</sup> <sup>=</sup> 0.06), although as usual the change in felt hand position was much smaller than the reaching aftereffects for the two feedback

groups (**Figure 4B**). More importantly, we found that despite significantly smaller reach aftereffects following terminal feedback training, compared to continuous feedback training, the overall change in felt hand position was similar between the two feedback groups, at least by the third block. Again, this suggests that the sensory changes are not directly related to motor changes.

# **DISCUSSION**

The main goal of the present study was to examine whether terminal feedback experienced during reach training affects our subsequent estimates of felt hand position. Subjects reached to three targets for a total of 99 trials with visual feedback of their hand rotated 30◦ CW relative to hand movement, in three reach training blocks. Visual feedback was only provided at the end of the primary movement. After each training set of 99 trials, subjects reached to the same targets without a cursor, and then estimated the position of their trained, unseen hand relative to reference markers at similar locations. On average, we found that subjects who experienced terminal visual feedback both adapted their reaches and recalibrated their felt hand position. Mean reach aftereffects approached 13◦ after the first rotated block, and were maintained at that level even after two additional training blocks. Sense of felt hand position was also recalibrated by 3.4◦ after the first training block; however, changes in felt hand position increased further and significantly to 7.41◦ by the third reach training block. Compared to another group of subjects who experienced continuous feedback (Salomonczyk et al., 2011), subjects experiencing terminal feedback appeared to adapt their reaches less (by about 33%) over the three training blocks, but their sense of felt hand position, although initially shifted less than subjects in the continuous feedback group, reached a comparable level by the third training block. Thus, we found that terminal feedback was sufficient to drive reach adaptation, and despite subjects seeing the visual representation of their hand only for a limited time at the end of the movement, they successfully recalibrated their felt hand position to a level comparable to subjects with continuous feedback training after additional training.

# **ROLE OF VISUAL FEEDBACK QUALITY IN REACH ADAPTATION**

In the current study, we provided three sets of reach training trials in order to determine how long it took for reach adaptation to saturate when terminal feedback was provided (by the end of the third training set, subjects had reached to each of the targets 99 times). Surprisingly, our extra training trials did not lead to increased aftereffects over successive training blocks, such that aftereffects following the first 99 training trials were not significantly different from those found after all 297 trials (reach adaptation equal to ∼13◦). This early saturation of reach adaptation is similar to our previous results in which subjects completed the same three training blocks to the same targets with a continuously visible rotated cursor (reach adaptation equal to ∼18.44◦; Salomonczyk et al., 2011). Moreover, we have seen reach adaptation saturate quickly in an earlier study of ours (Wong and Henriques, 2009), where we had subjects reach with a rotated cursor to similar targets for at least 200 trials each day for five consecutive days. Thus, we

have shown that increased training neither helped nor decreased the discrepancy in the extent of motor adaptation between terminal and continuous feedback conditions. Also, terminal feedback resulted in smaller reach aftereffects, compared to continuous feedback. These smaller aftereffects were not due to decay over the open-loop reach trials, since no cursor reaches were constant within a block.

The reach aftereffects we found following terminal feedback training were about 66% of the size of those found following continuous feedback in our earlier study (Salomonczyk et al., 2011), and reflect significant reach adaptation. These results differ from those of Hinder et al. (2008) and Shabbott and Sainburg (2010) who found no significant reach aftereffects. Their aftereffects were based on reaches produced when the aligned cursor was reintroduced (rather than removed, like in this study), and training feedback involved not just cursor endpoint alone, but the entire hand path display (what they called knowledge of results – KR). However, our results are consistent with the majority of studies that used endpoint feedback during training and measured aftereffects based on no cursor reaches (which would be associated with smaller washout). For instance, van der Kooij et al. (2013) and Taylor et al. (2014) both showed significant, yet smaller, reach aftereffects following terminal feedback training compared to continuous training. For example, van der Kooij et al. (2013) found significant changes in open-loop reaches following training with terminal feedback, or what they called realignment of the unseen hand, and these changes were about one third smaller than those produced by subjects who trained with continuous feedback. Again, the distortion they used, although abruptly introduced, was rather small (5◦ deviation relative to the cyclopean eye). In a recent paper by Taylor et al. (2014), following terminal feedback training with an abrupt, 45◦ cursor rotation, reach aftereffects were roughly 66% the size of those produced following training with abrupt continuous feedback. During reach training, some subjects verbally reported which target they were going to aim for prior to each reach – the instruction groups. The reach aftereffects for these subjects in the instruction group did not significantly differ from those produced by subjects who reached without making a verbal report, following either continuous or terminal feedback training. In addition, the relative magnitude of these reach aftereffects in their study (terminal vs. continuous) is similar to that found by van der Kooij et al. (2013) and the current study. And while Taylor et al. (2014) suggest that differences in reported aiming direction during training for the instruction groups indicates that terminal feedback resulted in greater explicit learning compared to continuous feedback, our results neither support nor refute this interpretation since our distortion was gradually introduced, and thus less likely to engage explicit learning processes. Interestingly, Bernier et al. (2005) showed that following training with continuous feedback, aftereffects washed out quickly while those following training with KR were initially large and did not washout. Like us, Bernier et al. (2005) also gradually introduced a rather small visual perturbation and had subjects reach 80 times to each of three nearby targets. Likewise, in Izawa et al. (2012), a gradually introduced and small, 8◦ cursor rotation led to near equivalent aftereffects in the direction

of the trained target (although generalization to novel but proximal targets was about 50% smaller). Thus, taken together, these studies suggest that significant reach aftereffects arise after training with terminal feedback, when assessed by open-loop reaches.

Previous results of ours suggest that when the cursor feedback is continuous during training, there is no difference in aftereffects regardless of whether the 30◦ cursor rotation was introduced gradually or abruptly (Salomonczyk et al., 2012). Klassen et al. (2005) also found no difference between abrupt and gradual rotated training (for a 30◦ rotation) when they measured retention of adaptation a day later. However, reach aftereffects have been found to be smaller following abrupt cursor rotation compared to a gradual one when the perturbation is particularly large [e.g., 90◦; Kagerer et al., 1997; Buch et al., 2003; *N.B.* Buch et al. (2003) only found this for their older subject group]. Thus, it is possible that for more challenging perturbations, including perhaps ones involving terminal feedback, the manner in which the distortion is introduced may influence reach aftereffects. In contrast, given that studies using an abrupt perturbation (van der Kooij et al., 2013; Taylor et al., 2014) and those using a gradually introduced perturbation (Izawa and Shadmehr, 2011; and the current study) found that changes in open-loop reaches after training with terminal feedback were at least two-thirds the size of those produced following training with continuous feedback, the manner in which the distortion is introduced may make little difference when the distortion is small (e.g., less than 45◦).

# **THE EFFECT OF TRAINING WITH TERMINAL FEEDBACK ON HAND PROPRIOCEPTION**

In our study, we derived subject's sense of felt hand position with a task that does not require goal-directed reaches, by asking subjects to report the location of their (robot-guided) felt hand position relative to a reference marker (Cressman and Henriques, 2010, 2011; Cressman et al., 2010; Salomonczyk et al., 2011, 2012; Clayton et al., 2013; Salomonczyk et al., 2013; Mostafa et al., 2014). We found that subjects recalibrated their felt hand position following rotated hand-cursor training, even after training with only terminally altered feedback of their hand. However, this proprioceptive shift only achieved significance after the second block of reach training, and continued to increase in size during the third and final block. By this final block of rotated terminal feedback training, subject's shift in felt hand position was comparable to shifts in felt hand position experienced by subjects in the continuous feedback condition. With continuous feedback, Salomonczyk et al. (2011) found that additional training, beyond the first block of 99 trials, did not lead to further recalibration following a 30◦ rotation; however, gradually increasing the cursor rotation (up to 70◦) did lead to larger changes in felt hand position (as well as reach aftereffects). This change in felt hand position following rotated continuous feedback training was similar whether the cursor was gradually or abruptly introduced (Salomonczyk et al., 2012). It is unknown whether introducing the terminally misaligned cursor abruptly would have a similar effect on proprioceptive recalibration.

In addition to changes in felt hand position, it has recently been shown that visuomotor adaptation leads to changes in estimating the sensory consequences of self-guided hand movements (Synofzik et al., 2008; Izawa and Shadmehr, 2011; Izawa et al., 2012). That is, people mislocalize the direction by which they move their unseen hand across a landmark following visuomotor adaptation to a rotated cursor. To look at this, Izawa and Shadmehr (2011) measured both reach aftereffects and hand localization errors under different feedback conditions, including training with continuous and terminal misaligned feedback of the hand that was gradually introduced. They found that reach aftereffects were equivalent, at least in the direction of training (generalization to novel directions was smaller for terminal feedback training than for continuous), and the errors in predicting the consequences of these movements (the hand localization errors) were about 30% smaller following training with terminal feedback compared to continuous feedback. Together, these studies show that changes in felt hand position and sensory prediction errors follow different patterns depending on whether there was continuous or terminal feedback.

# **INDEPENDENCE OF REACH ADAPTATION AND PROPRIOCEPTIVE RECALIBRATION**

Our results, along with those from prior studies from our lab and others, suggest that changes in reaches and changes in felt hand position following training with altered visual feedback of the hand are independent of each other. First, the point in training by which maximum changes were achieved was different for the two feedback conditions, such that 99 training trials were needed for motor adaptation to saturate, and 297 training trials were needed for changes in bias to reach maximum levels achieved in an earlier study. Similar to the results for continuous rotated feedback (Salomonczyk et al., 2011), we also found no significant correlation between the changes in reaches and hand proprioception. Results from related studies in our lab have also shown this lack of correlation, or different rates of change between motor adaptation and sensory consequences. Finally, and more convincingly, we have shown different patterns of generalization for reach aftereffects and changes in hand proprioception (Mostafa et al., 2014).

Along with results from our lab, results from studies testing patients with cerebellar damage or using a force-field perturbation suggest this independence of motor and sensory changes following training with a rotated cursor. For example, in Izawa et al. (2012), while cerebellar patients adapted their reaches to a perturbation that was gradually introduced to the same extent as controls (similar reach aftereffects), patients showed smaller changes in what the authors called the predictive consequences of unseen hand movements; these were measured by having subjects reach with their unadapted hand, to the location at which they perceived their unseen adapted hand had previously moved. Moreover, Synofzik et al. (2008) found that while cerebellar patients did not learn to adapt their reaches to a cursor rotation that increased by 6◦ per trial (i.e., somewhat abruptly) as well as controls, they did recalibrate their estimates of their arm movements. However, similar to Izawa et al. (2012), this recalibration level seen in the patients was less than in the controls. In a force-field perturbation

paradigm, Mattar et al. (2013) recently showed different patterns in the rate of adaptation and the rate of change in what they called the perceptual boundary of the adapted hand. In brief, the pattern of changes in motor adaptation and proprioceptive recalibration following training with terminal feedback in the current study add to the argument for motor adaptation and sensory recalibration's independence.

# **CONCLUSION**

Following visuomotor adaptation using terminal visual feedback, subjects adapted their reaches and recalibrated their sense of felt hand position, but these changes were smaller than those for subjects who received continuous visual feedback. Based on the present results, we suggest that terminal feedback provides sufficient information for motor learning, even after only 99 trials (33 trials per target). But, while motor adaptation remained relatively stable after the first rotated training block, additional training was necessary for attaining maximal changes in felt hand position. This difference in rate of motor adaptation vs. proprioceptive recalibration provides further support for the proposal that motor adaptation and sensory recalibration are two processes that change concurrently, yet independently. At present, the current results suggest that the amount of visual feedback available influences the time required for proprioceptive recalibration.

# **ACKNOWLEDGMENTS**

Thanks to Holly Clayton, Ahmed Mostafa, Petar Petrovic, and Nilufer Nourouzpour for their assistance during the preparation of this manuscript and to Golsa Bahari for her assistance with data collection. This work was supported by the Natural Sciences and Engineering Research council (Denise Y. P. Henriques and Erin K. Cressman).

# **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 April 2014; accepted: 22 August 2014; published online: 08 September 2014.*

*Citation: Barkley V, Salomonczyk D, Cressman EK and Henriques DYP (2014) Reach adaptation and proprioceptive recalibration following terminal visual feedback of the hand. Front. Hum. Neurosci. 8:705. doi: 10.3389/fnhum.2014.00705*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Barkley, Salomonczyk, Cressman and Henriques. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

MINI REVIEW ARTICLE published: 09 September 2014 doi: 10.3389/fnhum.2014.00686

# Neural correlates of grasping

# **Luca Turella<sup>1</sup>\* and Angelika Lingnau1,2**

<sup>1</sup> Center for Mind/Brain Sciences (CIMeC), University of Trento, Trento, Italy <sup>2</sup> Department of Cognitive Sciences, University of Trento, Trento, Italy

#### **Edited by:**

Simona Monaco, York University, Canada

#### **Reviewed by:**

Lennart Verhagen, Radboud University Nijmegen, Netherlands Umberto Castiello, Università di Padova, Italy

#### **\*Correspondence:**

Luca Turella, Center for Mind/Brain Sciences (CIMeC), University of Trento, Via delle Regole, 101, 38123 Trento, Italy e-mail: luca.turella@gmail.com; luca.turella@unitn.it

Prehension, the capacity to reach and grasp objects, comprises two main components: reaching, i.e., moving the hand towards an object, and grasping, i.e., shaping the hand with respect to its properties. Knowledge of this topic has gained a huge advance in recent years, dramatically changing our view on how prehension is represented within the dorsal stream. While our understanding of the various nodes coding the grasp component is rapidly progressing, little is known of the integration between grasping and reaching. With this Mini Review we aim to provide an up-to-date overview of the recent developments on the coding of prehension. We will start with a description of the regions coding various aspects of grasping in humans and monkeys, delineating where it might be integrated with reaching. To gain insights into the causal role of these nodes in the coding of prehension, we will link this functional description to lesion studies. Finally, we will discuss future directions that might be promising to unveil new insights on the coding of prehension movements.

**Keywords: prehension, grasping, reaching, fMRI, neurophysiology, motor system**

# **INTRODUCTION**

The capacity to reach and grasp objects, i.e., prehension, is at the basis of our daily interactions with objects. Prehension entails two main components: transport, i.e., reaching an object with the hand, and grasping, i.e., the preshaping of the hand with respect to the object's intrinsic properties (e.g., shape and size). Previous monkey neurophysiological and human neuroimaging studies demonstrated that planning and execution of this complex skilled behavior, and of its two components, are encoded within specific neural substrates: the "prehension" network (Jeannerod, 1981; Jeannerod et al., 1995; Brochier and Umiltà, 2007; Castiello and Begliomini, 2008; Filimon, 2010; Grafton, 2010; Davare et al., 2011).

This Mini Review is thought as a brief introduction and as an update of two recent reviews on this topic (Filimon, 2010; Grafton, 2010). Here, we will focus on grasp coding and on its integration with reaching, as reaching has already been covered in recent contributions (Crawford et al., 2011; Vesia and Crawford, 2012). We will focus on a description of the role of the dorsal stream in grasp coding, despite recent investigations pointing to a possible involvement of the ventral stream in prehension (Verhagen et al., 2008, 2012). Throughout the review, we will touch the following main questions, which are still matter of investigation: (i) *where* the prehension system codes the two components; (ii) which regions are *necessary* for their coding; and (iii) at which stage they are possibly *integrated*.

In the first part, we will provide an anatomical and functional description of the prehension system in monkeys and humans. In the second part, we will describe lesion studies which allow drawing causal inferences on the role of the regions within the prehension system. In the last part, we will cover recent advances on grasp coding with a focus on the temporal aspects which we consider fundamental for obtaining new insights on the neural basis of prehension.

# **ANATOMICAL AND FUNCTIONAL DESCRIPTION OF THE PREHENSION SYSTEM**

The classical description of the monkey prehension system was based on the definition of a series of parallel cortico-cortical pathways connecting regions within the posterior parietal cortex (PPC) with regions of the frontal cortex possessing similar response properties. These pathways are considered crucial in the sensorimotor processing for the *planning* and *online control* of reaching, grasping and saccadic eye movements (Rizzolatti et al., 1998; Andersen and Buneo, 2002).

According to the classical model of prehension, the dorsolateral pathway is coding grasping, i.e., different grip types, whereas the dorsomedial pathway is coding reaching, i.e., information related to the transport phase (**Figure 1A**; Jeannerod et al., 1995; Caminiti et al., 1998; Culham et al., 2003; Culham and Valyear, 2006).

The dorsolateral pathway connects two core regions: the anterior part of the intraparietal sulcus (AIP; Murata et al., 2000; Baumann et al., 2009) within the inferior parietal lobule (IPL) and area F5 within the ventral premotor cortex (PMv; Rizzolatti et al., 1988; Murata et al., 1997; Raos et al., 2006; Fluet et al., 2010). This pathway has been classically described to be involved in visually guided grasping via the transformation of intrinsic properties of the to-be-grasped object into appropriate motor commands for hand pre-shaping (Jeannerod et al., 1995; Brochier and Umiltà, 2007). The neurophysiological basis of this sensorimotor transformation

might be supported by visuomotor neurons ("canonical" neurons) described in AIP (Murata et al., 2000) and F5 (subareas F5p and F5c, Bonini et al., 2014) which are active while performing a grasping movement and while observing graspable objects. Most of these neurons show a strict congruence between the coded grip and the intrinsic properties of the object eliciting their visual response.

The dorsomedial pathway connects two regions within the PPC, area V6A (Bosco et al., 2010) and MIP (Johnson et al., 1996), with the dorsal premotor cortex (PMd; Caminiti et al., 1991). This pathway has been classically considered as coding reaching information for planning and controlling arm position during the transport phase, via the integration of somatosensory and visual information (Rizzolatti et al., 1998).

This initial model has been shown to be incomplete, as many neurophysiological investigations described neural activity related to both components of prehension within both pathways (**Figure 1B**). With respect to grasping, the IPL convexity (particularly area PFG), having direct connections to F5, seems to be critically involved in planning and executing grasping (Rozzi et al., 2008; Bonini et al., 2011, 2012). Core regions of the dorsomedial pathway, V6A and PMd (area F2vr), are coding not only reach, but also grasp-related information (Raos et al., 2004; Fattori et al., 2010, 2012). Similarly, many regions within both pathways are also involved in coding reaching (see Figure 3 in Battaglia-Mayer et al., 2006). Remarkably, even the core nodes of the grasp-related pathway (F5 and AIP) host neural populations coding reaching and even populations coding both reach and grasp information (Lehmann and Scherberger, 2013). Nevertheless, few other studies investigated the coding of both components within the same neural population (e.g., PMd and PMv, Stark et al., 2007). Consequently, it is difficult to assess, at least from a *functional* point of view, to which degree grasping and reaching are encoded independently, and at which stage they are integrated.

Monkey neurophysiological investigations provided the starting point for the definition of a similar human system via neuroimaging techniques which lack the high spatial and temporal resolution of neurophysiological recordings, but sample the whole brain, instead of only one or few nearby regions. The classical method for fMRI analysis adopts a univariate comparison of activity between different conditions for every single voxel. Using a univariate approach, a potential homologous prehension system has been described within the human PPC and premotor cortices (Culham et al., 2006; Culham and Valyear, 2006; Filimon, 2010; **Figure 2A**). With respect to the dorsolateral pathway, a possible homology was found for a region of the anterior intraparietal sulcus (aIPS; Culham et al., 2003; Frey et al., 2005) and for PMv (Cavina-Pratesi et al., 2010b), both recruited during grasping. Regarding the dorsomedial pathway, homologous reach-related areas were localized within the medial intraparietal sulcus (mIPS; Prado et al., 2005; Filimon et al., 2009), the superior parietal occipital cortex (area SPOC), the precuneus (Connolly et al., 2003; Prado et al., 2005; Filimon et al., 2009; Cavina-Pratesi et al., 2010b) and PMd (Filimon et al., 2007, 2009).

Univariate analyses also showed activity within the whole prehension network when comparing reaching only (Filimon et al., 2007, 2009) or reach-to-grasp movements (Culham et al., 2003; Turella et al., 2009) with respect to a baseline or control condition, resembling the widespread coding of both components of prehension shown in monkey. Whereas univariate analyses can identify areas in which either the reach or the grasp component leads to a higher overall signal, this approach does not allow drawing conclusions about the properties coded in these regions.

Recent advances in fMRI analysis permitted a more fine grained investigation of the properties of the prehension network by adopting Multivariate Pattern (MVP) analysis. Instead of carrying out massive univariate analysis separately for each voxel, this approach uses the pattern of activation over multiple voxels (Kriegeskorte and Bandettini, 2007). Recently, Gallivan et al. (2011, 2013) distinguished using MVP analysis between visually guided reach-to-grasp and reach-only movements (during planning and execution) within a number of predefined regions of the two pathways such as PMv, PMd, aIPS, mIPS and SPOC.

Similar results were obtained in a recent study using MVP analysis investigating the execution of non-visually guided actions (Fabbri et al., 2014). This study manipulated both grip type (i.e., whole hand grip vs. precision grip) and movement direction within the same paradigm. The results showed overlapping regions coding grasping and reaching within the whole prehension system (PMv, inferior PMd, anterior SPL, aIPS, see **Figure 2B**) and hint at a possible interaction between both types of coding within a subset of these regions (PMv, aIPS, anterior SPL).

To summarize, converging evidence from neurophysiological and neuroimaging studies suggests that, from a *functional* perspective, the strict subdivision of the prehension system in two independent pathways is not tenable as grasping seems to be coded, and possibly integrated with reaching, within both pathways.

# **LESION STUDIES**

Neurophysiological and neuroimaging methods are correlational by nature. Consequently, measuring grasp-related activity within a specific region does not prove its causal involvement in determining grasping at a behavioral level. Lesion studies provide fundamental information for the interpretation of neurophysiological and neuroimaging data.

A number of monkey lesion studies (Battaglini et al., 2002; Hwang et al., 2012; Yttri et al., 2014; for a review, see Andersen et al., 2014) showed that the so-called Parietal Reach Region, comprising V6A, MIP and area 5v (Andersen et al., 2014), is causally involved in the planning and online control of reaching. After resection of V6A, monkeys were unable to correctly perform object-directed prehension movements, not only misreaching targets but showing also grasping deficits, i.e., abnormal wrist orientation and incorrect preshaping (Battaglini et al., 2002; Galletti et al., 2003). Lesions in the core regions of the dorsolateral pathway (AIP and F5p), have been reported to affect hand preshaping (i.e., grasping), leaving the reach component unaffected. After inactivation of AIP, monkeys showed abnormal hand preshaping during prehension (Gallese et al., 1994). The deficit was evident only, or mainly, when a precision grip was required, whereas whole hand prehension was generally unimpaired. This suggests that the potential impairment was evident only when more precise sensorimotor control was required. Inactivation of F5p (Fogassi et al., 2001) leads to a similar impairment with abnormal preshaping of the hand and wrist orientation, mainly evident while grasping small objects. Crucially, inactivation of the nearby F5 subarea (F5c), possessing the same visuomotor properties (Bonini et al., 2014), did not lead to any grasping deficits (Fogassi et al., 2001).

These results show that both pathways are *causally* involved in processing grasping, and also support a behavioral dissociation: lesions in the dorsolateral pathway impair mainly grasping,

whereas damage within the dorsomedial pathway affects only reaching (MIP) or both reaching and grasping (V6A). If we link these results to neurophysiological findings, it is evident that the coding of both reaching and grasping within V6A has a clear behavioral relevance, possibly reflecting the processing of the whole act of prehension, integrating reaching and grasping information (Grafton, 2010; Fattori et al., 2010). The dorsolateral pathway (AIP and F5p) seems more strongly involved in coding visually guided grasping, particularly when this requires a high level of integration of visuospatial and contextual information for planning and controlling hand preshaping (Verhagen et al., 2008; Fattori et al., 2010).

It is more difficult to assess specific behavioral deficits based on human lesion studies, as the extent of brain damage is generally wider, encompassing more than a single cortical region. Nevertheless, recent studies support a similar account, with lesions in posterior PPC leading to reaching, and possibly also grasping deficits (Karnath and Perenin, 2005; Cavina-Pratesi et al., 2010a), and lesions in anterior PPC leading mainly to grasping impairments (Binkofski and Buxbaum, 2013).

Complementary information can be derived from "virtual lesion" TMS studies. This approach can inform us more accurately on where and at which stage (planning and/or online control) a temporary lesion affects grasp coding. The causal role of the dorsolateral pathway in coding grasp-related information has been demonstrated both for aIPS and PMv (Olivier et al., 2007). These studies demonstrated the specific role of aIPS in hand pre-shaping during visually guided prehension (Rice et al., 2006; Davare et al., 2007; Vesia et al., 2013) and during rapid online correction after object perturbation (Tunik et al., 2005; Rice et al., 2006).

A possible causal role of the dorsomedial pathway in grasp coding has been put forward on the basis of a dissociation between PMd and PMv in a visually guided grasp-to-lift task (Davare et al., 2006). TMS applied to PMv impaired hand preshaping, whereas TMS applied to PMd interfered only with lifting the object, as if the coupling between reaching and grasping was affected. These results seem to suggest that the dorsolateral pathway is *causally* involved in grasp coding, whereas the dorsomedial is *causally* involved in coding the interaction between the two components of prehension.

# **RECENT ADVANCES AND FUTURE DIRECTIONS**

One major limit in the description of grasp coding in monkeys consists in being primarily based on studies recording single cell activity. This description has a high temporal resolution allowing to map activity related to the different stages of prehension (planning, execution, online control), but it is difficult to understand how information is transferred to other cortical sites, as normally only one, or few nearby, areas are recorded simultaneously. A solution might be the widespread adoption of multielectrode and multiple site recordings which will help understanding the evolution of grasp coding within different regions.

As an example, Townsend et al. (2011) simultaneously investigated the neural response of AIP and F5 during a delayed motor task adopting a multivariate approach, i.e., trying to decode grip type and object orientation during planning. The analysis was based on multi-unit activity (MUA) which showed similar tuning as single-unit activity (SUA). Decoding of grip type or orientation alone showed significant above chance performance in both areas, with a preference of coding for grip type in F5 and for orientation in AIP. Decoding of grip type and orientation showed the best performance when combining data simultaneously recorded from the two regions, suggesting that they play complementary roles in grasp coding.

This study (Townsend et al., 2011) highlights the potential of multisite recordings in defining functional properties of simultaneously recorded regions. Moreover, it demonstrates that MUA conveys meaningful grasp information. Recent studies showed that also power modulations of Local Field Potentials (LFPs) code grip information both within F5 (Spinks et al., 2008) and IPL (comprising area AIP; Asher et al., 2007).

Stark and Abeles (2007) simultaneously recorded from PMv and PMd investigating reach and grasp coding, showing that it is possible to decode reach direction and grip type, and even their interaction, using SUA, LFPs and MUA (called multi-spike activity in this study) recorded from the same multiple electrodes. The limit of this study was that it pooled neural signal from PMd and PMv for decoding, so it is not possible to understand the specific role of each region in grasp and/or reach coding.

Taken together, these studies (Asher et al., 2007; Stark and Abeles, 2007; Spinks et al., 2008; Townsend et al., 2011) show that SUA, MUA, and LFPs convey grasp information. It is unclear, however, to which extent these measures play similar or different roles in the coding of grasping and in its integration with reaching. Furthermore, the decoding approach might be adopted not only to define the content, but also the different phases of prehension at which the coding of grasp information might happen, as recently shown for the early coding of observed graspable objects within AIP (Srivastava et al., 2009; Sakaguchi et al., 2010).

Monkey studies offer the unique possibility of obtaining a precise spatial and temporal map of the evolution of grasp coding, not only within one pathway but potentially within both. To explore the temporal relationship between coding within the two pathways and to test when and where grasping is integrated with reaching, future work might comprise simultaneous multisite recording (e.g., within AIP and V6A) during a grasping task. Reversible lesion studies might then be used to test the causal role of the same regions in the integration of the two components, identifying which signal (SUA, MUA, LFPs) or combination of signals conveys such integration.

Most of our knowledge on the human prehension system stems from neuroimaging data. Given the dynamic nature of prehension, it is crucial to understand the temporal evolution of its coding and of the interaction between grasping and reaching. fMRI lacks the temporal resolution needed for investigating these temporal aspects. In addition, it is difficult to understand when this integration would happen, as most fMRI studies did not separate between a planning and execution phase (but see Gallivan et al., 2011, 2013). A potential tool to unveil the neural dynamics of the integration of reach and grasp coding resides in exploiting high temporal resolution methods (EEG and MEG), which record signals more comparable with monkey neurophysiological data, particularly with LFPs.

A recent study started to tackle this issue by investigating prehension coding during planning using a combination of EEG, TMS and kinematic recordings (Verhagen et al., 2013). This study suggested a hierarchical organization of the two pathways, with the processing within the dorsomedial pathway being temporally dependent on aIPS activity. These results are further corroborated by another EEG study (Tunik et al., 2008) using a perturbation task, i.e., changing the orientation of the object during prehension. Adopting a different approach, i.e., microstate analysis, the results supported similar conclusions, showing two different processes after movement onset: one within aIPS and the other within posterior SPL. The process within the dorsomedial pathway was always following the one in the dorsolateral. This seems to suggest that aIPS is involved in integrating information for creating an action plan, whereas the activation of SPL was coincident with the start of the online adjustment, always following the end of aIPS recruitment.

These EEG results suggest that the two pathways interact during prehension coding and that the dorsolateral pathway could drive processing within the dorsomedial one. It is still unclear if this is the only type of interaction between the two pathways, or if other interactions can occur depending on task demands (e.g., level of online control, Grol et al., 2007) or between these two pathways and the ventral stream (Verhagen et al., 2008, 2012). Moreover, these EEG studies (Tunik et al., 2008; Verhagen et al., 2013; see also De Sanctis et al., 2013; Tarantino et al., 2014) demonstrate the potential of neurophysiological investigations as a tool for identifying potential time windows and cortical sites of integration, which could be subsequently tested adopting virtual lesions.

We have provided an up-to-date overview of the recent developments on grasp coding: at present we have a better understanding of *where* grasping (i.e., grip information) is coded and *which regions are causally involved* in its processing, but we still miss critical information about *when and where* this information is integrated with reaching (i.e., transport information). As described in the previous sections, integration between these two types of information might take place within both pathways at a functional level. By contrast, lesion studies seem to point to the integration of transport and grip information mainly within the dorsomedial pathway. How can we reconcile the discrepancy between these two levels of description?

Various accounts have proposed that the difference in coding characterizing the dorsolateral and the dorsomedial stream might emerge from a more general level of processing rather than from a distinction based on grasping and reaching (Rizzolatti and Matelli, 2003; Verhagen et al., 2008, 2012, 2013; Glover et al., 2012). Information about the temporal dynamics within the prehension system might be a critical factor to unravel these unsolved issues, permitting also to understand what type of information is driving the processing within these two pathways.

#### **ACKNOWLEDGMENTS**

This work was supported by the Provincia Autonoma di Trento. Luca Turella is supported by the "Futuro in Ricerca 2013" grant (FIRB 2013, project RBFR132BKP) awarded by MIUR. The authors would like to thank Giacomo Ariani and Raffaele Tucciarelli for their comments on an earlier version of this manuscript. The authors would like to thank also the two reviewers for their insightful comments.

#### **REFERENCES**


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 08 June 2014; accepted: 15 August 2014; published online: 09 September 2014*.

*Citation: Turella L and Lingnau A (2014) Neural correlates of grasping. Front. Hum. Neurosci. 8:686. doi: 10.3389/fnhum.2014.00686*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2014 Turella and Lingnau. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

# The left side of motor resonance

# *Luisa Sartori 1,2\*, Chiara Begliomini 1,2, Giulia Panozzo1, Alice Garolla1 and Umberto Castiello1,2*

*<sup>1</sup> Dipartimento di Psicologia Generale, Università degli Studi di Padova, Padova, Italy*

*<sup>2</sup> Cognitive Neuroscience Center, Università degli Studi di Padova, Padova, Italy*

#### *Edited by:*

*Simona Monaco, York University, Canada*

#### *Reviewed by:*

*Alessio Avenanti, University of Bologna, Italy Claudia L. R. Gonzalez, University of Lethbridge, Canada*

#### *\*Correspondence:*

*Luisa Sartori, Department of Psychology, University of Padova, Via Venezia 8, 35131 Padova, Italy e-mail: luisa.sartori@unipd.it*

Motor resonance is defined as the internal activation of an observer's motor system, specifically attuned to the perceived movement. In social contexts, however, different patterns of observed and executed muscular activation are frequently required. This is the case, for instance, of seeing a key offered with a precision grip and received by opening the hand. Novel evidence suggests that compatibility effects in motor resonance can be altered by social response preparation. What is not known is how handedness modulates this effect. The present study aimed at determining how a left- and a right-handed actor grasping an object and then asking for a complementary response influences corticospinal activation in left- and right-handers instructed to observe the scene. Transcranial magnetic stimulation (TMS)-induced motor evoked potentials (MEPs) were thus recorded from the dominant hands of left- and right-handers. Interestingly, requests posed by the right-handed actor induced a motor activation in the participants' respective dominant hands, suggesting that left-handers tend to mirror right-handers with their most efficient hand. Whereas requests posed by the left-handed actor activated the anatomically corresponding muscles (i.e., left hand) in all the participants, right-handers included. Motor resonance effects classically reported in the literature were confirmed when observing simple grasping actions performed by the right-handed actor. These findings indicate that handedness influences both congruent motor resonance and complementary motor preparation to observed actions.

**Keywords: action observation, motor resonance, complementary actions, handedness, transcranial magnetic stimulation, motor evoked potentials**

# **INTRODUCTION**

A considerable amount of data suggests that primary motor and somatosensory cortices, as well as premotor and parietal areas, are modulated during action observation, providing evidence of an activation of the observer's motor system (i.e., motor resonance; see for example Grèzes and Decety, 2001; Avenanti et al., 2007, 2013a,b). Motor resonance is thought to result from the activity of neurons homologous to the mirror neurons described in the monkey ventral premotor cortex (di Pellegrino et al., 1992; Gallese et al., 1996). In humans, a large number of functional magnetic resonance imaging (fMRI) studies have provided reliable evidence that the action observation network (i.e., the neural network activated by seeing others' actions) largely overlaps with the brain network involved in action execution (Etzel et al., 2008; Gazzola and Keysers, 2009; Kilner et al., 2009; Turella et al., 2009; Oosterhof et al., 2010). Moreover, transcranial magneticstimulation (TMS) studies have shown a corticospinal excitability facilitation during action observation, suggesting a role for the primary motor area (M1) in motor resonance (Fadiga et al., 1995; Strafella and Paus, 2000; Gangitano et al., 2001; Aziz-Zadeh et al., 2002; Clark et al., 2004; Catmur et al., 2007; Enticott et al., 2010, 2011; Senot et al., 2011). In neural terms, the resonant response would originate in inferior frontal cortex (IFC, including ventral premotor cortex and posterior part of inferior frontal gyrus) and in inferior parietal lobule (IPL), and descend to spinal motoneurones via M1 (Nishitani and Hari, 2000). This is demonstrated by perturb-and-measure studies (Paus, 2005; Avenanti et al., 2007) in which off-line suppression of neural activity in IFC disrupts the motor facilitation induced by action observation (Avenanti et al., 2007, 2013a,b; Enticott et al., 2012) and dual coil studies in which stimulation of IFC and IPL modulates motor cortex reactivity to observed actions (Koch et al., 2010; Catmur et al., 2011). The involvement of M1 has been further confirmed by experiments in which the left M1 hand area was temporarily inactivated by TMS conditioning, resulting in the loss of the resonant H-reflex modulation in the corresponding right hand muscle (Borroni and Baldissera, 2008). Much of this work involved magnetic stimulation of the human primary motor cortex (M1) and electromyography (EMG) recording of participants' contralateral hand muscles while they were watching hand movements. The amplitude of motor evoked potentials (MEPs) recorded from hand muscles was found to be increased during observation of others' actions as the product of a specific corticospinal (CS) facilitation. In this connection, a question which so far has received little attention is whether the tendency to automatically resonate with others' actions is inflexible in terms of handedness. To date, as left-handed participants have often been excluded from studies in the past, our understanding of the relationship between motor resonance and motor dominance is quite limited. Preliminary evidence paved the way indicating that observation of a hand movement can modulate the excitability of motor neurons innervating hand muscles of both sides, irrespective of whether the right or left hand is observed (Borroni et al., 2008). Such a bilateral involvement indicates that motor resonance is not limited to a one-to-one correspondence, but it evokes the subliminal implementation of the full activation pattern utilized during execution, including other limbs' muscles. In this light, it is possible that the premotor cortex is engaged bilaterally in motor resonance during observation of either left or right hands because it does not code the laterality of the observed hand, but a more abstract representation of the movement (Borroni et al., 2008). On the other hand, brain imaging studies have reported the importance of the *observers*' hand dominance in shaping the pattern of motor resonant responses (e.g., Cabinio et al., 2010). In particular, right-handers showed a leftlateralized activation of the mirror neuron system (MNS) when observing/performing a right hand grasp, and a more bilateral but still left-lateralized cortical pattern when observing/performing the same action with the left (non-dominant) hand. The opposite pattern of cortical activation was shown in left-handers, although less lateralized. Along these lines, a series of fMRI studies assessed the role of handedness during execution and observation of simple movements in right- and left-handed participants (Rocca et al., 2008; Rocca and Filippi, 2010). Results showed different pattern of activations of the MNS in left-handers during the performance of movements with their dominant upper and lower limbs, suggesting a complex interaction between innate and daily-life background. These findings support the notion that lefthanders can adapt their actions to a world that has been built for right-handed people and that they deal with the vast majority of common tools by simply mirroring right-handers (Rocca et al., 2008).

Support to this contention comes from a recent study in which TMS-induced MEPs were recorded from the dominant and non-dominant hands of left-and right-handed participants while they observed a left-or a right-handed actor grasping an object (Sartori et al., 2013a). The anatomical correspondence between the observed and the observer's effector classically reported in the literature on motor resonance was confirmed in the dominant hand of both left-as well as right-handers observing actors with their same hand preference. But when the observed and observers' hand preference was mismatched, that anatomical correspondence disappeared. In particular, motor resonance was noted in left handers' dominant effector while they were observing both right- and left-handed actors. This seems to suggest a propensity to functionally shift the motor resonant activation to their own dominant hand, in line with neural evidence of more bilaterally spread brain functions in left-than in right-handers (Matsuo et al., 2002; Jorgens et al., 2007; Krombholz, 2008; Müller et al., 2011). The observer's handedness shapes the motor resonant response. What is still unknown, then, is whether the same mechanism applies when a different rather than a similar action is elicited by the observed agent. That is, when an actor is shown leaning toward the observer in a request gesture implying a complementary response.

In specific social contexts requiring incongruent complementary rather than imitative forms of interaction, motor resonance to action observation can be an unsuitable response (for reviews, see Sebanz et al., 2006; Knoblich et al., 2011). For instance, when we observe someone handing us a mug holding it by its handle, we will, without thinking, grab the mug with a whole-handgrasp (the most appropriate gesture to perform in this situation, though different from that observed). Along these lines, recent evidence seems to suggest that the inflexible tendency to match observed actions onto our motor system can be reconciled with the request to prepare incongruent responses (Newman-Norlund et al., 2007; Ocampo et al., 2011; Hamilton, 2013). In a series of recent psychophysiological studies, researchers assessed CS facilitation while participants observed video-clips evoking complementary gestures (i.e., an actor pouring coffee/sugar and then inviting them to pick up a cup placed in the video foreground) and video-clips simply showing an actor pouring coffee/sugar and then coming back to the starting position (Sartori et al., 2012, 2013b,c,d). Consistent results showed a natural switch from an imitative to a context-related action in CS activity. A matching mechanism at the beginning of an action sequence turned into a complementary one if a request to the observer for a reciprocal action became evident. In particular, TMS-induced MEPs recorded at the time the observer initially perceived a grasp on a target object elicited a motor facilitation in the participant's corresponding hand muscles. Conversely, when the observed gesture elicited a complementary reaction in the observer, participants' hand muscles revealed an activation matching the socially appropriate response which could be performed. As expected, when the observed action did not convey any request to the observer, congruent facilitation effects emerged during action observation.

Capitalizing on these results and recent insights from the handedness literature (Borroni et al., 2008; Rocca et al., 2008; Sartori et al., 2013a), the present study was designed to specifically determine how CS facilitation is modulated when an individual with the same or a different hand dominance elicits a congruent or incongruent motor resonance in the observer. TMSinduced MEPs were then recorded from muscles of each hand per block as the participants watched video-clips. Because participants remained at rest throughout the task, the degree to which the motor system is activated provides an index of CS activity elicited by action observation. Half of the clips showed an actor reaching and grasping an object with her right hand, pouring something and then either coming back (non-social action) or leaning toward an out-of-reach cup crucially located close to the observer and then prompting a complementary response (social action); the other half displayed the same actor performing the same action with her left hand. We expect that observing an actor with a different hand preference might elicit different patterns of CS activation in right- and left-handers. Specifically, if left-handers are prone to functionally shift the motor resonant and complementary activation to their own dominant hand, then leftward activations should be noticed in all the experimental conditions. Otherwise, if handedness does not shape motor resonance, a mirroring pattern of CS facilitation should be found in all the participants. To date, no previous studies have investigated handedness and motor resonance in social contexts by means of TMS and EMG recording. In terms of action observation this might be a timely and tractable issue.

# **MATERIALS AND METHODS**

# **PARTICIPANTS**

Thirty right-handed (17 females and 13 males, mean age 24 years, range 19–56) and 30 left-handed (24 females and 6 males, mean age 23 years, range 20–47) participants took part in the experiment. The participants' degree of handedness was evaluated using a modified version of the Edinburgh Inventory (EHI) (Oldfield, 1971; Salmaso and Longoni, 1983). We converted the EHI total score into a dichotomous variable by computing the laterality quotient (LQ) that ranges from −100 (strong left handedness) to +100 (strong right-handedness), through the following standard expression: LQ = (R−L)/(R+L)∗100. R and L represent the total number of right-and left-hand items endorsed, respectively. A score below 0 (included) identified left-handed participants, while LQ *>* 0 detected right-handed participants. The LQ ranged between −100 and −11 (mean: −65) for the left-handed participants. For the right-handed participants, it ranged between 64 and 100 (mean: 88). None of the participants had any neurological, psychiatric, or other medical problems, nor did they have any contraindication to TMS (Wassermann, 1998; Rossi et al., 2009). None were aware of the experiment's purpose and all gave their written informed consent at the time they were recruited. The study protocol was approved by the Ethics Committee of the University of Padova and was carried out in accordance with the principles of the Declaration of Helsinki. None of the participants reported experiencing discomfort or adverse effects during the experiment.

# **EXPERIMENTAL STIMULI**

The stimuli were four digitally recorded video clips showing a right-handed actor naturally reaching and grasping an object located close to her hand (**Figures 1A–H**): in the first, the actor reached and grasped a sugar spoon (a), poured some sugar on three cups located nearby and then stretched out her arm trying to pour some sugar on a forth cup located out of her reach (b); in the second, the actor reached and grasped a sugar spoon (c), poured some sugar on three cups located nearby and then took back the sugar spoon to the starting point (d); in the third, the actor reached and grasped a thermos (e), poured some coffee on three coffee cups located nearby and then stretched out her arm trying to pour some coffee on a forth coffee cup located out of her reach (f); in the fourth and last, an actor was shown reaching and grasping a thermos (g), pouring some coffee on three coffee cups located nearby and then taking back the thermos to the starting point (h). The four video clips were then reflected on a horizontal plane using video editing procedures so that the actor appeared to be reaching and grasping the same object with her left hand (**Figures 1 I–P**), for a total of eight video clips. All of the videos were taken from a frontal view, clearly showing the model grasping the sugar spoon with a precision grip (PG; i.e., the opposition of the thumb with the index finger) and the thermos with a whole-hand grasp (WHG; i.e., the opposition of the thumb with the other fingers). Crucially, the out-of-reach object was located in the video foreground, closer to the participant watching the video, thus eliciting a complementary reaction with a whole-hand grasp on the big cup and with a precision grip on the coffee cup respectively. A preliminary pilot investigation, carried out with a questionnaire and the assistance of a group of 10 participants with characteristics that were similar to those participating in the study experiment, confirmed that the social type of action (i.e., the actor leaning toward the observer) was recognized by the participants as a request to grasp the salient object (98% of positive responses).

# **DATA RECORDING**

# *Transcranial magnetic stimulation*

Single-pulse TMS (pulse characteristics: 100μs rise time, 1 ms duration) was delivered using a 70 mm figure-of-eight coil (Magstim polyurethane-coated coil) connected to a Magstim BiStim<sup>2</sup> stimulator (The Magstim Company, UK). Pulses were delivered to the left and right M1 areas corresponding to the hand region in two separate blocks ("left M1" and "right M1" blocks, respectively). The coil was placed tangentially on the scalp, with the handle pointing laterally and caudally (Brasil-Neto et al., 1992; Mills et al., 1992). The coil was positioned in correspondence with the optimal scalp position (OSP), defined as the position at which TMS pulses of slightly suprathreshold intensity consistently produced the largest MEP from the ADM muscle. The OSP was determined by moving the intersection of the coil in approximately 0.5 cm steps around the target area until a position was reached at which a maximal MEP amplitude was produced in the target muscle with a minimal stimulation intensity. This position was marked on a tight-fitting cap that each participant was asked to wear. During the experimental sessions the coil was held by a tripod with an articulated arm. The position and orientation of the coil over the OSP was recorded and loaded into the Brainsight 2.0 neuronavigation system (Rogue Research, Montreal QC) to maintain accurate placement of the coil throughout the experiment. Defined as the minimum stimulation intensity on the OSP that induced reliable MEPs (≥50μV peak-to-peak amplitude) in a relaxed muscle of the dominant hand in five out of ten consecutive trials, the individual resting motor threshold (rMT) was determined for each participant (Rossini et al., 1994). The same stimulation intensity (110% of the rMT) was used for the left and right M1 sessions in each subject. Stimulation intensity during the recording session ranged between 40 and 70% of the maximum stimulator output intensity (mean 53%) for the right-handed participants. For the left-handed participants, it ranged between 39 and 61% of the maximum stimulator output intensity (mean 54%).

# *Electromyography*

MEPs were recorded from the first dorsal interosseus (FDI) and abductor digiti minimi (ADM) muscles of the right and left arms in separate blocks. Electromyography (EMG) activity was recorded through pairs of surface Ag-AgCl cup electrodes (9 mm diameter) placed in a belly-tendon montage. The ground electrode was placed over the participants' ipsilateral wrist. Electrodes were connected to an isolable portable ExG input box linked to the main EMG amplifier for signal transmission via a twin fiber optic cable (Professional BrainAmp ExG MR). The raw myographic signals were band-pass filtered (20 Hz–1 kHz), amplified prior to being digitalized (5 KHz sampling rate), and stored on a computer for off-line analysis. EMG data were recorded for a

and grasp a sugar spoon **(A)**, then she stretches out her arm trying to pour some sugar on a cup located out of her reach **(B)**. The actor reaches and grasp the same sugar spoon **(C)**, but then she takes it back to the starting position **(D)**. The actor reaches and grasp a thermos **(E)**, then she stretches out her arm trying to pour some coffee on a coffee cup located out of her reach **(F)**. The

that the actor appears to perform the same social and non-social actions, but with her left-hand. T1 and T2 are time-locked at the moment the actor makes contact with the object, and at the end of the action sequence. Red squares highlight the frames in which the out-of-reach object located in the video foreground elicits a complementary reaction: either a WHG **(B, J)** or a PG **(F, N)**.

300 ms interval. The interval was time-locked to the delivery of each magnetic stimulation pulse and began 100 ms prior to the onset of stimulation and ended 200 ms post-stimulation. Trials in which any EMG activity was present in the time window preceding the TMS pulse were discarded to prevent contamination of MEP measurements by background EMG activity.

# **PROCEDURE**

The participants were tested individually in a sound-attenuated Faraday room during a single experimental session lasting approximately 40 min and consisting in two blocks (left M1, right M1). Each participant was directed to sit in a comfortable armchair with his/her head positioned on a fixed head rest so that the eye–screen distance was 80 cm. Both arms were positioned on full-arm supports. Each participant was instructed to keep his/her hands in a prone position and as still and relaxed as possible. The task was to pay attention to the visual stimuli presented on a 19" monitor (resolution 1280 × 1024 pixels, refresh frequency 75 Hz, background luminance of 0.5 cd/m2) set at eye level. The participants were instructed to passively watch the video-clips and to avoid making any movements. To ensure that the participants paid attention to the contents of the video clips, they were told that they would be questioned at the end of the session about the visual stimuli presented. Electromyography recordings were made in the contralateral hand (**Figure 2B**). During the "left M1" blocks, TMS-induced MEPs were acquired from the participant's right ADM and FDI muscles during stimulation of the left M1. During the "right M1" blocks, MEPs were acquired from the participant's left ADM and FDI muscles during stimulation of the right M1. The order in which the two blocks were delivered was counterbalanced across participants. Prior to the video presentation, a baseline corticospinal excitability was assessed by acquiring 10 MEPs per block while the participants passively watched a white fixation cross on the black background on the computer screen. Ten more MEPs were recorded at the end of each block. By comparing the MEP amplitudes for the two baseline series it was possible to check for any corticospinal excitability changes related to TMS *per se* in each block. The average amplitude of the two

collapsed series was utilized to set each participant's individual baseline for the data normalization process.

All the participants watched four types of video-clips presented in random order:


The MEPs were recorded from the ADM muscle (i.e., the muscle serving little finger abduction) and FDI muscle (i.e., the muscle serving index finger abduction) due to their involvement respectively in WHG and PG. Crucially, each video clip was characterized by a mismatch between the type of grasp being observed (i.e., WHG) and the grip implicitly being requested to the observer (i.e., PG). Specifically, observing the grasp on the thermos and the large cup should elicit a pronounced activation in both FDI and ADM muscles because such muscles are involved in a WHG. When observing the grip on the sugar spoon and the coffee cup, instead, only MEPs recorded from the FDI muscle should reveal an increase because a PG does not imply the recruitment of the ADM muscle. A single TMS pulse was released during each video presentation at two specific time points: (i) during the frame showing the actor's fingers making contact with the object (T1; 1125 ms) and (ii) during the frame showing the lowest peak of the actor's arm trajectory (T2; 5900 ms; **Figure 2A**). The same timing was applied to all of the non-social conditions. The first time point (T1) was chosen to evaluate the motor resonant response. As recently demonstrated by Lago and Fernandez-del-Olmo (2011), a muscle-specific motor program is activated via the action observation system when the contact between an effector and an object is shown. The second time point (T2) was set at the lowest peak of the arm's trajectory to maximize the reaction to the implicit request, as identified by kinematics (Sartori et al., 2013c,d) and modeling studies (Chinellato et al., 2013) with stimuli similar to those adopted in the present study. The order of the videos and of the two different TMS delays were randomized within each of the two blocks. A total of 640 MEPs (4 muscles × 2 types of action × 2 actors × 2 types of grasp × 2 time points × 10 repetitions) was recorded for each participant. Prior to presenting the videos, each participant's baseline CS excitability was assessed by acquiring 10 MEPs per block while they passively watched a white-colored fixation cross on a black background on the computer screen. Ten more MEPs were recorded at the end of each block. By comparing MEP amplitudes recorded during the two baseline series it was possible to check for any CSE changes related to TMS *per se* in each block. The average amplitude of the two series was then utilized to set each participant's individual baseline for data normalization procedure. An inter-pulse interval lasting 10 s was presented between trials in order to minimize the potential risk of carryover effect of a TMS pulse on the subsequent one. During the first 5 s of the rest period, a message reminding the participants to keep their hands still and fully relaxed appeared on the screen. A fixation cross (10 × 10 mm) was presented for the remaining 5 s. Stimuli presentation, EMG recordings and timing of TMS stimulation were managed by E-Prime V2.0 software (Psychology Software Tools) running on a PC.

#### **DATA ANALYSIS**

The CS facilitation of FDI and ADM muscles was quantified at each stimulation point during each experimental condition by the MEP peak-to-peak amplitude (mV). Those amplitudes deviating more than 3 standard deviations from the mean and the trials contaminated by muscular pre-activation were excluded as outliers (*<*5%). A paired-sample *t*-test (2-tailed) was used to compare the amplitude of MEPs recorded during the two baseline trials carried out at the beginning and at the end of each block. Ratios were computed using the participants' individual mean MEP amplitude recorded during the two fixation-cross periods as baseline (MEP ratio = MEPobtained/MEPbaseline). We entered the MEP ratios in a mixed-design analysis of variance (ANOVA) with "muscle" (right FDI, right ADM, left FDI, left ADM), "type of action" (social, non-social), "actor" (right-handed, lefthanded), "type of grasp" (PG, WHG) and "stimulation time" (T1, T2) as within-subjects factors, and "group" (right-handed, lefthanded) as between-subjects factor. The sphericity of the data was verified prior to performing statistical analysis (Mauchly's test, *p >* 0*.*05). *Post-hoc* pairwise comparisons were carried out using *t*-tests and Bonferroni correction was applied to control *P*-values for multiple comparisons. A significance threshold of *P <* 0*.*05 was set for all statistical analyses.

# **RESULTS**

The mean raw MEP amplitudes recorded during the two baseline series at the beginning and the end of each block were not significantly different in the right-handed participants neither during the "left M1" block [1406.15 vs. 1330.67μV, respectively; *t*(59) = 0*.*48, *p* = 0*.*63] nor the "right M1" block [1132.99 vs. 916.59μV, respectively; *t*(59) = 1*.*96, *p* = 0*.*07]. Similarly, the two baseline series were not significantly different in the lefthanded participants neither during the "left M1" block [1796.58 vs. 1745.20μV, respectively; *t*(59) = 0*.*31, *p* = 0*.*76] nor the "right M1" block [1388.17 vs. 1101.22μV, respectively; *t*(59) = 1*.*94, *p* = 0*.*06]. Altogether these findings suggest that TMS *per se* did not induce any changes in corticospinal excitability during our experimental procedure. The mean MEP ratios from the left and right ADM and FDI muscles for each group are outlined in **Table 1**. The mixed-design ANOVA on the normalized MEP amplitudes showed a significant main effect of muscle [*F*(3*,* 174) = 2*.*80, *p <* 0*.*05, η<sup>2</sup> <sup>p</sup> = 0*.*05] and stimulation time [*F*(1*,*58) = 22*.*56, *p <* 0*.*001, η<sup>2</sup> <sup>p</sup> = 0*.*28]. The following interactions were also significant: "muscle by stimulation time" [*F*(1*,* 174) = 3*.*72, *p <* 0*.*05, η2 <sup>p</sup> = 0*.*06], "muscle by actor by type of action" [*F*(3*,* 174) = 2*.*98, *p <* 0*.*05, η<sup>2</sup> <sup>p</sup> = 0*.*05], "muscle by actor by type of grasp by type of action" [*F*(3*,* 174) <sup>=</sup> <sup>2</sup>*.*74, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*05, <sup>η</sup><sup>2</sup> <sup>p</sup> = 0*.*05], "muscle by type of grasp by type of action by stimulation time" [*F*(3*,* 174) = 4*.*20, *p <* 0*.*05, η<sup>2</sup> <sup>p</sup> = 0*.*07], "actor by type of grasp by stimulation time by group" [*F*(1*,* 58) <sup>=</sup> <sup>4</sup>*.*27, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*05, <sup>η</sup><sup>2</sup> <sup>p</sup> = 0.07], "muscle by actor by type of action by group" [*F*(3*,* 174) <sup>=</sup> <sup>2</sup>*.*84, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*05, <sup>η</sup><sup>2</sup> <sup>p</sup> = 0.05] and "muscle by actor by type of grasp by type of action by stimulation time" [*F*(3*,* 174) <sup>=</sup> <sup>4</sup>*.*81, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*05, <sup>η</sup><sup>2</sup> <sup>p</sup> = 0.08]. The results obtained for *post-hoc* contrasts stemming from the five-way interaction are reported as follows.

# **EFFECTS OF MOTOR RESONANCE**

#### *Left-handed actor*

*Post-hoc* comparisons revealed a reliable activation in all of the participants' left hand when observing a left-handed actor. In particular, observing the left-handed actor grasping a thermos at T1 with both a social and non-social type of action induced a greater activation in the left ADM muscle compared to the ipsilateral FDI muscle (*ps <* 0.05; **Table 1**). This was confirmed for the non-social type of action at T2 by an increase in the left ADM muscle compared to the ipsilateral FDI muscle (*p <* 0*.*05; **Table 1**) and compared to the video in which the actor was grasping a sugar spoon (non-social type of action; *p <* 0*.*05; **Table 1**). Interestingly, observing the left-handed actor holding a thermos in the non-social type of action at T2 prompted a greater activation in the left ADM muscle that observing the very same action performed by the right-handed actor (*p <* 0*.*05; **Table 1**). Furthermore, *post-hoc* analysis on the four-way interaction "actor by muscle by type of action by group" showed that observing the left-handed actor performing a non-social action induced a greater activation in the left hand of both right and left-handers, with respect to their ipsilateral FDI muscles (*ps <* 0.05). This suggests that motor resonance to an observed action performed by a left-handed actor is likely to activate the anatomically corresponding muscles (i.e., left hand) in both right- and left-handers.

# *Right-handed actor*

*Post-hoc* comparisons revealed a mixed pattern of activation when observing the right-handed actor. In particular, a classical increase in the right ADM muscle was found when observing the actor performing a WHG on the thermos compared to a PG on the sugar spoon for both the social and non-social types of actions at T1 and T2 (*ps <* 0.05; **Table 1**). But observing the right-handed actor grasping a thermos (WHG) at T1 also induced an increase in both right and left ADM muscles with respect to the corresponding ipsilateral FDI muscles (*ps <*0.05; **Table 1**). This seems to suggest that participants were resonating with both hands. Statistically significant differences were also found in both FDI muscles when observing a WHG compared to a PG at T1 and T2 (*ps <* 0.05; **Table 1**). These results are in line with the literature on reach-to-grasp kinematics, suggesting a major involvement of FDI during precision grips compared to whole-hand grasps (Sartori et al., 2012).

# **EFFECTS OF RECIPROCITY**

# *Left-handed actor*

*Post-hoc* comparisons revealed that observing the left-handed actor holding the sugar spoon and leaning toward the out of reach cup eliciting a WHG in the participant's hand induced a predictable increase in the left ADM muscle at T2 with respect to T1(*p <* 0*.*05). The same was found with respect to the contralateral ADM muscle (*p <* 0*.*05), to the ipsilateral FDI muscle (*p <* 0*.*05), to the non-social type of action showing the actor simply holding the sugar spoon back to the starting point (*p <* 0*.*05), to the other social action eliciting a PG toward the coffee cup (*p <* 0*.*05), and to the very same action performed by the right-handed actor. A significant decrease in MEPs activity was also found in the left ADM muscle when observing the actor holding the thermos and leaning toward the out of reach coffee cup eliciting a PG in the participant's hand, with respect to the non-social action (*p <* 0*.*05). Observing the left-handed actor performing a complementary request in the social types of actions induced in right-handers a greater muscular activation of left hand muscles with respect to observing the non-social actions (*ps <* 0.05). Interestingly, a greater activation of the left ADM muscle was found in right-handers and left-handers with respect to their ipsilateral FDI muscles (*ps <* 0.05; **Figure 3A**) for the social PG actions performed by the left-handed actor.


**Table 1 | Normalized mean (± s.e.m.) peak to peak amplitude of MEPs recorded from the ADM and the FDI muscles of both groups during the two stimulation blocks for each type of observed actor, observed grasp and type of action at each stimulation time point.**

## *Right-handed actor*

*Post-hoc* comparisons revealed that observing the right-handed actor holding the sugar spoon and leaning toward the out of reach cup eliciting a WHG in the participant's hand induced a predictable increase in the right ADM muscle at T2 with respect to T1 (*p <* 0*.*05). The same was found with respect to the ipsilateral FDI muscle (*p <* 0*.*05), to the non-social type of action showing the actor simply holding the sugar spoon back to the starting point (*p <* 0*.*05), to the other social action eliciting a PG toward the coffee cup (*p <* 0*.*05), and to the very same action performed by the left-handed actor. A significant decrease in activation was also found in the right ADM muscle when observing the other video clip showing the actor holding the thermos and leaning toward the out of reach coffee cup eliciting a PG in the participant's hand, with respect to the non-social action of holding the thermos back to the starting point (*p <* 0*.*05). The effect of complementary activation previously described in the literature was confirmed (Sartori et al., 2012, 2013b,c,d). But an increase was found at T2 also in the left ADM muscle with respect to the ipsilateral FDI muscle (*p <* 0*.*05) for the social type of action requiring a WHG on the cup. Interestingly, observing the right-handed actor performing a complementary request induced a greater muscular activation with respect to observing the non-social actions (*ps <* 0.05; **Table 1**). An effect supported by previous literature (Sartori et al., 2011). But this increase was only evident in the right hand of right-handers and in the left hand of left-handers, suggesting that they translated the observed movement into their dominant effector for planning the most appropriate response. This was confirmed by the greater activation of the right ADM muscle in right-handers and of the left ADM muscle in left-handers with

hands of schematic drawings representing the participants highlight that left-handers activate the left hands independently from the object's location. Whereas right-handers activate the left hand when the object is located to their left side, and the right hand when the object is located to their right side.

respect to their ipsilateral FDI muscles (*ps <* 0.05; **Figure 3B**) for the social PG actions performed by the right-handed actor.

# **DISCUSSION**

The main aim of the present study was to bring a substantial advancement in our knowledge of the role played by hand dominance in modulating motor resonant and complementary responses in social contexts. Are motor resonance and reciprocity shaped by handedness?

To test this issue, we adopted video clips showing a righthanded actor performing social and non-social actions eliciting in the observer congruent and incongruent types of motor activations, along with the very same actions performed by a lefthanded actor (i.e., obtained through digital flipping of the original ones). Participants were both right and left-handers. Results show that, independently from group handedness, motor resonance effects strictly linked to the observed muscles emerged in all participants, though with a more unilateral pattern of activations when observing a left- with respect to a right-handed actor. This effect could be explained on the basis of previous findings showing that left-handers tend to translate any observed motor program into their dominant effector (Sartori et al., 2013a). This is in agreement with previous evidence of more bilaterally spread brain functions in left-than in right-handers (Matsuo et al., 2002; Jorgens et al., 2007; Krombholz, 2008; Müller et al., 2011). In neural terms, very few studies have tried to shed light on the underpinnings of hand grasping actions in both right-and left-handers (e.g., Begliomini et al., 2008). In this respect, evidence suggests a specific right hemisphere contribution to grip formation (Hermsdorfer et al., 1999; Farne et al., 2003), and in particular a significant role of the right dorsal premotor cortex (dPMC) in the control of goal-related hand movements depending on handedness (Begliomini et al., 2008). Specifically, a similar activity within the right dPMC for both right-and left-handers was found when they performed the task with the right hand, and a different activity between the two groups was found when the left hand was used. This was evident when looking at the significant increase in activation when left-handers used the dominant left rather than the right hand. This observation is in line with our data demonstrating a preferential leftward hand activation in left-handers observing both left-and right-handers, and with the anatomical observation of differences in inter-hemispheric connections in relation to handedness (Amunts et al., 2000). And it might also suggest differences in the functional organization motor and premotor areas in right- and left-handed people (Solodkin et al., 2001).

In view of the fact that motor resonance reflects the motor representation evoked by a perceived action in an observer, our results suggest that in the context of a social request directed to the observers, independently from their handedness, the perceptualmotor matching of the observed action give the way to an incongruent activation in the muscles directly involved in the interaction. That is, motor activation in right handers is found in the right hand when the actor asks for a right complementary gesture and in the left hand when the actor asks for a left complementary action. This supports the hypothesis of a sophisticated model of motor resonance. The direct-matching hypothesis postulates that viewing an action automatically evokes in the observer a representation of the motor commands necessary to execute that same action. TMS experiments typically show that observed movements are processed in a strictly time-locked, muscle specific fashion (Baldissera et al., 2001; Gangitano et al., 2001; Borroni et al., 2005; Montagna et al., 2005; Borroni and Baldissera, 2008; Candidi et al., 2008; Alaerts et al., 2009; Cavallo et al., 2011). However, when a complementary reaction is implicitly required by the observed agent, incongruent patterns of motor activations take place (Sartori et al., 2012, 2013b,c,d; Hamilton, 2013). The findings outlined here, suggesting that the perceptual-motor mapping of a movement is also sensitive to the observed handedness complement those studies and take research one step further.

Another explanation for this effect could be ascribed to the motor coding of action affordance elicited by the salient object in the social type of action. This would point to a mechanism for recognizing "social affordances," that is specific types of affordances (Gibson, 1979; Jeannerod, 1994; Craighero et al., 1998; Tucker and Ellis, 1998; Buccino et al., 2009) produced by the establishment of a shared intentional space (Tomasello, 1999). The implicit request by the actor -facing the participants-toward the object inside their peripersonal space is a crucial aspect which favors a readiness to engage in a complementary interaction (Costantini et al., 2010, 2011). In line with this, we specifically devised control conditions in which the actor was finally directed to bring her hand back to its initial position, despite the presence of the fourth object still visible in the foreground. That control conditions were created in order to detach the role of the intentional request from object affordances. Indeed, the present results seem to suggest that only making affordances salient evokes a readiness to enact them. As long as an object becomes relevant to the goal of an action, it is conceivable that a highly efficient mechanism enables subjects to correctly plan movements toward this target in a functional action-specific mode. And this indeed happens in right-handers. Whereas left-handers tend to persistently activate their dominant effector. With respect to the relation between motor resonance, reciprocity, and dominance, our results extend previous evidence, showing that the observed handedness differently shapes motor resonant and complementary reactions in right-and left-handers. Assuming that this modulation might be an index of motor representations' capability of taking into account the observed hand dominance and the target location, the findings outlined here can support the evidence of a sophisticated mechanism allowing right handers to plan movements toward the target in a functional action-specific mode and left-handers to convert another person's pattern of movement into their optimal motor commands.

# **AUTHOR CONTRIBUTIONS**

Luisa Sartori contributed to the acquisition, analysis and interpretation of data, and to drafting the work. Chiara Begliomini contributed revising the draft critically. Giulia Panozzo and Alice Garolla contributed to the acquisition of data for the work. Umberto Castiello contributed to drafting the work and revising it for important intellectual content.

# **ACKNOWLEDGMENTS**

This work was supported by a grant from the MIUR, a grant N. 287713 of FP7: REWIRE project and the Progetto Strategico, Università di Padova to Umberto Castiello.

# **REFERENCES**


*Hum. Brain Mapp.* 12, 1–19. doi: 10.1002/1097-0193(200101)12:1*<*1::AID-HBM10*>*3.0.CO;2-V


the brain, spinal cord and roots: basic principles and procedures for routine clinical application. *Electroencephalogr. Clin. Neurophysiol.* 91, 79–92. doi: 10.1016/0013-4694(94)90029-9


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 16 April 2014; accepted: 21 August 2014; published online: 09 September 2014.*

*Citation: Sartori L, Begliomini C, Panozzo G, Garolla A and Castiello U (2014) The left side of motor resonance. Front. Hum. Neurosci. 8:702. doi: 10.3389/fnhum. 2014.00702*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Sartori, Begliomini, Panozzo, Garolla and Castiello. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Visual field preferences of object analysis for grasping with one hand

# *Ada Le and Matthias Niemeier\**

*Psychology, University of Toronto Scarborough, Toronto, ON, Canada*

#### *Edited by:*

*Simona Monaco, York University, Canada*

#### *Reviewed by:*

*Daniel Kent Wood, Northwestern University, USA Robert Leslie Whitwell, The University of Western Ontario, Canada*

#### *\*Correspondence:*

*Matthias Niemeier, University of Toronto Scarborough, 1265 Military Trail, Toronto, ON M1C 1A4, Canada e-mail: niemeier@utsc.utoronto.ca*

When we grasp an object using one hand, the opposite hemisphere predominantly guides the motor control of grasp movements (Davare et al., 2007; Rice et al., 2007). However, it is unclear whether visual object analysis for grasp control relies more on inputs (a) from the contralateral than the ipsilateral visual field, (b) from one dominant visual field regardless of the grasping hand, or (c) from both visual fields equally. For bimanual grasping of a single object we have recently demonstrated a visual field preference for the left visual field (Le and Niemeier, 2013a,b), consistent with a general right-hemisphere dominance for sensorimotor control of bimanual grasps (Le et al., 2014). But visual field differences have never been tested for unimanual grasping. Therefore, here we asked right-handed participants to fixate to the left or right of an object and then grasp the object either with their right or left hand using a precision grip. We found that participants grasping with their right hand performed better with objects in the right visual field: maximum grip apertures (MGAs) were more closely matched to the object width and were smaller than for objects in the left visual field. In contrast, when people grasped with their left hand, preferences switched to the left visual field. What is more, MGA scaling with the left hand showed greater visual field differences compared to right-hand grasping. Our data suggest that, visual object analysis for unimanual grasping shows a preference for visual information from the ipsilateral visual field, and that the left hemisphere is better equipped to control grasps in both visual fields.

**Keywords: grasping, visual field effect, contralateral, lateralization, unimanual**

# **INTRODUCTION**

Vision plays a crucial role in the sensorimotor control of actions. To grasp an object, the brain may analyze visual input to estimate grasp-relevant object features. For example, an object's shape and size, center of mass, and apparent surface friction are relevant to identify grasp points on the surface of the object (Blake, 1992; Voudouris et al., 2012). These points may then guide grasp movements, especially during "precision grips," such as with the thumb in opposition to the index finger of the same hand (Napier, 1956).

Grasp movements originate from visuomotor control mechanisms that are computed by a cortical network in the inferior frontal and intraparietal cortex (Castiello, 2005; Castiello and Begliomini, 2008; Grafton, 2010; Davare et al., 2011). The hub of this dorsolateral network is the anterior intraparietal sulcus (aIPS; Culham et al., 2003; Frey et al., 2005) which has been shown to implement the initial steps of the visual analysis for grasps (Rizzolatti and Luppino, 2001; Tunik et al., 2005, 2008; Culham and Valyear, 2006; Castiello and Begliomini, 2008; Grafton, 2010; Le et al., 2014) as well as perform ensuing transformations for visuomotor control (Castiello, 2005; Davare et al., 2007, 2010; Cavina-Pratesi et al., 2010; Koch et al., 2010; Monaco et al., 2013; Theys et al., 2013).

Disrupting aIPS activity with transcranial magnetic stimulation (TMS) impacts the prehension component of reach-to-grasp trajectories in a contralateral manner such that stimulation in one hemisphere affects the movements of the hand on the respective opposite side of the body (Rice et al., 2007). Nevertheless, TMS paradigms also suggest a left aIPS dominance for certain aspects of the grasp such as grip force control (Davare et al., 2007).

Consistent with a relative left-hemisphere dominance are data from behavioral and fMRI studies. For instance, the scaling of right hand grasps is less affected by size-contrast illusions than grasps with the left hand (Gonzalez et al., 2006). Also, grip-type selection predominantly activates the left ventral premotor cortex, regardless of the hand dominance (Martin et al., 2011). In addition, other fine motor skills show an equivalent right hand advantage (Serrien et al., 2006). In sum, sensorimotor control of grasping with one hand shows a contralateral organization with a relative dominance of the left hemisphere on the side of motor output.

To date, however, it is unclear whether lateralized motor control is complemented by an equivalent visual field preference or dominance at an earlier stage of visual object analysis for grasp computations. That is, does grasp control (a) rely more on visual object analysis in the contralateral than the ipsilateral visual field, does control (b) show a general preference for the right visual field, or does it (c) use information from both visual fields equally?

To our knowledge, data about visual field preferences for grasping are largely incomplete. Le and Niemeier (2013a,b) tested bimanual rather than unimanual grasping and found a visual field preference for the left visual field. This shows that it is feasible to assume that unimanual grasping prefers one visual field as well, given that the mechanisms underlying bimanual grasping and unimanual grasping partially overlap (Le et al., 2014). Even so, this does not necessarily imply that unimanual grasping would show a visual field preference. Shmuelof and Zohary (2006)found that brain activity in the right hemisphere varied as a function of visual field but not left hemisphere activity. However, these authors had participants observe unimanual grasp actions, rather than perform them; thus, their observations can only provide indirect evidence about visual object analysis for grasping.

To directly answer the question of a visual field preference for unimanual grasping, here we used a visual field paradigm. Participants grasped either with their right hand (Experiment 1) or left hand (Experiment 2) while viewing objects either in their left or right visual field. For right-hand grasping, we found that maximum grip apertures (MGA) were more closely matched to object width and were smaller for objects in the right visual field than for objects in the left visual field, indicating a left hemisphere advantage. In contrast, for left-hand grasping, we found a left visual field advantage, consistent with a greater involvement of the right hemisphere. What is more, left-hand grasping showed greater differences between left and right visual field. Together, our data suggest that visual object analysis for unimanual grasping prefers visual information from the contralateral visual fields and that the left hemisphere may be better equipped to control right-handed grasp movements in both visual fields.

# **METHODS**

#### **PARTICIPANTS**

A total of 49 healthy undergraduate students (Experiment 1: *N* = 28, 14 females, mean age of 20 years; Experiment 2, *N* = 21, 12 females, mean age of 21 years) gave their informed and written consent to participate in this study. All participants had normal or corrected to normal vision, and were right handed as confirmed with the Edinburgh handedness inventory [Experiment 1: laterality quotient = 76.4; Experiment 2: laterality quotient = 92.2; *t*(47) = −2*.*36, *p* = 0*.*023; Oldfield, 1971]. All procedures were approved by the Human Participants Review Sub-Committee of the University of Toronto and therefore have been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

#### **APPARATUS**

Participants sat in a dark room (*Lv* <sup>=</sup> <sup>0</sup>*.*01 cd/m2, measured at the location of the object) at a table with their head stabilized in a chin rest and their sight controlled by a set of Plato goggles (Translucent Technology, Toronto). A 19 inch LCD monitor (1024 × 768 pixels, 100 Hz refresh rate) was mounted on the table 60 cm away from the participant at eye level. Twenty centimeters in front of the monitor and aligned with the participant's body midline, we installed a pedestal on which we placed a gray wooden block (75 mm by 50 mm by 24 mm) 3 cm below eye level. Also, a tactile marker on the table 24 cm in front of the participant's trunk served as a start position for the hand movements. Hand trajectories were recorded with three infrared Qualisys motion tracking cameras (Qualisys, 240 Hz) and passive spherical markers (10 mm in diameter) fixed on the tips of the index finger, thumb, and on the wrist at the junction of the ulna and the carpal. Eye position of the left eye was tracked while the Plato goggles were transparent using an EyeLink II system (SR Research, Ottawa; sampling rate: 250 Hz) to allow us to exclude trials with improper fixation. Both, eye tracking and visual stimuli were controlled by Matlab (MathWorks) together with the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) and Eyelink Toolbox extensions (Cornelissen et al., 2002). The Plato goggles were controlled by a custom-made program.

# **PROCEDURE**

A spatial and temporal illustration of an experimental trial is given in **Figure 1** and adopted from Le and Niemeier (2013a). In brief, at the beginning of each trial, the Plato goggles were opaque and the participants rested their index finger and thumb adjacent to each other on the tactile start position. Meanwhile, the experimenter placed, in darkness, the object on the pedestal so that its 75 × 50 mm side faced the participant. Its orientation was chosen to be horizontal or vertical following a random protocol generated by the Matlab program and displayed as a small-fonted "H" or "V" on the monitor (i.e., invisible through the translucent goggles). Next, hand position recordings were started and this triggered the goggles to turn transparent so that the pupils became visible to the eye tracker. The Matlab program waiting for this signal then presented a red fixation dot (∼0.5 visual degrees in diameter) 15 visual degrees to the left or to the right of the object. Participants were asked to move their eyes to the dot and fixate it to manipulate visual field presentation. Note that fixating to the left of the object brings the object into the right visual field ("right VF"), whereas fixating to the right brings the object into the left visual field ("left VF"). Also, it is important to note that the low luminance levels during the initial fixation period made it very unlikely that any object information, useful for grasping, entered the visual system (e.g., in pilot tests no conscious object perception was possible even with 20 min dark adaptation, whereas the actual experiment prevented dark adaptation). Seven hundred milliseconds after initial fixation, the screen background became white, thus illuminating the object so that it appeared in the participant's right or left visual field (manipulating visual fields with different fixation locations is one commonly used strategy, e.g., Macaluso et al., 2002; it avoids biomechanical confounds because reach-to-grasp movements are kept the same). Participants then moved their hands (Experiment 1: right hand; Experiment 2: left hand) to grasp the object at its left and right sides and lift it off the pedestal (i.e., a horizontal object orientation required grasps across the wide object side, a vertical orientation required grasps across the narrow side). Only precision grasps with index finger and thumb were permitted, power grasps were not permitted because it is unclear to which extent they require detailed grasp point computations (Ehrsson et al., 2000). Participants' grasps were visually monitored during the illumination period to ensure they grasped as instructed. Participants were told to move as fast as possible without sacrificing accuracy. After 2500 ms, the hand tracking stopped, the monitor turned black, and the goggles became opaque once again.

**FIGURE 1 | (A) Participants were asked to fixate either to the left or right of the object.** When they fixated to the left, the object was in their right visual field. When they fixated to the right, the object was in their left visual field. The object was presented either horizontally (dotted-line; i.e., participants grasp the widest dimension of the object), or vertically (solid-line; i.e., participants grasp the narrow dimension of the object). To grasp the object, participants used the index finger and thumb to pick up the object from its left and right sides. **(B)** Time sequence of one trial: As soon as the Plato goggles opened, the fixation point (FP) was visible and participants fixated on it. Seven hundred and fifty milliseconds after the onset of the FP, the monitor ("lamp") turned white, thus illuminating the object. As soon as the "lamp" came on, participants grasped the object. Reaction time was calculated as the time from "lamp" onset to movement onset. Time of maximum grip aperture (tMGA) was calculated as the time from movement onset to maximum grip aperture (MGA).

Thirty of such trials were conducted in each block, and there were 2 blocks in total.

#### **DATA ANALYSIS**

Hand tracking data were preprocessed trial by trial with the Qualisys software and then further analyzed together with the eye position data in MATLAB. Initiation and termination of hand movement for each trial was determined based on a 5% criterion of peak velocity of that trial (right-hand grasping: thumb *M* = 68*.*8 mm/s, *SD* = 25*.*7, index *M* = 52*.*3 mm/s, *SD* = 9*.*7; left-hand grasping: thumb *M* = 64*.*7 mm/s, *SD* = 7*.*8, index *M* = 54*.*5 mm/s, *SD* = 3*.*5; movement start and end identification was further monitored on a trial-by-trial basis, and any inaccuracies were manually corrected). The MGA was defined as the largest distance between the index finger and thumb during our participants' reach-to-grasp movements. The data were visually inspected to identify and exclude invalid trials (20% were invalid trials for right-hand grasping, and 16% for left hand grasping). Exclusion criteria were: eye fixation errors after the onset of the white screen (e.g., not maintaining fixation throughout the trial by deviating more than 3.75 visual degrees away from fixation; see Le and Niemeier, 2013a), reaction times shorter than 50 ms, and incomplete or noisy hand trajectories due to artifacts.

For each individual participant, we then extracted seven dependent variables, mostly in line with our previous visual field parameters (Le and Niemeier, 2013a,b). The first four variables were measures of grasp movement metrics: To look at MGA scaling, we calculated slopes ([MGA of wide object width – MGA of narrow object width]/[wide object width – narrow object width]), which has traditionally been used to indicate grasp proficiency (see Smeets and Brenner, 1999 for a review). A slope of 0 indicates no scaling of the MGA to the object, whereas a slope of 1 indicates perfect scaling, and so higher slopes reflect greater proficiency in grasping, although the typical range is 0.7–1 (Smeets and Brenner, 1999). The second and third measure were the absolute size of MGA and MGA in proportion to the respective object width (proportional MGA = absolute MGA/object width; calculated for narrow and wide widths separately). Here, smaller values closer to the actual width of the respective object reflect greater grasp proficiency (Schlicht and Schrater, 2007; thus proportional MGA values closer to but larger than 1 are more ideal). We examined standard deviations of MGA, calculated for each participant separately, as a fourth measure of grasp metrics to allow for comparisons with our previous studies on bimanual grasping (Le and Niemeier, 2013a,b). In the latter studies we found reduced variability of MGA for the left visual field together with other signs of visual field dominance for that side and a matching righthemisphere dominance of bimanual grasping (Le et al., 2014). Three additional variables inspected the timing of grasp movements: Reaction time captured the time from the object becoming visible to the fingers starting to move, time of MGA (tMGA) measured the time from movement onset to MGA, and total movement time measured the time from movement onset till end of the movement at object contact.

# **RESULTS**

#### **EXPERIMENT 1: RIGHT-HAND GRASPING**

Grasping trajectories showed a MGA during the second half of the movement (∼66% of total movement time), which resembled the typical trajectories for unimanual grasping (**Figure 2**; see Jeannerod, 1984; Tresilian and Stelmach, 1997; Castiello, 2005).

#### *Visual field effects on the MGA of reach-to-grasp movements*

To see whether the metrics of grasping movements were influenced by visual field, we studied four measures related to MGA for the two visual fields separately: MGA scaling, absolute and proportional MGA size, and MGA variability (see Methods). For MGA scaling (**Figure 3A**), we obtained values of 0.67 and 0.80 for the left VF and right VF, respectively; with the slopes for left VF being slightly below the lower end of the normal

range (0.7–1.0, Smeets and Brenner, 1999). This difference was significant [*t*(27) = 3*.*69, *p* = 0*.*001, *d* = 1*.*42; 22 out of 28 participants showed the effect], indicating a right visual field advantage [although both slopes were larger than zero, *t*(27) = 24*.*14, *p <* 0*.*001; *t*(27) = 13*.*65, *p <* 0*.*001, respectively, indicating that both visual fields permitted functional grasps].

Because using slopes alone to capture grasp performance could overlook systematic errors in grasping, next we submitted absolute MGA values (**Figure 3B**) to a repeated-measures ANOVA with factors "visual field" and "object width." We observed a main effect of object width [*F*(1*,* 27) <sup>=</sup> <sup>260</sup>*.*02, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*83] and a main effect of visual field [*F*(1*,* 27) = 116*.*85, *p <* 0*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*05] such that MGA was larger in the left VF, consistent with reports that MGA increases as grasping becomes more difficult (Schlicht and Schrater, 2007). A significant visual field × object width interaction [*F*(1*,* 27) = 15*.*32, *p <* 0*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*01] reflected that for the narrow object width the MGA was larger in the left VF than the right VF [LVF – RVF = 2.38 mm; *t*(27) = −8*.*13, *p <* 0*.*001, *d* = 3*.*12; all participants showed the effect]. For the wide object width, MGA was similarly modulated by visual field, although somewhat less [LVF – RVF = 0.98 mm; *t*(27) = −5*.*98, *p <* 0*.*001, *d* = 2*.*30; 25 of 28 participants showed the effect]. The reduced visual field effect could be due to the restrictions of the hand span on the size of MGA for wider objects, thus, the interaction could reflect a ceiling effect. Consistent with this, we found that skewness values of MGA for individual participants were significantly more negative for the wide object size [*F*(1*,* 24) = 7*.*52, *p* = 0*.*01]. Nevertheless, it is interesting to note that we have made the same kind of observation of smaller visual field effects for a larger among a similarly sized set of object sizes for bimanual precision grasps with no apparent biomechanical constraints and ceiling effects (Le and Niemeier, 2013a; also see the MGA analysis for Experiment 2 of the current study).

To further ensure that our approach of inspecting absolute MGA values did not overlook any effects, we submitted proportional MGA (**Figure 3C**) values to a repeated-measures ANOVA with factors "visual field" (left, right), and "object width" (narrow 50 mm, wide 75 mm), and found a significant main effect of visual field [*F*(1*,* 27) <sup>=</sup> <sup>112</sup>*.*17, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*01] and its interaction with object width [*F*(1*,* 27) <sup>=</sup> <sup>20</sup>*.*83, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> 0*.*003], such that the proportional MGA values were closer to 1 when the object was in the right visual field, and especially so for the narrow object width (narrow: LVF – RVF = 0.05 mm; wide: LVF – RVF = 0.02 mm). Lastly, proportional MGAs were closer to 1 for the wider object width compared to the narrow object width [wide – narrow = −0*.*25 mm; "object width" factor: *<sup>F</sup>*(1*,* 27) <sup>=</sup> <sup>866</sup>*.*03, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*94].

As a fourth measure of grasp metrics adopted from our previous studies (Le and Niemeier, 2013a,b): we examined standard deviations of MGA calculated for each participant separately (**Figure 3D**). However, in contrast to our earlier work we found no main effect of visual field [*F*(1*,* 27) = 0*.*51, *p* = 0*.*48] or interactions with object size [*F*(1*,* 27) = 1*.*60, *p* = 0*.*22]. The narrow object width yielded more MGA variability than the wide object width [Narrow – Wide = 1.25 mm; *F*(1*,* 27) = 73*.*70, *p <* 0*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*44; 26 out of 28 participants showed this trend; cf. Ganel et al., 2008 and Heath et al., 2011]. In sum, three out of four grasp metric variables showed a preference for the right visual field.

# *Visual field effects on timing of reach-to-grasp movements*

Next, we inspected the temporal aspects of grasping for visual field differences. Reaction times (**Figure 4A**) were submitted to a repeated-measures ANOVA with factors "visual field" (left, right), "object width" (narrow, wide), and "digit" (thumb, index). We found no main or interaction effects involving visual field, object width, or digit (*F*'s ≤ 2.40, *p*'s ≥ 0.13). The tMGA revealed significantly delayed tMGAs for the wider object width compared to narrow [Wide – Narrow = 15.92 ms, *F*(1*,* 27) = 14*.*48, *p <* 0*.*001], as expected given typical grasp kinematics (see Smeets and Brenner, 1999). However, we did not observe significant main effects of visual field [*F*(1*,* 27) = 1*.*51, *p* = 0*.*23], although there was a non-significant trend for earlier MGA times in the right VF condition, especially for the narrow object width [LVF − RVF = 45.9 ms; **Figure 4B**; visual field x object width interaction: *F*(1*,* 27) = 2*.*81, *p* = 0*.*11]. Again, this interaction trend is consistent with previous reports of visual field effects modulated by object size (see Le and Niemeier, 2013a). Lastly, for total movement time, main effects of visual field and object size, along with all interaction effects, were not significant (*F*'s ≤ 3.67, *p* ≥ 0*.*07). We found a main effect of digit [*F*(1*,* 27) <sup>=</sup> <sup>50</sup>*.*05, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> 0*.*05] such that the thumb arrived at the object before the index finger (**Figure 4C**), perhaps due to the different biomechanical as well as task/goal constraints (e.g., Melmoth and Grant, 2012). The results here suggest that visual fields did not affect overall timing during grasping with the right hand.

# *Training effects*

To look for learning effects we calculated group averages based on the first and second half of grasping trials for each participant. Though this reduced the power of our data, trends in both halves of trials showed the same direction of visual field effects as the original analysis.

object width]/[wide object width – narrow object width]. **(B)** MGA size

#### **EXPERIMENT 2: LEFT-HAND GRASPING**

Similar to right-hand grasping, the left-hand grasping trajectories showed a MGA during the second half of the movement (∼68% of total movement time), which resembled the typical trajectories for unimanual grasping (**Figure 5**; see Jeannerod, 1984; Tresilian and Stelmach, 1997; Castiello, 2005).

#### *Visual field effects on the MGA of reach-to-grasp movements*

As for MGA scaling (**Figure 6A**), we obtained slopes of 0.60 and 0.34 for the left VF and right VF, respectively. This difference

absolute MGA/object width. **(D)** Standard deviation of MGA during grasping for narrow and wide object widths.

was significant [*t*(20) = −7*.*60, *p <* 0*.*001, *d* = 3*.*40; all participants showed the effect], indicating a left visual field advantage [although both slopes were larger than zero, *t*(20) = 11*.*48, *p <* 0*.*001; *t*(20) = 10*.*82, *p <* 0*.*001, respectively, indicating that both visual fields permitted functional grasps].

Next, absolute MGA values (**Figure 6B**) submitted to a repeated-measures ANOVA with factors "object width" and "visual field" revealed a main effect of object width [*F*(1*,* 20) = <sup>141</sup>*.*89, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*70] and a main effect of visual field [*F*(1*,*20) <sup>=</sup> <sup>36</sup>*.*17, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*08] such that MGA was larger in the right VF. A significant visual field x object width interaction [*F*(1*,* 20) <sup>=</sup> <sup>57</sup>*.*82, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*06] reflected that for the narrow object width the MGA was larger in the right VF than the left VF [RVF – LVF = 2.96 mm; *t*(20) = 7*.*18, *p <* 0*.*001, *d* = 3*.*21; all participants showed the effect]. For the wide object width, MGA was not significantly modulated by visual field [RVF – LVF = 0.27 mm; *t*(20) = 1*.*42, *p* = 0*.*17]. To test for possible

**movements. (A)** Average reaction time for grasping for narrow and wide object width. Thb = thumb, Ind = index. **(B)** Time of maximum grip aperture (MGA) for grasping for narrow and wide object width. **(C)** Total movement time for grasping for narrow and wide object width.

ceiling effects, an additional analysis of skewness of the MGA data was conducted but found no evidence for more negative skewness for the wider object width [*F*(1*,* 19) = 0*.*778, *p* = 0*.*389].

An ANOVA conducted for proportional MGAs (**Figure 6C**) revealed a significant main effect of visual field [*F*(1*,* 20) = 38*.*48, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*01] and an interaction with object width [*F*(1*,* 20) <sup>=</sup> <sup>58</sup>*.*67, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*007], such that the proportional MGA values were closer to 1 when the object was in the left visual field, and especially so for the narrow object width (narrow: LVF – RVF = −0*.*05 mm; wide: LVF – RVF = −0*.*005 mm).

Lastly, proportional MGAs were closer to 1 for the wider object width compared to the narrow object width [wide − narrow = −0*.*29 mm; *<sup>F</sup>*(1*,* 20) <sup>=</sup> <sup>1270</sup>*.*80, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*95].

Lastly, we examined the standard deviations of MGA (**Figure 6D**). Here we found significant main effects of visual field [*F*(1*,* 20) <sup>=</sup> <sup>85</sup>*.*58, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*25] and its interaction with object size [*F*(1*,* 20) <sup>=</sup> <sup>200</sup>*.*28, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*18]. That is, the MGA was less variable in the right VF than in the left VF, especially for the narrow object width (narrow: RVF – LVF = −2*.*55 mm; wide: RVF – LVF = −0*.*21 mm; all participants showed this effect). Moreover, the narrow object width yielded more MGA variability than the wide object width [Narrow – Wide <sup>=</sup> 1.70 mm; *<sup>F</sup>*(1*,* 20) <sup>=</sup> <sup>68</sup>*.*15, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> 0*.*38]. These results could suggest greater proficiency in the left compared to the right visual field (in contrast to the three previous measures of left-hand grasping), together with a lefthemisphere dominance for the underlying neural processes. However, inconsistent with this interpretation, right-hand grasping did not produce a comparable right visual field advantage. Given this, we will re-analyze our different dependent measures in factor analyses near the end of the Results. In sum, three out of four grasp metric variables showed a preference for the left visual field, and one showed a preference for the right visual field.

#### *Visual field effects on timing of reach-to-grasp movements*

Next, we inspected reaction times (**Figure 7A**), which were submitted to a repeated-measures ANOVA with factors "visual field" (left, right), "object width" (narrow, wide), and "digit" (thumb, index). We found no main or interaction effects involving visual field, object width, or digit (*F*'s ≤ 3.45, *p*'s ≥ 0.08). The tMGA showed no significant effects either (**Figure 7B**; *F*'s ≤ 1.40, *p*'s ≥ 0.25). Lastly, for total movement time, main effects of visual field and object size, along with most interaction effects, were not significant (*F*'s ≤ 1.09, *p* ≥ 0*.*31). However, we did find a main effect

of digit [*F*(1*,* 20) <sup>=</sup> <sup>54</sup>*.*07, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*16] and its interaction with visual field [*F*(1*,* 20) <sup>=</sup> <sup>7</sup>*.*41, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>*.*01, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*003] such that the thumb showed faster movement times in the left VF whereas the index finger showed faster movement times in the right VF (**Figure 7C**). In general, the results here suggest that visual fields did not affect overall timing during grasping with one hand, although the index and thumb showed different visual field advantages.

## *Training effects*

To look for learning effects, we calculated group averages based on the first and second half of grasping trials for each participant. Trends showed the same direction of visual field effects the first half of trials compared to the overall analyses above. However, in the second half of trials, visual field differences were greatly reduced, in particular for the slope and MGA (absolute and proportional).

# *Influences of left-hand proficiency*

To look for possible influences of left-hand proficiency on our results, and importantly the unexpected visual field effect for MGA variability, we recalculated group averages for all the dependent variables based on participants who demonstrated good left-hand proficiency based on more accurate scaling of the grip to object size (i.e., minimum slope = 0.5). Although this reduced the reliability of our data, trends showed the same direction of effects in most cases, suggesting that the visual field effects on left-hand grasping cannot be explained by some form (or lack) of proficiency with using the left hand.

#### *Factorial structure of grasp performance*

To better understand the dimensions of grasp performance governing our participants' reach-to-grasp performance, we submitted all seven dependent variables for both the left and right VF (averaged across object width and/or digit where appropriate) to

factor analyses, one for Experiment 1 and one for Experiment 2. Both factor analyses extracted 4 factors (eigenvalues *>* 0.5; varimax rotation, extraction method = principal component analysis). **Table 1** provides a summary of the solution found for Experiment 1 with the loads of the different variables on the four factors (loads higher than 0.3 are bolded; loads smaller than 0.3 can be considered insignificant). The result suggests one factor of timing and three factors of grasp metrics (one primarily for MGA, one for standard deviation of MGA, and one primarily for slope). **Table 2** provides a summary of the solution found for Experiment 2. The result confirms the separation of temporal and **Table 1 | Right-hand grasping factor analysis rotated component matrix.**


*Loadings greater than 0.3 are highlighted in bold.*

metric measures. However, there now are two factors of timing, one rather associated with reaction time (Factor 2) and the other more associated with movement time (Factor 3). Two additional factors explain variability of metric aspects of grasping: Factor 1 captures MGA (absolute and proportional) and slope, and Factor 4 captures standard deviation of MGA. Although more research is warranted, both factor analyses agree that the standard deviation of MGA loads onto a separate factor. Together with the non-intuitive results for left-hand grasping, this arguably indicates that standard deviation of MGA reveals processes that are separate from reach-to-grasp movement performance.

## *Omnibus analysis of visual field preferences*

As the final step of our data analysis, we compared visual field preferences in the two experiments. A first approach used ANOVAs with mixed design (between-subjects factor: right-hand users vs. left-hand users; within-subject factor: contralateral vs. ipsilateral visual field; data were averaged across object width, and/or digit). To sort out any possible effects of handedness level between the two groups of participants, we conducted all ANOVAs with and without the participants' handedness laterality score as a covariate, but this had no influence on the pattern of significant *F*-tests. Also, to keep numbers of tests small, these analyses focused on three dependent variables that had produced significant visual field effects for each experiment separately: slope of MGA, absolute MGA and proportional MGA. Standard deviation of MGA was not considered given its apparently idiosyncratic underlying mechanisms. A complete account of the

narrow and wide object width.



*Loadings greater than 0.3 are highlighted in bold.*

ANOVA results including handedness as a covariate is provided in **Table 3**. In summary, we observed a significant group-by-visual field interaction for slope (*<sup>p</sup>* <sup>=</sup> <sup>0</sup>*.*026, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0*.*103) indicating a more pronounced visual field difference for left hand grasping in terms of grasp-relevant object-size processing. Closer inspection of the right-hand data showed equal variability for slopes in the left and right VF. This rules out the possibility that the smaller visual field effect, compared to left-hand data, was caused by a ceiling effect. In addition, all three measures revealed a main effect of hand use, with poorer performance for left (non-dominant) hand use (slope: Left-Hand – Right-Hand = −0*.*5 mm; proportional MGA: Left-Hand – Right-Hand = 0.05; respectively; *p*'s *<* 0.041; absolute MGA: Left-Hand – Right-Hand = 2.59 mm, although this could be affected by placement of the markers). Moreover, and as expected, all three measures produced general visual field effects such that there was a visual field advantage for the side ipsilateral to the respective hand (*p*'s *<* 0.008).

In the second approach, we compared the overall size of the visual-field effect for left- vs. right-hand grasping by conducting a bootstrapping procedure. To do this, for each grasp-type, we calculated an averaged laterality index ([contralateral VF – ipsilateral VF]/[[contralateral VF + ipsilateral VF]/2]) across the three dependent variables that had produced significant visual field effects (slope, absolute and proportional MGA), such that the higher the index value, the greater the general visual-field effect. Then, we used random selection with replacement to recreate the groups of participants for the two experiments (*n* = 28 for Experiment 1, *n* = 21 for Experiment 2) and calculated

#### **Table 3 | Mixed-design ANOVA results.**


*Within-subjects factor: contralateral vs. ipsilateral visual field (F1); Betweensubjects factor: right-hand users vs. left-hand users (F2). \*p < 0.05, \*\*p < 0.01.*

group means of the laterality indices. Next we repeated this procedure 10,000 times to then determine percentiles of 50% for the medians as well as 2.5 and 97.5% for the confidence intervals (**Figure 8**). As shown, there we found that left-hand grasping had significantly greater overall visual-field effects than right-hand grasping. Thus, the results here confirm that left-hand grasping has a more pronounced preference for the ipsilateral visual field compared to right-hand grasping.

## **DISCUSSION**

In the present study, we tested whether visual input of unimanual grasping varies as a function of visual field. Equivalent lateralizations are well known for the control processes of motor output for grasps with the right or left hand. That is, grasping with one or the other hand is controlled by the contralateral hemisphere (Rice et al., 2007) with a relative dominance of the left hemisphere for certain aspects of grasping (Davare et al., 2007). Consistent with our predictions, we found similar lateralizations for the visual input: for right-hand grasping, objects appearing in the right visual field (i.e., fixation to the left) were grasped with greater proficiency than when grasped in the left visual field (i.e., fixation to the right). In contrast, for left-hand grasping, objects in the left visual field were grasped with greater proficiency than when grasped in the right visual field. This suggests that sensorimotor demands for using the left- vs. right-hand trigger grasp circuits in the contralateral hemisphere, which is more sensitive to the respective contralateral visual field. Moreover, we found that visual field differences for left-hand grasping were more pronounced compared to the differences observed for right-hand grasping. In the following we will discuss contralateral specialization and differences between left- and right-hand grasping separately.

# **EVIDENCE FOR VISUAL FIELD LATERALIZATION**

The finding of lateralized specialization is consistent with previous research. For bimanual grasping of a single object we have found visual field differences with a preference for the left visual field (Le and Niemeier, 2013a,b), which is the field contralateral to the hemisphere that controls bimanual grasps (Le et al., 2014). For observations of unimanual grasp actions, Shmuelof and Zohary (2006) found that brain activity in the right hemisphere (but not the left hemisphere, see below) varied as a function of visual field. Given the employed visual stimuli there is a good possibility that visual input for actual unimanual grasping might be treated similarly. Nevertheless, to our knowledge the present data are the first to directly demonstrate that visual object analysis used for overt unimanual grasping shows a preference for visual information from the visual field on the same side as the grasping hand.

We found this visual field advantage for our kinematic measures of MGA and MGA scaling. These measures have been demonstrated to reflect the proficiency of the grasp component for reach-to-grasp movements (Jeannerod, 1981, 1984; Tresilian and Stelmach, 1997). For example, for right-hand grasping in the right visual field our participants showed scaling of the MGA well within the normal ranges of proficient grasping (Smeets and Brenner, 1999). This said, direct comparability is limited because our participants grasped objects without directly looking at them, which should make grasping more challenging. Still, our participants grasped only two object widths so that they had a greater chance to memorize proper MGA sizes compared to grasp paradigms with multiple different object sizes. Nevertheless, right-hand MGA scaling in the left visual field was significantly reduced to a level at the lower range of functional grasps or below. For left-hand grasping we found the reversed pattern with worse scaling in the right than the left visual field (on overall lower levels of proficiency, see the section on left- vs. right-hand grasping).

In addition, we found that MGAs in the visual field ipsilateral to the grasping hand were smaller than in the contralateral visual field. This is consistent with observations that MGA increases with visual uncertainty, arguably in an attempt to "err" in a direction where the fingers are less likely to collide with the object (e.g., Schlicht and Schrater, 2007). Interestingly, the difference in MGA size was more apparent for the narrow object width as opposed to the wide object width. It is possible that the difference reflects a natural biomechanical constraint of opening up the distance between thumb and index finger beyond a certain point. However, this is unlikely given that we have observed a similar object size effect for bimanual grasping of similarly sized objects, so when there is no similar biomechanical constraint (Le and Niemeier, 2013a). Because larger MGAs occur at later stages of the reach-to-grasp movement (Jakobson and Goodale, 1991), together these data could be reconciled such that visual field effects might be more apparent for MGAs attained at earlier times of the movements. However, further research is required to confirm that smaller objects with earlier MGAs yield more pronounced visual field effects.

The present finding that visual field differences are reduced for wider objects (which have later MGAs) rules out the possibility that the visual field effects were caused by hand-distractions on one side of the object (e.g., right-hand grasps are more distracting for right fixations). That is, if the hand was a potential distraction, we should see stronger visual field effects for wider objects because the MGA would have been wider and closer to the object; however, we did not observe this. Moreover, it is unlikely that the hand trajectories would have produced distractions by passing the line of sight because hand movements started from a position below the object and fixation points. Even so, visual feedback of the hand generally does not have a facilitative effect (and hence distracting effect) on grip aperture during reach-to-grasp movements (Connolly and Goodale, 1999).

Unlike MGA and MGA scaling, MGA variability exhibited visual field differences that eluded any straight forward interpretation. For bimanual grasping, this measure was smaller in the dominant left visual field (Le and Niemeier, 2013a,b). Thus, smaller MGA variabilities seemed to serve as measures of greater grasp precision and, therefore, grasp proficiency. But, for unimanual grasping we found opposite trends: grasping with the left hand produced less MGA variability than grasping with the dominant right hand, and left-hand grasping in the (otherwise preferred) left visual field increased MGA variability. Also, the factorial structure of our dependent variables always had MGA variability load on its own factor, separate from other grasp metrics. So in sum, MGA variability is a measure that might be rather disconnected from other grasp measures and require further investigations of the underlying mechanisms.

More research will also be necessary to clarify why visual field presentation modulated only the metrics of the grasps, and not the timing or stability. One explanation could be task difficulty: Unimanually grasping an object that has a suitable width (i.e., as in the present study) may be easy and natural, thus allowing for an equal advantage for both visual fields in grasp timing and stability. In contrast, unimanual grasping an object that may be too wide or narrow might require increased inter-digit coordination in both metrics and timing, due to the increased level of difficulty in keeping the grasp stable, and thereby increasing visual field effects. Consistent with this explanation, we found visual field effects for the timing as well as metrics in our previous visual field study on bimanual grasping of small objects (Le and Niemeier, 2013a), which are presumably more difficult compared to bimanual grasping of larger objects. Indeed, when we examined bimanual grasping of large objects, we found visual field effects for the metrics only (Le and Niemeier, 2013b).

## **DIFFERENCES BETWEEN LEFT- AND RIGHT-HAND GRASPING**

Left- and right-hand grasping differed in several ways. As expected, non-dominant left-hand grasping showed signs of reduced proficiency: MGA was larger for left-hand grasping and MGA scaling showed flatter slopes than right-hand grasping. Our skewness data suggested that no MGA ceiling effects during left hand grasping and thus the comparatively flat slopes (Smeets and Brenner, 1999) were unlikely caused by ceiling effects during left-hand grasping, such as, limits of hand span that would limit MGA sizes of the wider object width. If at all, MGA showed limitations during right-hand grasping. Nevertheless, its slopes were steeper—and thereby closer to ideal—than usual (Smeets and Brenner, 1999). At any rate, limitations imposed by hand span could not explain visual field specific differences. It is possible however that, as one contributing factor, grasp performance in the two experiments differed because different participants were tested. Importantly, people in the second experiment showed higher handedness scores (Oldfield, 1971). Although the handedness inventory is a relative measure it could reflect that Experiment 2 participants were systematically less proficient with their left hand. However, if at all, the handedness difference seems to have played a small role because controlling for the influence of handedness did not alter the results in the omnibus ANOVAs.

Interestingly, left- and right-hand grasping differed in the strength of the visual field effects; specifically, left-hand grasping showed substantial differences in MGA scaling in the left and right visual field but for right-hand grasping the differences were less pronounced so that the omnibus ANOVA flagged a significant group-by-visual field interaction. The interaction was strong enough to generate a greater bootstrapped lateralization score for left- than right-hand grasping. Once again, it is possible that unspecific differences between participant groups contributed to the interaction. However, these differences might have contributed relatively little because handedness as a covariate was unsuccessful in explaining the group-by-visual field interaction. Moreover, only MGA scaling showed an interaction but not the other MGA measures, whereas an unspecific group differences should have had a generalized influence on interactions for multiple measures.

We speculate that the difference in visual field effects could reflect two possible causes. First, people could have scaled righthand grasps better and less differently for the visual fields because right-hand grasping has more privileged visuomotor coupling or is better capable of forming proprioceptive or procedural memories of grasps in the course of the experiment. Especially with two object sizes only, visual field differences could have leveled out for right-hand grasping, although here we found little evidence for training effects in Experiment 1. Still, left-hand grasping might have relied more on online control (e.g., Haaland and Harrington, 1989; Haaland et al., 2004; Gonzalez et al., 2008; Tremblay et al., 2013). If so, it still remains unclear why MGA scaling showed a group-by-visual field interaction but not the MGA measures. Moreover, we found that the visual field differences leveled out by the second-half of the trials for left-hand grasping instead. Thus, a second perhaps more plausible cause of the interaction is that the left hemisphere is better equipped to perform visual object analysis across the two visual fields. For example, left-hemisphere areas performing visual object analysis for grasping, likely including aIPS (Rizzolatti and Luppino, 2001), might contain neurons with visual receptive fields that expand further into the left visual field compared to equivalent areas in the right hemisphere. This interpretation agrees well with Shmuelof and Zohary's (2006) observation that activity in right-brain grasp areas varied as a function of visual field but not activity in the left hemisphere. Consistent with this idea, that sensorimotor control in the left hemisphere might have a greater need to access visual information in both visual fields to permit greater flexibility in grasp actions of the dominant right hand across the entire visual field.

In conclusion, here we investigated visual field effects on unimanual grasping. We found a visual field advantage for the side ipsilateral to the grasping hand indicating a contralateral organization of visual object analysis. In addition, we observed differences in the degree of lateralization that could reflect a relative dominance of the left hemisphere for the visual analysis component of prehension. Our data contribute to the mechanistic understanding of the visual processes that give rise to the sensorimotor control of grasping. In addition, our findings might be of clinical significance as they could help refine rehabilitation programs for patients with motor deficits after cortical damage such as stroke (e.g., Metrot et al., 2012).

# **ACKNOWLEDGMENT**

This work was supported by an NSERC grant to Dr. Matthias Niemeier. Ada Le held NSERC CGS D and OGS scholarships for the duration of this project. We would also like to thank Moein Bayat-Mokhtari, Medhat Chowdhury, and Marija Zivcevska for their help with data collection, as well as the numerous undergraduate volunteers who assisted with data processing.

# **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 15 April 2014; accepted: 15 September 2014; published online: 01 October 2014.*

*Citation: Le A and Niemeier M (2014) Visual field preferences of object analysis for grasping with one hand. Front. Hum. Neurosci. 8:782. doi: 10.3389/fnhum. 2014.00782*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Le and Niemeier. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Real-world objects are more memorable than photographs of objects

# *Jacqueline C. Snow\*, Rafal M. Skiba , Taylor L. Coleman and Marian E. Berryhill*

*Cognitive and Brain Sciences Group, Department of Psychology, University of Nevada, Reno, NV, USA*

#### *Edited by:*

*Irene Sperandio, University of East Anglia, UK*

*Reviewed by:*

*Talia Konkle, Harvard University, USA Matteo Valsecchi, Justus-Liebig Universität Giessen, Germany*

#### *\*Correspondence:*

*Jacqueline C. Snow, Program in Cognitive and Brain Sciences, Department of Psychology/296, 1664 N. Virginia Street, Reno, NV 89557, USA e-mail: jacqueline.c. snow@gmail.com*

Research studies in psychology typically use two-dimensional (2D) images of objects as proxies for real-world three-dimensional (3D) stimuli. There are, however, a number of important differences between real objects and images that could influence cognition and behavior. Although human memory has been studied extensively, only a handful of studies have used real objects in the context of memory and virtually none have directly compared memory for real objects vs. their 2D counterparts. Here we examined whether or not episodic memory is influenced by the format in which objects are displayed. We conducted two experiments asking participants to freely recall, and to recognize, a set of 44 common household objects. Critically, the exemplars were displayed to observers in one of three viewing conditions: real-world objects, colored photographs, or black and white line drawings. Stimuli were closely matched across conditions for size, orientation, and illumination. Surprisingly, recall and recognition performance was significantly better for real objects compared to colored photographs or line drawings (for which memory performance was equivalent). We replicated this pattern in a second experiment comparing memory for real objects vs. color photos, when the stimuli were matched for viewing angle across conditions. Again, recall and recognition performance was significantly better for the real objects than matched color photos of the same items. Taken together, our data suggest that real objects are more memorable than pictorial stimuli. Our results highlight the importance of studying real-world object cognition and raise the potential for applied use in developing effective strategies for education, marketing, and further research on object-related cognition.

**Keywords: real-world, real objects, pictures, memory, recall, recognition memory**

# **INTRODUCTION**

Our current scientific knowledge in areas such as human visual perception, attention, and memory, is founded almost exclusively on experiments that rely upon 2D image presentations. However, the human visuomotor system has largely evolved to perceive and interact with real objects and environments, not images (Gibson, 1979; Norman, 2002). Despite many fundamental differences between real objects and images, there has been very little investigation of whether real objects have a unique influence on cognition and action compared with pictorial displays. In the domain of human memory, studies have used real world objects (e.g., Dirks and Neisser, 1977; Mandler et al., 1977; Pezdek et al., 1986; Droll and Eckstein, 2009), but to our knowledge none have specifically examined whether memory performance is superior for real objects vs. matched image displays. In other words, the underlying and unexplored assumption is that representations of real objects are remembered equivalently to real objects. However, real objects may have a memory advantage that is important to consider, both for empirical reasons, and because of the potential benefits in other domains—such as education and marketing.

Real objects differ from pictures in a number of important respects, several of which could influence memory. First, real objects (when viewed with two eyes) possess additional cues to 3D shape than 2D pictures. When we look at the world with two eyes, each receives information about objects from a slightly different horizontal viewpoint—the geometrical discrepancy between which is known as binocular disparity (Harris and Wilcox, 2009; Blake and Wilson, 2011). The brain is able to resolve the discrepancy in these two images to produce a unitary sense of depth (Blake and Wilson, 2011). Conversely, when we view a static 2D picture of an object, no additional information about the depth structure of the object is available, and consequently we experience the stimulus as being "flat." Further, 2D images present the visual system with inherent cue conflicts; monocular cues to 3D shape, such as surface texture, specular highlights, and linear perspective, indicate that the stimulus has depth, whereas binocular cues indicate that the stimulus is flat (Vishwanath and Kowler, 2004). Stimuli that lack stereo cues can profoundly disrupt object recognition in brain damaged patients. For example, visual agnosia patients are better at recognizing real objects than 2D pictures (Riddoch and Humphreys, 1987; Young and Ellis, 1989; Servos et al., 1993; Humphrey et al., 1994; Chainay and Humphreys, 2001; Hiraoka et al., 2009)—an effect that has been argued to be due to the additional depth cues inherent to real exemplars (Servos et al., 1993; Chainay and Humphreys, 2001). It is possible therefore, that additional information about the geometrical structure of real objects could facilitate memory compared with 2D image displays.

Second, unlike pictures, real objects afford action such as grasping and manipulation. In terms of neural responsivity, viewing real objects and images of objects activates similar networks, particularly the lateral occipital complex along the lateral and ventral convexity of occipito-temporal cortex (Snow et al., 2011). However, because real objects afford action, they can have a unique effect on neural responses. For example, when viewing object images, stimulus repetition leads to a characteristic reduction in fMRI responses—an effect known as fMR-adaptation or repetition suppression (RS) (Grill-Spector and Malach, 2001; Malach, 2012). Yet, recent research has highlighted important differences in RS as a function of the type of stimulus presented; RS for real objects is weak, if not absent, when observers view *real* objects compared to matched 2D photographs of the same items (Snow et al., 2011). Further, some brain areas, such as superior parieto-occipital cortex, respond differently depending on whether or not a graspable object is within reach of the dominant hand, regardless of whether or not a grasp is planned or executed (Gallivan et al., 2009, 2011). It is reasonable to suspect therefore that real-world, graspable objects are stored, represented, and/or processed differently than images of objects (Snow et al., 2011). Moreover, the potential for a motor interaction with real objects could strengthen or enhance the associated memory trace by automatically enhancing depth of processing at encoding (Craik and Tulving, 1975).

Third, real objects have an actual or veridical size, distance, and location relative to the observer, whereas images do not, and these cues to object identity could facilitate memory. Images although often described as having a "real world size"—have only an *expected* size based on our experience with other similar exemplars in the natural environment (Konkle and Oliva, 2011, 2012b; Brady et al., 2013). As a consequence, when viewing images there is often a striking discrepancy between retinal size and real-world size, relative to the distance of the image from the observer. For some types of displays, such as scenes that possess background contextual information, we can make inferences about the relative size of objects depicted in the image (i.e., "the cow is smaller than the tree, but larger than the sheep") yet the stimuli lack an *actual* size that would be relevant for motor planning. In most behavioral and neuroimaging studies, object stimuli are presented in isolation without background context making real world size even less apparent (i.e., is that a toy-sized object or is it real-life sized?) (e.g., Konen and Kastner, 2008). When background information *is* provided, retinotopic regions in the dorsal stream do track perceived distance (Berryhill and Olson, 2009). Thus, knowing the size, distance and location of a stimulus has consequences for the way in which it is perceived and this shapes future neural processing for cognition, action, and memory.

Given the fundamental differences between real objects and image displays outlined above, we wondered whether or not observers would show enhanced memory for everyday objects displayed as real exemplars vs. pictures. In the current study, we examined episodic memory performance by testing free recall and recognition for common household objects encoded under different viewing conditions. In Experiment 1 healthy college-aged students studied objects that were either viewed in the form of real world exemplars, high-resolution color photographs, or black and white line drawings. The line-drawing condition was included to determine the extent to which color and monocular cues to 3D shape (such as shading, and surface texture—all of which were present in the color photograph condition) bolster memory performance. Importantly, the stimuli in our study were closely matched for size, illumination, and orientation across the different viewing conditions. In Experiment 2, we controlled for the viewing angle at which the real objects and matched colored photographs were presented. Stimuli in all experiments were presented within arm's reach and viewed in their real world size. We predicted that if there were a real object benefit it would be reflected in significantly better memory performance. We included two measures of episodic memory, free recall and recognition, to assess more comprehensively the nature of any performance differences.

# **MATERIALS AND METHODS PARTICIPANTS**

One seventy two second-year psychology students at the University of Nevada, Reno participated in the studies, in exchange for course extra-credit (80 subjects Experiment 1; 92 subjects Experiment 2). All subjects gave informed assent or consent and the experimental protocols were approved by the University of Nevada, Reno Social, Behavioral, and Educational Institutional Review Board. There were no potential conflicts of interest as there were no commercial or financial benefits for any party.

## **MATERIALS**

The stimuli in each Experiment consisted of 44 common household objects and high-resolution photographs of the same items (**Figure 1**). The photographs were reproduced in color, and as black and white line-drawings. In Experiment 1 we compared memory performance for objects in three different Viewing Conditions: real objects, color photographs, and line-drawings. In Experiment 2 we compared memory for real objects vs. color photographs. Photograph (and line drawing) stimuli were matched to the real objects in terms of size, and orientation using the methods described below. Line drawings of each stimulus were created using Adobe Photoshop to remove all color and most surface texture cues by isolating the object in the image, using the Sketch-Photocopy filter, and raising contrast values in the image. Stimuli in each Viewing Condition were presented to observers within a custom display box; the boxes were constructed from black foamcore, and a gray curtain (that covered the entire display) was attached to the top of the box. Each real object was attached to a removable black foam-core shelf that could be quickly inserted into position within the display box. Each shelf held two objects, and each box held two shelves (upper/lower), yielding a total of four stimuli per trial. We used a Canon Rebel T2i DSLR camera with constant F-stop and shutter speed to photograph the real objects, separately for each shelf, thereby matching for stimulus orientation between the real object and photograph conditions. Image size was adjusted using Adobe Photoshop, and the resulting photograph stimuli were printed to match the real objects in

size. In Experiment 1, the real object shelves were positioned at a 45◦ angle within the display box, and photographs of the shelves were taken at the same display angle (**Figure 1A**). In Experiment 2, the object stimuli were attached to shelves that were positioned in a *vertical* orientation, to match the real object and photograph stimuli for viewing angle (**Figure 1B**). Photograph stimuli were printed on HP Satin Q8923 paper and attached to shelves of identical size to the real object displays using double-sided tape. The timing of events in both experiments was controlled using Matlab (Mathworks, USA) and Psychtoolbox software packages.

# **PROCEDURE**

We used a between-subjects design to compare memory performance in each Viewing Condition. The procedure was identical in Experiments 1 and 2 (**Figure 1C**). Subjects were assigned randomly to one Viewing Condition (Experiment 1: real objects: *n* = 26, color photographs: *n* = 27, line-drawings: *n* = 26; Experiment 2: real objects: *n* = 46, color photographs: *n* = 46), and each group was tested in a separate room. Subjects were not informed of the Condition to which they had been assigned, or to the fact that there were different Viewing Conditions in the experiment. Identical display boxes were situated in each testing room, and two experimenters were present in each room to run the study. During the Study Phase, participants were instructed to stand facing the display box at a distance of ∼56 cm, with the stimuli within arm's reach. A 50 ms auditory tone (1000 Hz) signaled trial onset, at which time the experimenters lifted the curtain to reveal the stimuli. The shelves were displayed for 5 s, after which a second 50 ms auditory tone (500 Hz) signaled subjects to close their eyes during the 40 s ITI. At the onset of the second tone, the experimenters dropped the curtain, turned the display box around (facing away from the subjects), and prepared the display box for the upcoming trial. White noise was played throughout the ITI to mask extraneous noise during stimulus changeovers. Stimuli were presented in the same position, shelf, and in the same sequential order, in each Viewing Condition. Eleven trials were presented, for a total study set of forty-four objects.

In the subsequent Test Phase, participants were instructed to remember as many items as they could from the Study Phase. To control for recency effects (Bjork and Whitten, 1974) participants were first given a *semantic memory task* in which they were given 1 min to write down as many US states as they could. Next, in the *free-recall task*, participants were given 5 min to write the names of all objects they could remember from the Study Phase. All participants had finished the recall task to the best of their ability before the end of the 5-min time limit. Finally, subjects were given 10 min to complete a *recognition task* in which they were given a printed list of 88 object names, 44 of which were objects presented in the Study Phase, and the remaining 44 were distractor objects. The subjects' task was to judge whether or not each item had been presented in the Study Phase (True/False).

Participant responses were scored as either correct or incorrect (score /44 for the free recall task, and /88 for the recognition task). The mean number of falsely recalled items was also calculated for both experiments. Importantly, to control for any difficulty observers may have had in recognizing the stimuli (particularly the line-drawing condition), we conducted an item analysis. The number of times each stimulus was correctly recalled or identified was calculated separately for each Viewing Condition and Task. Next, mean recall and recognition performance was calculated across all items in the set for each Viewing Condition and Task, and objects for which memory performance fell below 3 standard deviations from the mean were eliminated from further analyses. Using this method, no items were removed from the recall task in either Experiment. For recognition, in Experiment 1 one distractor item ("Wrench") was removed due to the high number of false positive responses, and in Experiment 2 two study items were removed from further analysis ("Trowel" and "Hat") due to the high number of misses—possibly due to our subjects' vernacular for these terms, given that their free recall for the same items (using terms such as "Garden Shovel" and "Beanie") was relatively high (see **Tables 1, 2**). Recognition performance in each Experiment was analyzed according to percent (%) correct, and using a signal detection (SD) analysis to disentangle sensitivity to the stimuli from possible effects of response bias (Green and Swets, 1974).

$$\mathbf{d}' = \mathbf{z}(\mathbf{H}) - \mathbf{z}(\mathbf{F}) \tag{1}$$

For each observer, we calculated sensitivity to the study material (d in Equation 1), where Hits (H in Equation 1) = the number of items that *were present* in the study set, and which a subject *correctly identified* as being present; False Alarms (F in Equation 1) = the number of items that were *not present* in the study set, but which a subject *incorrectly identified* as being present; and z (in Equation 1) = z-transform. Mean (SD) d was calculated across all observers in each Viewing Condition, and compared using ANOVA and follow-up pairwise comparisons where appropriate. Finally, we conducted an Item × Viewing Condition analysis using mixed model ANOVA to examine whether the pattern of recall was similar across items in each viewing condition, with a view to elucidating whether or not the advantage of real object displays on memory performance was related to the types of items we used in our study set.

#### **RESULTS**

#### **EXPERIMENT 1**

In Experiment 1 we compared memory performance for stimuli presented in one of three Viewing Conditions: real objects, color photographs, and line-drawings. Recall responses for one subject were absent from the color photograph condition, and so the data were analyzed with 26 subjects in the line drawing and real object groups, and 27 for color photographs. Memory performance in each Viewing Condition was compared using one-way betweensubjects Analysis of Variance (ANOVA), followed by planned comparisons between each pair of means (Tukey's honestly significant difference; HSD). The Greenhouse-Geisser correction for violations of sphericity was applied where appropriate for within-subjects analyses.

In the free recall task, we observed a significant difference in memory performance for items in each Viewing Condition [*F*(2*,* 76) <sup>=</sup> <sup>11</sup>*.*277, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*229]. Observers' ability to **Table 1 | Recall data for each stimulus item in Experiment 1.**


*Percent (%) correct recall and recall quartile are displayed separately for each Item and Viewing Condition. Data are ranked according to % correct recall for real object displays. Note that items with a higher quartile ranking (i.e., 4th) are recalled more frequently than those in lower quartiles (i.e., 1st).*



*Percent (%) correct recall and recall quartile are displayed separately for each item and Viewing Condition. Data are ranked according to % correct recall for real object displays.*

recall real objects (Mean = 49.04%, *SD* = 13*.*19%) was significantly better than for color photograph (Mean = 36.62%, *SD* = 11*.*26%) or line-drawing displays (Mean = 34.27%, *SD* = 11*.*71%); *p* = 0*.*001 and *p <* 0*.*001, respectively; **Figure 2A**). Interestingly, recall performance for color photographs was not significantly different from that of line-drawings (*p* = 0*.*759), suggesting that additional monocular shape and color cues were not sufficient to facilitate object memory.

Next, we examined the number of *falsely* recalled items—items that participants listed in the recall task as being part of the study set, but were not in fact present (**Figure 2B**). Interestingly, there was also a significant difference in the number of falsely recalled items across the Viewing Conditions [*F*(2*,* 76) = 9*.*42, p *<* 0.001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*199]: participants who viewed real objects made *fewer* false recalls (Mean = 0.93, *SD* = 1*.*32) than those who viewed color photographs (Mean = 2.11, *SD* = 1*.*58; p = 0.02, Tukey's HSD) or line drawings (Mean = 2.81, *SD* = 1*.*81; p *<* 0.001, Tukey's HSD). There was no difference in mean number of falsely recalled items in the two pictorial conditions (*p* = 0*.*25, Tukey's HSD). These analyses confirm that observers' superior recall performance of real objects was not simply attributable to producing longer lists of items (thereby inflating the probability of correctly identifying items from the study set due to chance guessing), but demonstrate rather that their knowledge of the study material was also more specific than observers who viewed 2D pictures.

For the recognition task, we also observed a significant difference in % items correctly recalled across Viewing Conditions [*F*(2*,*77) <sup>=</sup> <sup>10</sup>*.*<sup>359</sup> *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*212; **Figure 2C**]. Recognition performance was superior for subjects who viewed real objects (Mean = 87.67%, *SD* = 9*.*04%), vs. colored photograph (Mean = 81.78, *SD* = 6*.*53; *p* = 0*.*025, Tukey's HSD) or line-drawing displays (Mean = 77.45, *SD* = 7*.*74; *p <* 0*.*001, Tukey's HSD). There was no significant difference in recognition performance for the line-drawing vs. color photograph conditions (*p* = 0*.*131). To exclude the possibility that differences in recall performance could be explained simply by response biases, we examined the recall data using a Signal Detection (SD) analysis (**Figure 2D**; see Methods). There was a significant difference in sensitivity to the study material across the different Viewing Conditions [*F*(2*,* 77) <sup>=</sup> <sup>14</sup>*.*96, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*28]. Observers who viewed real objects in the study phase showed significantly greater sensitivity to the study items (Mean *d* =2.64, *SD* = 0*.*97) than those in the color photograph (Mean *d* = 1.89, *SD* = 0*.*52; *p <* 0*.*004, Games-Howell *post-hoc* test for inequality of variance) and line drawing conditions (Mean *d* = 1.59, *SD* = 0*.*82, *p <* 0*.*001, Games-Howell). There was no difference in d between the two pictorial conditions (*p* = 0*.*12, Games-Howell).

Finally, we considered whether there were commonalities in the *types* of stimulus items that were recalled in the real object vs. pictorial viewing conditions. Our stimulus set was varied and the items may be categorized a number of ways (i.e., fruits vs. non-fruits, natural vs. man-made objects, tools vs. nontools, etc.). We examined observers' recall performance across Items in each Viewing Condition using a mixed-model 2-way ANOVA with the between-subjects factor of Viewing Condition (real, photographs, line drawings) and the within-subjects factor of Item ID (*n* = 44). This analysis revealed a main effect of Viewing Condition [*F*(2*,* 76) <sup>=</sup> <sup>11</sup>*.*28, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*23], and Item [*F*(25*.*16*,* 3268) <sup>=</sup> <sup>7</sup>*.*50, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*09], and a significant Viewing Condition × Item interaction [*F*(50*.*33*,* 3268), *p <* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*05], suggesting that recall performance differed

across items as a function of Viewing Condition. **Table 1** presents percent recall data for each item in each viewing condition, and the quartile into which each item fell in % recall (e.g., with items in the 4th quartile being recalled more frequently than those in the 1st). Items that were recalled *most* frequently in the real object displays (e.g., 4th quartile: flashlight, soap, lemon, dice) but less so in the pictorial conditions (below 4th quartile)

object condition also made significantly fewer false recalls than those in

fell into a range of categories including man-made and natural objects, tools and non-tool objects, and objects without scent (note that all fruit/vegetable items were made of plastic). Taken together, the recall and recognition data from Experiment 1 indicate that memory is enhanced for real object displays, and that this enhancement generalizes across a range of stimulus sub-categories.

represent ∗∗*p <* 0*.*001.

It is possible, however, that the improved memory performance for stimuli displayed as real objects could be attributed to differences in viewing angle of the stimuli across the real vs. pictorial conditions. Although the 2D stimuli were generated by photographing the real objects mounted in the presentation box (thereby allowing control of stimulus position, order, and orientation across conditions), the 2D images were *themselves* mounted at an angle of 30◦ on the display shelves, which further increased viewing angle relative to the real objects—possibly making the 2D images more difficult to identify. Although none of our observers complained of an inability to recognize the stimuli (and outlier stimuli were filtered in the initial item analysis described above), it is nevertheless possible that a subtle increase in difficulty in stimulus identification could have manifested as poorer recall/recognition performance. We examined this possibility in a follow-up experiment.

#### **EXPERIMENT 2**

In Experiment 2 we compared memory performance for real objects vs. color photographs in a separate group of observers. Importantly, in Experiment 2 the real object stimuli were presented (and photographed to create matching 2D images) on "shelves" that were oriented vertically, rather than at 30◦ as in Experiment 1, to match viewing angle across the real object and photograph conditions (**Figure 1B**). Free recall and recognition performance were compared using planned comparisons between the Viewing Conditions (two-tailed independent-samples *t*-tests, significant at *p <* 0*.*05).

As in Experiment 1, observers recalled a greater number of items in the real object (Mean = 48.17%, *SD* = 14*.*86%) than the color photograph condition [Mean = 42.05%, *SD* = 11*.*48%; *t*(90) = −2*.*149, *p* = 0*.*034, Cohen's *d* = −0*.*45] (**Figure 3A**). Although there were again fewer falsely recalled items in the real (Mean = 0.93, *SD* = 1*.*05) than the color photograph (Mean = 1.21, *SD* = 1*.*30) condition (**Figure 3B**), this difference did not reach statistical significance [*t*(90) = 1*.*11, *p* = 0*.*27]. Recognition performance (% correct) was also significantly better for real objects (*M* = 90.24%, *SD* = 7*.*38%) than color photographs [Mean = 90.24%, *SD* = 7*.*37%; *t*(90) = −2*.*261, *p* = 0*.*026, Cohen's *d* = −0*.*47] (**Figure 3C**). A SD analysis of the recognition data revealed that sensitivity (mean d ) was significantly higher for real objects (Mean = 2.70, *SD* = 0*.*86) than colored photographs (Mean = 2.33, *SD* = 0*.*91) (**Figure 3D**), [*t*(90) = 2*.*03, *p* = 0*.*045, Cohen's *d* = 0*.*43].

In summary, Experiment 2 replicated the findings of Experiment 1 showing that free recall and recognition performance for real objects was significantly better than matched color photographs of the same items. Importantly, Experiment 2 confirmed that the memory advantage for real objects observed was not attributable to subtle differences in viewing angle of the items across conditions.

Finally, as in Experiment 1, we examined observers' recall performance across Items in each Viewing Condition using a mixed-model 2-way ANOVA with the between-subjects factor of Viewing Condition (real, photographs) and the withinsubjects factor of Item ID (*n* = 44). We observed a main effect of Viewing Condition [*F*(1*,* 92) <sup>=</sup> <sup>5</sup>*.*03, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*05], and Item [*F*(25*.*635*,* 3956) <sup>=</sup> <sup>6</sup>*.*00, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*06], but no higher-level interaction effect [*F*(50*.*33*,* 3268) = 1*.*14, *p* = 0*.*29, <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>*.*01], indicating that although some items were recalled more frequently than others, the pattern of recall performance for the various items was similar across the different viewing conditions. **Table 2** presents percent recall data for each item in each viewing condition, and the quartile into which each item fell in percent recall in each Viewing Condition. As in Experiment 1, items that were recalled more frequently from real object (3 rd–4th quartiles) than color photograph displays fell into a range of different sub-categories.

# **DISCUSSION**

We compared episodic memory for everyday household objects that were viewed as real 3D objects vs. 2D pictures. In Experiment 1 free recall and recognition performance were examined for identical stimulus sets that were viewed in one of three different display formats: real objects, color photographs, and black and white line drawings. Memory performance was superior for stimuli that were displayed as real 3D objects in the Study Phase than line drawings or color photographs of the same stimuli. Analysis of erroneous responses (falsely recalled items, and signal detection analysis of the recognition data) confirmed that observers who viewed real objects were indeed more sensitive to the study material than those who viewed the same stimuli in pictorial form. In Experiment 2, free recall and recognition performance were compared for real objects vs. color photographs when viewing angle was matched across the different display conditions. Again we found superior memory performance for real objects over color photo displays, and the pattern of data could not be explained by effects of response bias. Finally, item-based analyses of the recall data in Experiments 1 and 2 indicated that the memory advantage for real objects generalized across a range of different stimulus types, including man-made and natural, tools and non-tool objects. Taken together, our data demonstrate for the first time that real objects are more memorable than picture representations.

We argue that the influence of real object displays on memory is due to a fundamental mnemonic advantage for real objects. In terms of alternative interpretations we considered the possibility that there was a systematic difference in encoding strategy adopted by observers in the different viewing conditions. This account seems unlikely for several reasons. The stimuli in each condition were presented in the same order and position on the display shelves, such that the spatial and semantic relationships between the items were constant across conditions. Viewing time was also equivalent across conditions. Further, given our large sample size it is unlikely that all subjects within a given group employed the same encoding strategy and that these strategies were reliably different across groups but consistent across experiments.

Below, we discuss several factors that might be important in driving the real object advantage in memory. As outlined in the introduction, binocular vision provides information about the geometric structure of real objects that is not available when looking at 2D images (Blake and Wilson, 2011). The memory advantage for real objects over images may be due to additional

of the same items. **(B)** There was no difference in the number of falsely recalled items between real objects and photos. **(C)** Analysis of the

material than those who studied color photos of the same objects. Error bars represent *SE*. ∗*p <* 0*.*05.

binocular cues to 3D depth structure in our real object displays. If additional shape cues facilitate memory, the question arises as to exactly what *type* of shape cues? In Experiment 1, memory performance was equivalent in the black and white line drawing and color photo condition, suggesting that color and monocular shape cues (i.e., shading and surface texture) were insufficient to influence object memory, despite the fact that color and other surface cues have been shown to have influence on object identification (Humphrey et al., 1994). The possibility remains however, that additional binocular stereo cues to shape could enhance memory performance. Although depth information from disparity may have a modest effect on the time taken to *recognize* an object (Edelman and Bulthoff, 1992; Humphrey and Khan, 1992), this result has not been supported in all studies (D'erme et al., 1994). Interestingly, previous studies comparing memory and other cognitive measures for 2D displays with matched virtual 3D computer-generated displays have found that performance actually declines when observers move from 2D to 3D conditions—at least in cluttered environments such as digital aviation panels, or navigating complex web-pages (Wickens et al., 1996; Risden et al., 2000; Cockburn and Mckenzie, 2002). Valsecchi and Gegenfurtner (2012) reported a stereo-viewing enhancement in long term memory for forest pictures, even when subjects had no explicit memory of the format in which the stimuli had originally been displayed. Interestingly, this memory enhancement was contingent on lengthy (7 s) display times and was not apparent across all stimulus categories: stereo viewing did not enhance recognition of other scene categories such as car and house images. Valsecchi and Gegenfurtner (2012) concluded that the beneficial effect of stereo information on memory is apparent only when the 3D structure of the object or scene is relevant to the subject's task (i.e., spatial layout), and when observers have sufficient time to study the image. With simple displays containing isolated objects, such as those used here, memory may be less influenced by task-irrelevant or otherwise distracting visual information, thereby revealing a beneficial effect of 3D stereo cues on memory relative to 2D images. Critically, however, none of these studies of stereo vision have compared memory for virtual 3D object displays with *real* 3D objects.

It is interesting to consider whether the real object advantage might be explained by other higher-level attributes intrinsic to real objects. Real objects are tangible substances that exist in 3D space, with a definite surface texture, compliance, and function. Images, conversely, are abstract representations of objects that must be learned during childhood to be fully understood (Deloache et al., 2003). Perhaps most importantly, objects that are placed within reaching distance afford action, such as grasping and manipulation (Gibson, 1979; Norman, 2002) whereas images of objects do not. Indeed, graspable objects are particularly relevant for dorsal-stream motor networks (Chao and Martin, 2000; Creem-Regehr and Lee, 2005; Handy et al., 2006; Proverbio et al., 2011). Several of these areas are known to be highly sensitive to whether a real graspable object is within or outside of reach (Gallivan et al., 2011). The objects in our study were presented within reach of all subjects, thereby reasonably offering affordances. Dorsal stream regions within inferior parietal cortex are known to be active during working memory (Todd and Marois, 2004; Xu and Chun, 2006), episodic memory (reviewed in Wagner et al., 2005; Cabeza et al., 2008; Vilberg and Rugg, 2008), and motor planning (Chao and Martin, 2000; Handy et al., 2003; Creem-Regehr and Lee, 2005; Handy et al., 2006; Proverbio et al., 2011; Makris et al., 2013; Garrido-Vasquez and Schubo, 2014). Our results raise the intriguing possibility that real objects may be more memorable because they more strongly activate dorsal stream regions at encoding, perhaps promoting deeper processing and superior memory. In other words, real objects may have a memory benefit due to embodied cognition (Glenberg, 1997; Barsalou, 2008). There are some data to support this interpretation, showing differential neural responses when maintaining objects with and without affordances in memory. For example, Mecklinger et al. (1998) examined the role of motor affordances on working memory using object images. These authors found that maintaining manipulable objects in memory increased ventral premotor cortex activity whereas maintaining non-manipulable objects in memory activated inferior frontal gyrus. In line with this idea, previous studies have reported memory improvements for images of graspable objects (Apel et al., 2012; Downing-Doucet and Guerard, 2014); but see, Pecher (2013) and Quak et al. (2014). It should be noted however, that thus far, *all* of these studies used 2D pictures of objects (which do not afford action in and of themselves), rather than real world exemplars.

It is possible that the real objects in our study were perceived as being more useful or valuable (perhaps because of their direct relevance for action), thereby influencing how memorable they were. In a recent study, Bushong et al. (2010) gave college students money to "bid" on different types of snack foods, which (depending on the bid) could be purchased at the end of the study. Using a between-subjects design, the students viewed the snack foods in the bidding phase in one of three different viewing conditions: real foods, color photographs of the same foods, or a text display of the snack food name. Bushong et al. (2010) found that students were willing to pay over 60% more for foods that were displayed as real objects vs. image or text displays. The same effect was replicated using small trinkets. In the domain of human memory, previous studies have shown that items associated with a higher incentive can be remembered strategically better than items with a lower perceived payoff (Castel et al., 2002). To the extent that our real everyday objects were perceived as being more valuable than the matched image displays, this could also have resulted in a beneficial mnemonic influence.

Real objects have an unambiguous size, distance, and location relative to the observer. In the current study stimuli in all viewing conditions were matched for retinal size. It is the case, however, that observers in the picture conditions were not explicitly aware that the size of the images corresponded to the real world size of the objects. Recent evidence suggests that information about the real world or "familiar" size of objects is accessed relatively automatically (Konkle and Oliva, 2012a), and may be a guiding principle in the large-scale organization of object representations in occipitotemporal cortex (Konkle and Oliva, 2012b). It is interesting to speculate as to whether the real object benefit is heightened in our study by permitting immediate access to stored representations of object identity in object-selective cortex, relative to those who viewed images and whose size information is not explicit. Complementing these findings, damage to inferior occipitoparietal cortex can disrupt distance perception and motor planning that can be partially rescued by object familiarity (Berryhill and Olson, 2009).

In conclusion we found that memory for real objects was significantly better than 2D image representations of the same exemplars. Our data shed important new light on the fundamental yet largely overlooked question of whether pictures are an appropriate proxy for real objects in psychology and neuroscience. These results pave the way for more detailed investigations of the mechanism for the memory advantage for real objects and their underlying neural basis. The findings reported here suggest that although convenient, the use of images in memory research is likely to underestimate memory performance and to reflect incomplete mnemonic processing.

# **ACKNOWLEDGMENTS**

We wish to thank Alexis Carroll and Scott Turek for assistance with data collection. The research reported in this publication was supported by National Institute of General Medical Sciences of the National Institutes of Health under grant number P20 GM103650, and from the National Eye Institute, National Institutes of Health under grant number EY022775.

# **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 01 August 2014; accepted: 01 October 2014; published online: 20 October 2014.*

*Citation: Snow JC, Skiba RM, Coleman TL and Berryhill ME (2014) Real-world objects are more memorable than photographs of objects. Front. Hum. Neurosci. 8:837. doi: 10.3389/fnhum.2014.00837*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Snow, Skiba, Coleman and Berryhill. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Directional interactions between current and prior saccades

# *Stephanie A. H. Jones , Christopher D. Cowper-Smith and David A. Westwood\**

*Action Lab, School of Health and Human Performance, Dalhousie University, Halifax, NS, Canada*

#### *Edited by:*

*Simona Monaco, York University, Canada*

#### *Reviewed by:*

*Joe MacInnes, Higher School of Economics, Russia Amirsaman Sajad, York University, Canada*

#### *\*Correspondence:*

*David A. Westwood, School of Health and Human Performance, Dalhousie University, Dalplex - Kinesiology Suite, 6260 South Street, PO Box 15000, Halifax, NS B3H 4R2, Canada e-mail: david.westwood@dal.ca*

One way to explore how prior sensory and motor events impact eye movements is to ask someone to look to targets located about a central point, returning gaze to the central point after each eye movement. Concerned about the contribution of this return to center movement, Anderson et al. (2008) used a sequential saccade paradigm in which participants made a continuous series of saccades to peripheral targets that appeared to the left or right of the currently fixated location in a random sequence (the next eye movement began from the last target location). Examining the effects of previous saccades (*n*−*x*) on current saccade latency (*n*), they found that saccadic reaction times (RT) were reduced when the direction of the current saccade matched that of a preceding saccade (e.g., two left saccades), even when the two saccades in question were separated by multiple saccades in any direction. We examined if this pattern extends to conditions in which targets appear inside continuously marked locations that provide stable visual features (i.e., target "placeholders") and when saccades are prompted by central arrows. Participants completed 3 conditions: peripheral targets (PT; continuous, sequential saccades to peripherally presented targets) without placeholders; PT with placeholders; and centrally presented arrows (CA; left or right pointing arrows at the currently fixated location instructing participants to saccade to the left or right). We found reduced saccadic RT when the immediately preceding saccade (*n*−*1*) was in the same (vs. opposite) direction in the PT without placeholders and CA conditions. This effect varied when considering the effect of the previous 2–5 (*n*−*x*) saccades on current saccade latency (*n*). The effects of previous eye movements on current saccade latency may be determined by multiple, time-varying mechanisms related to sensory (i.e., retinotopic location), motor (i.e., saccade direction), and environmental (i.e., persistent visual objects) factors.

#### **Keywords: saccade latency, peripheral cue, central cue, random walk paradigm, sequential saccades**

# **INTRODUCTION**

The ability to direct our gaze to relevant stimuli (e.g., locations, objects, events) within the environment is an important part of the process through which we detect and perceive visual information and interact with the world around us. Spatial and temporal changes in eye movements observed in relation to previous sensory and/or motor events can provide insight into the underlying neural mechanisms associated with both perception and action. A common approach in this regard has been to elicit saccades to targets distributed about a point of central fixation which serves as the starting point for all trials and to which the gaze must return after each target-directed saccade (e.g., Taylor and Klein, 2000; Fecteau et al., 2004; Reuter et al., 2006; Cowper-Smith and Westwood, 2013; Cowper-Smith et al., 2013). However, it is possible that any effects observed while employing such methodology may be influenced (or even contingent upon) the task structure requiring the participant to return their gaze to a central fixation point, because of the predictability that it introduces. In effect, following each saccade to an eccentric location, there is a completely predictable spatial, and often temporal, return of attention (or gaze) to the central fixation position.

As a way to circumvent the requirement for a return of gaze to a central position between saccade events, Anderson et al. (2008) employed a *random walk* consecutive saccade paradigm in which three participants (two of which were authors) made a continuous series of saccades (200 per run; participants completed 60 or 120 runs for a total of 12,000 or 24,000 saccades) to targets that appeared to the left or right (1.4◦) of the currently fixated location in a random sequence. Each successive target appeared a constant distance to the left or right of the currently fixated location on a random basis, such that any saccade could equally well be followed by a saccade in the same or opposite direction. No placeholders (persistently visual target representations that indicate to participants where targets might appear) were used to mark the location of possible targets, and the currently fixated target disappeared simultaneously with the appearance of the subsequent target. Anderson et al. (2008) analyzed saccadic reaction time as a function of current saccade (*n*) direction (left or right), but more importantly the direction of preceding saccades (*n*−*1*, *n*−*2*, *n*−*3* and so on; same or opposite direction). It is important to note that the random walk paradigm permits an analysis of the independent effect of any number of preceding saccades' directions because the directions of the intervening saccades are randomly determined.

The authors observed significantly reduced saccadic latencies when the immediately preceding saccade (*n*−*1*) was in the same (vs. opposite) direction (i.e., leftward saccade reaction times were reduced when preceded by a leftward rather than rightward saccade). We refer to this pattern as a "same direction benefit" (SDB), in reference to the reduction in saccadic reaction time that was observed when a preceding eye movement was made in the same direction as the current eye movement. The magnitude of the SDB observed at the *1-back* level (i.e., the reaction time of a saccade as a function of the relative direction of the immediately preceding saccade) ranged from 4 to 14 ms for the three participants in the study (c.f. Anderson et al., 2008, Figures 1 and 4 in particular). Interestingly, the results of their *1-back* analysis are also entirely consistent with the presence of the phenomenon of inhibition of return (IOR, the time-dependent slowing in participants' ability to orient to and process information in a previously attended and/or fixated location; Posner and Cohen, 1984) since consecutive saccades in opposite directions return the gaze to the most recently inspected locations and should result in longer reaction times.

Unlike paradigms requiring a return of gaze to a central location between each saccade, the random-walk paradigm permits an analysis of the reaction time of a particular saccade (*n*) as afunction of the relative direction of any number of previous saccades (*n*−*1*, *n*−*2*, etc.). These effects are computed by averaging across all instancesin the sequencewhen a saccadeis preceded by an "n-back" saccade (i.e., a saccade preceding the current saccade) in the same or opposite direction; such averaging does not create systematic biases because the intervening saccades in each instance are in random directions. Interestingly, Anderson et al.'s result showed a significant but exponentially decreasing SDB for *n-back* levels greater than one: SDB, previous saccades in the *same* direction contributed to a significantly decreased saccadic latency, even if separated by as many as 5 saccades, regardless of the directions of the intervening eye movements.

This is an important result because it indicates that the SDB cannot be attributed to a simple location-based effect such as IOR. Only at the *n*−*1* level is it true that two saccades in opposite directions *necessarily* return the gaze to the most recently inspected location, and that two saccades in the same direction *necessarily* bring the gaze to a new location. For *n*−*2* and higher levels, there can be multiple intervening saccades in any direction so there is no systematic relationship between the relative directions of the two saccades in question and the locations from which those saccades were generated. Hence, any difference in saccadic reaction time due to the relative direction of the *n*−*2 or higher* saccade cannot be due to location-based effects. As such, Anderson et al. (2008) suggest that the previous-saccade effects they observe in the random walk paradigm are distinct from IOR and are driven by the similarity of the directions of the current and previous saccades, rather than the return of gaze to a previously inspected location. This is an important consideration for studies that use a return to a central fixation task structure, because in this case the slower "same location" trials consist of a series of three movements (L-R-L or R-L-R) for which the final saccade is necessarily in the opposite direction from the most recently completed eye movement. According to Anderson et al.'s interpretation, this sequence of saccades may result in a slower final saccade not because the gaze returns to an old location but rather because the final saccade is immediately preceded by an eye movement in the opposite direction.

Although Anderson et al. (2008) highlighted the relevance of their results ". . . in the real world. . . " (p. 614) and suggested that their results might indicate ". . . [that] the neural centers responsible for directing our gaze—and consequently, our overt attention—have evolved to reflect the patterns of the real world environment." (Anderson et al., 2008, p. 617), their initial use of the random walk paradigm lacks several characteristics associated with eye movements in the "real world." First, Anderson et al. (2008) showed only one target on the screen at a time, which turned off when the next target appeared. In the real world, gaze is often directed from one object to another, both of which remain visible before, during, and after the eye movement. Indeed, many studies exploring the effects on saccades of prior sensory and motor events, particularly those examining IOR, leave placeholder markers in the possible target locations (to indicate to participants approximate areas in which targets may appear) and it has been reported that the magnitude of IOR [which is relevant to the *1-back* analysis employed by Anderson et al. (2008)] can be reduced or eliminated in the absence of target location placeholders or other stable or permanent objects (e.g., Klein, 1988; Klein and MacInnes, 1999; Birmingham and Pratt, 2005). Second, Anderson et al. (2008) employed peripheral onset targets to draw gaze (and attention) to subsequent target locations. While some stimuli in the real world appear as a peripheral change in luminance, oftentimes the target of an eye movement is determined by a subjective change in the salience of an object whose luminance remains constant (e.g., identifying a currently fixated stimulus as a non-target and moving the gaze to a different object in the visual array, or perhaps looking at an object to which a friend is pointing). From a mechanistic standpoint, peripheral onset targets confound sensory and motor processes; an eye movement to a peripheral target that appears to the left of the currently fixated location could exhibit a reduced latency because it shares the same location on the retina as a previous peripheral target or because the direction of the eye movement is the same as the previous saccade. If the SDB observed by Anderson et al. (2008) arises from motor rather than sensory processes (as they propose), then similar effects should be observed regardless of whether saccades are elicited using peripheral onset targets or endogenous target selection. Here, we have employed central arrows and peripheral onset targets to elicit saccadic eye movements.

Therefore, the purpose of the present study was to extend Anderson et al.'s (2008) random walk sequential saccade paradigm to a group design involving task characteristics more similar to real-world stimulus detection scenarios. Twenty-six participants completed the random walk sequential saccade paradigm in three blocked, randomly ordered, stimulus conditions: (1) peripheral targets without placeholders: a single target appeared simultaneously with the offset of the previously presented target (similar to the methods employed by Anderson et al., 2008); (2) peripheral targets with placeholders: all possible target locations remained visible on the computer screen throughout the entire sequence of saccades (i.e., placeholders); and (3) centrally presented arrow targets with placeholders: an arrow presented at the currently fixated location signaled the direction of the subsequent saccade and placeholders for all possible target locations remained visible on the computer screen for the entire block of saccades. To date, ours is only the second paper to employ the random walk paradigm to examine saccadic dependencies.

# **MATERIALS AND METHODS**

# **PARTICIPANTS**

Twenty-six undergraduate Psychology students (5 males) aged 17–28 years (Mean age = 20.34 ± 2.60 years) from Dalhousie University, Halifax, N.S. participated in the current study in exchange for partial course credit. All participants had normal or corrected-to-normal vision, were right-handed (self-reported) and had no history of visual, motor or neurological abnormalities (self-reported). This study was approved by the Dalhousie University Social Sciences and Humanities Research Ethics Board. Written informed consent was provided by all participants prior to their participation. All participants completed all three sequential saccade conditions in which they responded to peripherally or centrally signaled targets (described in more detail below) in a random order.

# **PROCEDURES**

Each participant completed a sequential saccade task (the random walk paradigm; Anderson et al., 2008) in three separate and randomly ordered conditions. A schematic sequence for each of the three conditions is presented in **Figure 1** (not to scale). Participants were comfortably seated in a dimly lit room (consistent with Anderson et al., 2008), 75 cm in front of a 31 computer monitor (Tyco Electronics ©) on which all stimuli were presented using Experiment Builder® version 1.10.1 (SR Research Inc, Canada). Targets were circles (1.9◦ in diameter, black, presented on a white background) that could appear at 19 possible locations spaced 2.7◦ apart (center to center) along an imaginary horizontal line vertically centered on the computer screen (only a subset of the possible target locations is presented in **Figure 1**). Each condition began with the participant foveating the central target location (labeled "Start" in **Figure 1**). An infrared headmounted eye tracking system (EyeLink II®; SR Research Inc, Canada) was used to record the position of the right eye at 250 Hz. The eye tracking system was calibrated before participants completed each condition. Participants were given a break in between conditions. Participants were instructed to move only their eyes in response to the onset of a target and were given an opportunity to practice in advance of their participation. The absence of head movement was ensured by the experimenter throughout the entire testing session.

# *Peripheral target conditions*

In the peripheral target (PT) with placeholders condition (**Figure 1A**), all 19 possible target locations were continuously marked on the computer screen (i.e., as

**the peripheral target (PT) with placeholders (A), peripheral target (PT) without placeholders (B) and central arrow conditions (C).** Participants began by fixating the center circle (labeled "start"). A cue was then presented (bold outline of the target circle, the appearance of the target

cued target location where they maintained fixation for 770–1760 ms, after which another cue was presented signaling the subsequent target location to the left or right of the currently fixated location randomly. Only 9 of the 19 possible target locations are shown in the Figure.

"placeholders"—distinguished from the target by a single feature, outline thickness) by circles matching the size of the target but with reduced line thickness (0.06 mm thick). The task began once the participant foveated the central circle (labeled "start" in **Figure 1A**). Subsequent targets were indicated by a "boldening" of the outline of the appropriate placeholder circle for 300 ms (0.16 mm thick). To create a random walk sequence of leftward and rightward saccades, targets were randomly presented at the placeholder one position to the left or right of the currently fixated location. Participants were instructed to saccade directly to the target circle and maintain fixation on that circle during the 770–1760 ms response-stimulus-interval until the next target appeared [response stimulus interval (RSI): the amount of time in between gaze reaching the region of interest of the target (the response) and the onset of the subsequent target]. The currently fixated circle was restored to its original line thickness (0.06 mm) coincident with the "boldening" of the subsequent target. Participants completed 204 saccades in this condition, comprised of 20 sequences of 10 saccades, separated by a self-timed drift correct (see below). Errors are defined below in Section Errors.

To limit fixational eye movements such as drift (e.g., Di Statsi et al., 2013), after every 10 saccades, the program would pause for the presentation of a drift correct [a dot positioned in the center of the central target location (i.e., "start")]. As the beginning of a subsequent sequence of saccades after the drift correct was selfdirected by the participant (through the press of a button or by asking the experimenter to advance the task), participants could use this time to rest their eyes if needed (they were instructed to limit their head and body movement during these breaks). Once ready to continue, participants began the next trial sequence with their gaze at the central target location (start); the program would not progress unless central target fixation was achieved. A description of the number of valid trial sequences for each n-back level is presented in the error section below and in **Table 1**.

In the PT without placeholders condition (**Figure 1B**), no placeholders were used to mark potential target locations. Circle targets simply appeared at new locations coincident with the offset of the circle at the currently fixated location (i.e., they appeared to "jump" from one location to another). All other spatial and temporal characteristics were identical to the PT with placeholders condition. This condition is most similar to that employed by Anderson et al. (2008).

# *Centrally presented arrows*

In the central arrow condition (**Figure 1C**) all 19 possible target locations were continuously marked with circular placeholders

**Table 1 | Number of valid trial sequences for analysis, averaged across participants, for each condition and n-back level.**


(as in the PT with placeholders condition). Once the participant achieved fixation at the central location, the task would begin. A randomly selected leftward or rightward pointing arrow (1.9◦ long, 0.6◦ in width, equivalent area at the head and tail of the arrow) was presented for 300 ms, horizontally and vertically centered within the boundaries of the currently fixated placeholder. Participants were asked to saccade directly to the placeholder location immediately to the left or right of the current location, depending on the direction of the arrow (i.e., saccade to the placeholder location to the left if the arrow is pointing leftward and saccade to the placeholder location to the right if the arrow is pointing rightward). As in the other two conditions, participants were asked to maintain fixation within the placeholder until the next arrow was presented 770–1760 ms later. Consistent with the peripheral target conditions described above, participants completed 204 saccades in this condition, comprised of 20 sequences of 10 saccades, separated by a self-timed drift correct (as described above).

# *Errors*

In all three conditions, if a response to a target took longer than 2000 ms, saccade endpoint was not within the region of interest of the target (i.e., within 0.6◦ from the perimeter of the target circle) or gaze deviated, drifted or was directed outside the region of interest of the current target in advance of a new target being presented, participants received an error message. A four second time penalty accompanied the error message and participants were instructed to return their gaze to the central target location (i.e., the start). A new target sequence began once the participants' gaze reached the central target location. In all conditions, the percentage of errors across all target events was used to assess the accuracy of saccades for each participant (Mean accuracy: PT without placeholders: 82% of all saccades; PT with placeholder: 89% of all saccades; CA with placeholder: 88% of all saccades). Target events associated with errors were excluded from further analyses (i.e., saccades associated with this target event were not considered as saccade *n*). Moreover, all *n-back* analyses (described later) were restricted to continuous sequences of target events free from errors or drift corrects, because errors and drift corrects interrupted the series of movements by requiring a return to the center target location. As such, RTs were computed for accurate saccades within error and drift correct free trial sequences in each of the three stimulus conditions. The average number of valid trial sequences for each n-back level for each condition are presented in **Table 1**.

# **ANALYSES AND HYPOTHESES**

# **THE EFFECT OF THE IMMEDIATELY PRECEDING SACCADE (N−1)**

The primary analysis considered the latency of saccade *n* as a function of saccade *n* direction (left or right; this factor was included to take into account the possibility of asymmetries in saccade latency) and the relative direction of the immediately preceding saccade (saccade *n*−*1*: same or opposite). Each condition was analyzed separately. The *n*−*1* level was considered separately from all other *n* levels because, in this unique case, the presence of a SDB (i.e., for two consecutive leftward or rightward saccades) could be due to a directional interaction within the saccade motor control system, a location-based inhibition of return, or some combination of the two.

**Figure 2** depicts three possible outcomes for the *n*−*1* level analysis. If, as Anderson et al. (2008) speculated, a SDB arises from the execution of eye movements *per se,* then similar SDB should be observed in all three conditions in the present study because all require sequential saccades (**Figure 2A**: for all conditions, saccadic *n* latency is shorter when the *n*−*1* saccade is in the same vs. opposite direction). If, instead, SDB arises from a sensory mechanism related to the different retinal locations stimulated by peripheral targets in the "opposite direction" as compared to "same direction" sequences, then SDB should be observed in the two peripheral target conditions but not in the central arrow condition (**Figure 2B**: for PT without placeholders and PT with placeholders only, saccade *n* latency is shorter when saccade *n*−*1* is in the same direction). If SDB at the *n*−*1* level is indeed distinct from the IOR phenomenon, then it should be observed whether or not possible target locations remain persistently visible during the task. In contrast, if SDB is related to IOR then it should be strongest in the conditions associated with placeholders and weakest in the single condition with no placeholders (Klein, 1988; Klein and MacInnes, 1999; Birmingham and Pratt, 2005) (**Figure 2C**: in all conditions, saccade *n* latencies are shorter when saccade *n*−*1* is in the same direction, but less so for PT without placeholders).

# **THE EFFECT OF PRECEDING SACCADES (N−2 THROUGH N−5)**

Secondary analyses considered the latency of saccade *n* as a function of saccade *n* direction (left or right), and the direction of the saccade completed 2, 3, 4, or 5 targets ago (i.e., *n*−*2* through *n*−*5*). Our choice to examine the effects previous saccades on current saccade latency up until 5 saccades previous was based on the primary finding reported by Anderson et al. (2008), that previous saccades in the *same* direction contributed to a significantly decreased saccadic latency, even if separated by as many as 5 saccades. Each condition was analyzed separately. Importantly, since target presentation is random, the two saccades of interest in these analyses (*n* and *n*−*x*) could be separated by saccades in any combination of directions (left or right). All possible random combinations of left and right intervening saccades are pooled together for analysis. Anderson et al.'s results showed a significant SDB for the *n*−*1* through *n*−*5* preceding saccades, so we expected to find something similar in the PT without placeholders condition (the task most similar to Anderson et al.). For the secondary *n-back* analyses, it was not clear what might be observed in the PT with placeholders and central arrow conditions.

#### **DATA ANALYSIS**

The dependent measure of interest, reaction time (RT) for saccade *n* was analyzed using a 2 (current saccade direction: left or right) × 2 (direction of the previous saccade of interest (*n*−*1, n*−*2, n*−*3, n*−*4, n*−*5* [separate analyses]: same and opposite) × 3 (condition: PT without placeholders, PT with placeholders, central arrow) repeated measures ANOVA. This analysis revealed significant interactions between condition and direction of the preceding saccade of interest at the *n*−*2* [*F*(1.74, <sup>43</sup>.62) = 3.75, *p* = 0.03], *n*−*4* [*F*(1.96, <sup>47</sup>.20) = 6.12, *p* = 0.005] and *n*−*5* levels

**FIGURE 2 | Visual depiction of three possible outcomes of the** *n***−***1* **level analysis. (A)** For all conditions, saccade *n* latency is shorter when saccade *n*−*1* is in the same direction. **(B)** For PT without placeholders (black) and PT with placeholders (dark gray), saccade *n* latency is shorter when saccade *n*−*1* is in the same direction). No such difference is found between same and opposite conditions in the central arrow condition (light gray). **(C)** In all conditions, saccade n latencies are shorter when saccade *n*−*1* is in the same direction, but less so for PT without placeholders (black).

[*F*(1.75, <sup>42</sup>.22) = 3.61, *p* = 0.04]. As such, subsequent analyses examined the effects of preceding saccade on current saccade latency for each condition separately using separate 2 (direction of saccade *n*: left and right) × 2 [direction of the preceding saccade of interest (*n*−*1, n*−*2, n*−*3, n*−*4, n*−*5*) (separate analyses): same and opposite] repeated measures ANOVAs (alpha = 0.05). As reaction times significantly differed as a function of the direction of saccade (i.e., whether the saccade was to the left or right) only in the PT with placeholders condition [*F*(1, <sup>25</sup>.08) = 4.63, *p* = 0.041] at the 1-back level, our results and discussion will focus on the effect of the relative direction of the *n-back* saccade on the reaction time of saccade *n*. Greenhouse-Geisser corrected *p*-values are reported. All analyses were conducted using IBM SPSS Statistics version 21 (IBM Corp. Released 2012. IBM SPSS Statistics for Windows, Version 21.0. Armonk, NY: IBM Corp.).

#### **RESULTS**

**Figure 3** shows average saccadic reaction times for each of the PT without placeholders (A), PT with placeholders (B) and central arrow conditions (C) as a function of previous saccade direction

**FIGURE 3 | Average saccadic reaction time (in ms) for each of the peripheral target (PT): without placeholders (A), peripheral target (PT): with placeholders (B) and central arrow conditions (C) as a function of previous saccade direction relative to the current saccade [same direction as the current saccade (light gray lines and symbols) or opposite direction as the current saccade (black lines and symbols)] and the number of intervening saccades [***1-back* **(0 intervening saccades) to** *5-back* **(4 intervening saccades)].** Error bars are standard error of the mean. Asterisks indicate a significant difference in reaction time between same and opposite.

relative to the current saccade [same direction as the current saccade (light gray lines and symbols) or opposite direction as the current saccade (black lines and symbols)] and the number of intervening saccades [*1-back* (0 intervening saccades) to *5-back* (4 intervening saccades in any direction)]. Error bars are standard error of the mean. A summary of mean reaction times (in ms) for each saccade direction relative to the current saccade (same vs. opposite) as well as the *F* values, *p*-values and effect sizes (Cohen's d) for these comparison are presented in **Table 2**. Overall, reaction times varied across the three conditions [*F*(1.75, <sup>43</sup>.74) = 399.03, *p* < 0.001], with significantly faster reaction times observed in the PT without placeholders condition (*M* = 180 ms, *SD* = 19 ms) than the PT with placeholders condition (*M* = 259, *SD* = 28) than the central arrow condition (*M* = 372, *SD* = 38).

**Table 2 | Average saccadic reaction times (in ms),** *F***-values,** *p***-values and Cohen's** *d* **for each n-back comparison between same and opposite trial types (alpha <sup>=</sup> 0.05)†.**


*†Boldface type indicates significance at the p* < *0.05 level.*

# **THE EFFECT OF THE IMMEDIATELY PRECEDING SACCADE (N−1)**

Analysis revealed an effect of the direction of the immediately preceding (*n*−*1*) saccade on current saccade latency in the PT without placeholders [2A, same: 175 ms, opposite: 178 ms, *F*(1, <sup>26</sup>.11) = 5.78, *p* = 0.024] and central arrow conditions only [2C, same: 359 ms, opposite: 367 ms, *F*(1, 25) = 6.29, *p* = 0.019, **Table 2**]. In particular, in both of these conditions, saccadic reaction times were faster when the immediately preceding saccade (*n*−*1*) was in the same direction as the current saccade *n* (i.e., LL or RR) or a "same direction benefit" (SDB). A non-significant SDB was found in the PT with placeholders condition [2B, same: 249 ms, opposite: 253 ms, *F*(1, <sup>25</sup>.21) = 3.44, *p* = 0.075]. For greater *n-back* levels, this pattern diverged between conditions (as summarized below).

# **THE EFFECT OF PREVIOUS SACCADES (N−2 THROUGH N−5)**

In the PT without placeholders condition, the SDB observed at the *n*−*1* level persisted at the *n*−*2* level [same: 176 ms, opposite: 179 ms,; *F*(1, <sup>28</sup>.45) = 4.75, *p* = 0.038], but was replaced by a significant opposite direction benefit (ODB) at the *n*−*3* and *n*−*5* levels (**Figure 3A**, **Table 2**, no statistically significant difference between same and opposite at the *n*−*4* level). In the PT with placeholders condition, a significant ODB was found at the *n*−*3* and *n*−*4* levels (no difference between same and opposite at the *n*−*2* or *n*−*5* levels although a trend toward an ODB at these *n-back* levels, **Table 2**). In the central arrow condition, a significant SDB was revealed at the *n*−*1, n*−*2,* and *n*−*5* levels (non-significant SDB at the *n*−*4* level), with a significant ODB found at the *n*−*3* level (**Table 2** and **Figure 3C**).

# **DISCUSSION**

We adapted Anderson et al.'s (2008) random walk sequential saccade paradigm to examine the effects of preceding eye movements on saccadic latency in three stimulus conditions: PT without placeholders (as employed by Anderson et al., 2008), PT with placeholders (in which all possible target locations remained present during the entirety of the testing session), and central arrows (with placeholders). Overall, reaction times were fastest in the PT without placeholders condition and slowest in the central arrow condition. When examining the effect of the immediately preceding saccade on current saccade latency, we found that in both the PT without placeholders and central arrow conditions, saccadic reaction times were faster when the immediately preceding saccade was in the same direction as the current saccade. This pattern did not persist across all n-back levels and diverged between conditions. Ours is the second paper to employ the random walk paradigm as a method for examining directional relationships among saccades and the first to explore the effects of placeholders and central arrow cues on the interactions between current and prior saccades.

# **STIMULUS AND SACCADE HISTORY EFFECTS DO NOT SIMPLY DIMINISH OVER TIME**

Consistent with Anderson et al. (2008), we found significantly faster saccadic reaction times when the immediately preceding saccade (*n*−*1*) was in the same direction as the current saccade in our PT without placeholders condition (same: 175 ms vs. opposite: 178 ms) or a SDB. Additionally, we have extended this result to a new random walk task: the central arrow condition (Central arrows: 359 vs. 367 ms). Although the effect of previous saccade direction on current saccade latency in the PT with placeholders condition was not statistically significant at the *n*−*1* level, a similar trend was found in this condition. We will begin by discussing our results as related to our previously presented scenarios about the effects of saccade *n*−*1* on saccade *n* (hypotheses presented in Section Analyses and Hypotheses and **Figure 2**).

# *Stimulus and saccade history effects as a product of the execution of eye movements*

We first considered the possibility that the *n*−*x* effects on saccade *n* latency presented by Anderson et al. (2008) might have arose due to the interactions related to the execution of eye movements in general, as opposed to repeated retinal stimulation due to peripheral targets on the opposite direction trials (Anderson et al., 2008). For example, in their exploration of saccadic dependencies in a real world visual search paradigm, Smith and Henderson (2011) suggested that the increase in saccade latency for two saccades to the same location might be a large part due to an overall bias that we have to direct saccades in the same direction (i.e., saccadic momentum), as opposed to only the avoidance of a previously fixated location. Saccadic momentum effects, whereby participants are biased to continue to saccade in the same direction, have been found in a number of visual search tasks including free (e.g., MacInnes et al., 2014) and array-defined visual search tasks (Hooge and Erkelens, 1996; Hooge and Frens, 2000). However, if this were the case in the random walk paradigm, we would have expected an extension of the SDB observed by Anderson et al. to the two new conditions we employed (PT with placeholders and central arrows). We failed to find a significant SDB at any *n*−*x* level in our PT with placeholders condition and the SDB observed in the PT without placeholders and central arrow conditions did not persist across *n*−*x* levels. This result suggests that the result reported by Anderson et al. (2008) is not due to interactions related to the execution of eye movements in general.

It is possible that the nature of the task or targets employed (e.g., salience, separation between targets) play a role in saccade direction biases such as those reported by saccadic momentum. In fact, research has substantiated the existence of "gradients of importance" across parts of real world targets, such that a bias to return to a previously inspected target or part of a target is dependent on the complexity and functional importance of the target (e.g., Wilming et al., 2013). For example, visually inspecting a coffee mug might result in a bias to return gaze repeatedly to the handle because the handle guides physical interaction with the mug. As such, unlike many objects that are visually inspected in the real-world, the targets employed in the current study do not have functional use and so there is no pre-determined component of the target that might draw gaze more than any other component (e.g., the handle of a tool, the opening of a vessel etc.). Likewise, unlike, the tasks reported by Hooge and colleagues and MacInnes et al. (2014), the random walk paradigm might not elicit location-dependent gradients in salience because it is not self-paced or free visual search.

# *Stimulus and saccade history effects as a product of peripheral onset targets*

A second possibility that was considered was that the SDB reported by Anderson et al. arose from a sensory mechanism related to the different retinal locations stimulated by peripheral targets in the "opposite direction" as compared to "same direction" sequences. This scenario would predict that a SDB would be observed in the two PT conditions only, but not the central arrow condition (note the flat light gray line in **Figure 2B**). While our results did not substantiate this possibility completely, we are not willing to completely dismiss this explanation either.

We found differences in the overall reaction times across the three conditions, with the shortest saccade latencies observed in the PT without placeholders condition and the longest latency observed in the central arrow condition. This difference in saccadic latency across conditions is reasonable as peripheral targets elicit more reflexive saccadic responses, responses that are likely to be faster than those elicited using central arrows (Abrams and Dobkin, 1994; Taylor and Klein, 2000; Fischer et al., 2003; Hilchey et al., 2012; Cowper-Smith et al., 2013). However, not only did we find differences in the overall reaction times across our three conditions, we also found differences in the patterns of effects of previous saccades on current saccade latency as a function of stimulus condition. Overall, these results provide some support for the presence of different mechanisms operating for peripheral and centrally presented targets.

For example, In the IOR literature, central arrows have been used to distinguish between effects that might arise from sensory vs. motor processes (Abrams and Dobkin, 1994; Taylor and Klein, 2000; Fischer et al., 2003; Hilchey et al., 2012; Cowper-Smith et al., 2013). It has been suggested that peripheral cues might elicit sensory effects due to repeated stimulation in the same retinal location and motor effects due to eye movement initiation, whereas central arrows are likely to elicit motor-related effects without the confound of repeated retinal stimulation. Although, based on the current literature, it is still unclear as to whether or not there are meaningful differences in the nature of saccadic dependencies that occur for saccades in response to centrally vs. peripherally presented targets. Abrams and Dobkin's (1994) results suggested that peripheral targets might result in additive sensory and motor contributions, based on the smaller effects observed for saccades guided by central arrows (in which a motor but not sensory contribution is possible) vs. peripheral targets (in which both sensory and motor contributions are possible). In contrast, Taylor and Klein (2000) found similar IOR for saccades to peripheral targets and central arrows. Hilchey et al. (2012) reconciled these disparate findings by demonstrating that differences between central and peripheral targets arise only when stimulus conditions are blocked (i.e., participants complete all trials with peripheral targets, followed by all trials for the central target type), and are therefore likely driven by attentional control settings related to the processing of peripheral cues rather than by differences in the nature of the effect occurring for peripheral and central targets. Peripheral and central arrow cues were blocked in the current study. To date, we are the first to examine the use of centrally presented cues in a random walk paradigm.

# *Stimulus and saccade history effects as inhibition of return*

In our last hypothesis, we considered the relevance of the *n*−*1* analysis to the phenomenon of inhibition of return. Oculomotor IOR has been proposed to promote efficient visual search behavior by reducing the likelihood of revisiting previously searched locations (Klein, 1988; Klein and MacInnes, 1999). In real world visual search—one could imagine a foraging scenario in which food needs to be found for survival—discouraging the return of gaze to old locations would increase the efficiency of visual search and may increase the likelihood that the target of interest would be found. If the SDB is distinct from the IOR phenomenon, then we might expect it to be observed whether or not possible target locations remain persistently visible during the task. In contrast, IOR is moderated by whether target placeholders are present (Klein, 1988; Klein and MacInnes, 1999; Birmingham and Pratt, 2005). So, if the SDB is related to IOR, then we might expect the SDB to also be moderated by the presence or absence of placeholders such that the absence of placeholders (as in our PT without placeholders condition) should result in a smaller SDB than the two conditions in which placeholders are employed. While the magnitude of the SDB was greater in the central arrow (with placeholders) than the PT without placeholders condition, we did not observe the SDB in the PT with placeholders condition so we cannot completely substantiate this explanation.

The effects of previous saccade direction observed in the present investigation—and those reported by Anderson et al. (2008)—are smaller than IOR that has been previously reported in target-target saccade paradigms with the predictable return to center movement in between target presentations. For example, Taylor and Klein (2000) reported 21 ms of IOR in their saccadesaccade condition with peripheral targets, which included placeholders (vs. 4 ms in the PT with placeholders condition here) and 21 ms of IOR in their saccade-saccade condition with central arrows (vs. 8 ms in our central arrow condition). Superficially, the relatively small "IOR" observed for IOR-like sequences occurring by chance in the context of an entirely random sequence of left/right target directions in the present study might suggest that the predictability of the return to center saccade in typically employed IOR paradigms might be an important contributor to the IOR phenomenon. A direct comparison of predictable and unpredictable return to center sequences is necessary before reaching this conclusion; after all, target separation was dramatically different in the present study (2.7◦) and Taylor and Klein (2000) (7.9◦) among other methodological differences related to stimulus timing and location within the display.

Overall, our results suggest that the interactions between prior and current eye movements are complex and may not conform to the argument that saccade history effects diminish over time (Anderson et al., 2008). Regardless, as objects in the environment do not disappear, the maintenance of placeholders in our peripheral target with placeholders condition is more ecologically valid than ours or Anderson et al.'s (2008) PT without placeholders conditions.

It is worth noting that Anderson et al. (2008) used a singlesubjects design involving only a small number of participants (*n* = 3, the two authors of the study and one naïve participant), a large number of trials per participant (12,000–24,000 saccades), and single subject statistical analysis (which is more sensitive to individual differences). By comparison, most studies exploring saccadic interactions employ a group approach with many participants (often 12 or more), a relatively small number of trials (usually less than 300), and within-subject or mixed analysis of variance (i.e., ANOVA). Because single subject designs rely on many repeated measurements of the variable(s) of interest over a longer period of time, they can better detect the true pattern and magnitude of the effects of interest, while simultaneously accounting for variations in individual participant behavior that might influence the effects (c.f. Gravetter and Frozano, 2012, pp. 395–430). It is possible that the SDB observed in the singlesubjects design is not sufficiently robust to be detectable using a group design. However, in the absence of a direct replication of Anderson et al.'s (2008) results using their single subject analysis, we cannot draw this conclusion.

# **OTHER EFFECTS OF STIMULUS AND SACCADE HISTORY** *Reduced latencies for repeated locations*

Munoz and colleagues (Dorris et al., 1999; Gore et al., 2002) have reported shorter saccadic latencies when gaze is brought to the same location repeatedly—an effect that appears to be opposite to that reported by the oculomotor inhibition of return literature. While these effects could be a product of involving nonhuman primates as test subjects, somewhat consistent with this finding is the significant ODB that we found at higher n-back levels. Together, these results might suggest that there is a timedependent effect of the influence of previous saccades (*n*−*x*) on saccade *n* latency (e.g., perhaps due to residual neuronal activation due to previously executed saccades). In fact, Gore et al. (2002) reported a non-significant decrease in the benefit observed for two consecutive saccades to the same location as the inter-trial interval (time in between the first and second saccade) increased. Future studies employing the random walk paradigm might benefit from varying the time course of the elicitation of saccades, perhaps identifying instances of inhibitory and/or facilitative effects of previous saccades on current saccade latencies.

### *Current fixation location*

We considered the possibility that current fixation location (i.e., screen location) may co-vary with saccade direction to affect saccade latency. In particular, the possibility that, as subsequent saccades bring gaze to more eccentric locations (far left or far right), that participants may *expect* a subsequent saccade to be cued in the opposite direction (despite the random nature of the random walk paradigm). In this case, we might expect faster leftward saccades when current fixation location was farther right and faster rightward saccades when current fixation location was farther left. **Figure 4** plots average reaction time as a function of current fixation location eccentricity and saccade direction (left or right). When we regressed reaction time on current fixation location (19 possible locations) and the interaction between current fixation location and saccade direction (2 directions; 38 cells in total) within each of the three conditions, we found no significant interactions between current fixation location and saccade direction in our PT without placeholders [*t*(3395) = −0.09, *p* = 0.93, β = −0.002] and central arrow conditions [*t*(3845) = −0.504, *p* = 0.61, β = −0.009]. While the interaction between current fixation location and saccade direction was significant in the PT with placeholders condition [*t*(3968) = −2.45, *p* = 0.014, β = −0.04], the pattern of results do not suggest that as current fixation location became more eccentric, participants anticipated a cue to saccade in the opposite direction. In particular, saccades to the left beginning from eccentric rightward locations were not faster than right saccades from the same locations (likewise for right saccades from eccentric leftward locations), as would be expected if participants were anticipating cues to direct subsequent saccades toward the center (c.f. **Figure 4**). These results are consistent with those reported by Anderson et al. (2008, Figure 4 in particular).

# *Other potential effects*

Similar to the consideration of current fixation location described in the previous section, biases similar to "gambler's fallacy" might arise in the current paradigm. For example, participants might decide that the probability of a target being presented to the left of their current fixation location is higher following several sequential presentations of rightward targets (or vice versa). Such biases could operate at any of the *n*−*x* levels, but the strength of the effect would be relatively small for small *n*−*x* levels and larger for greater *n*−*x* levels (e.g., like flipping a coin, the statistically ignorant gambler begins to expect a "head" outcome only after a relative large number of "tail" outcomes in a row). As such, the potential contribution of a gambler's fallacy to the current study is likely to be relatively minor. Further to this, there are relatively few sequences in our data for which all *n*−*x* saccades

**location eccentricity (in degrees) and saccade direction (left: gray lines or right: black lines) for each of the three conditions [peripheral target without placeholders (A), peripheral target with placeholders (B) and central arrow (C)].** Error bars are standard error of the mean.

are in the same direction (of course this is more pronounced for longer sequences) and an analysis of the gambler's fallacy would be (necessarily) confounded by the potential accumulation of "same direction" effects. In other words, even if one were to demonstrate that a "different" saccade had a reduced reaction time relative to a "same" saccade after a sequence of "same" saccades, one could not say for certain that this was the result of incorrect participant expectations about the likelihood of a particular saccade direction, or if it was the result of an accumulation of effects due to the previous saccades themselves. Future research might benefit from an investigation of this question.

Although our criterion for correct saccades was appropriate for our task, we accepted saccades that terminated outside the perimeter of our targets as correct. It is, therefore, possible that participants made corrective saccades following their initial saccades as a way to return their gaze to a more central portion of the target. We did not quantify the prevalence of such corrective saccades, nor did we consider any effect they might have on subsequent saccade latencies. An inspection of the visual representations of saccadic history (video) for a subset of participants in our study suggests that while corrective saccades were possible, they were infrequent and within the boundary of the target.

Lastly, we must consider that we have assumed that the effects of "same" and "opposite" trial sequences on saccade *n* latency are approximately equal. In the absence of a baseline to which the "same" and "opposite" conditions can be compared, our methods and analysis permit only a comparison *between* "same" and "opposite" trials and do not allow us to determine if the magnitude of the effect of "same" and "opposite" on saccade *n* latency was equivalent.

#### **SUMMARY AND CONCLUSIONS**

The random walk sequential saccade paradigm (Anderson et al., 2008) permits an exploration of the influence of prior saccades on current eye movements, avoiding the potential pitfalls associated with the use of a central fixation location to which gaze (and attention) is drawn after saccades to targets. Here, we extended the random walk paradigm to: (1) examine the role of visual placeholders in saccade history effects in the random walk paradigm with peripheral targets; and (2) compare saccade history effects in peripheral and central stimulus conditions which differ in sensory but not motor characteristics.

We identified small but statistically reliable previous saccade effects at many *n-back* levels. At the *n*−*1* level, these effects are broadly consistent with the presence of oculomotor IOR, revealing that saccades had longer latencies when previous saccades were in the opposite direction, as would occur when revisiting a previously inspected target location. IOR is known to be reduced or eliminated when stable visual references are eliminated (Klein, 1988; Klein and MacInnes, 1999; Birmingham and Pratt, 2005), a result not apparent in our data. In all conditions, an ODB emerged at one or more of the higher *n-back* levels, indicating that there might be a time-dependent effect of previous saccade history on saccade *n* latencies. The present results also indicate some differences between central and peripheral target conditions, consistent with the possibility that interactions between prior and current saccades are likely due to multiple sensory and motor mechanisms. Overall, our results suggest that sequential saccade effects could be due to multiple, time-varying mechanisms related to sensory (i.e., retinotopic stimulus location), motor (i.e., saccade direction), and environmental (i.e., persistent visual objects) aspects of the task structure. Further research is needed to distinguish between these possibilities.

#### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 April 2014; accepted: 08 October 2014; published online: 28 October 2014. Citation: Jones SAH, Cowper-Smith CD and Westwood DA (2014) Directional interactions between current and prior saccades. Front. Hum. Neurosci. 8:872. doi: 10.3389/ fnhum.2014.00872*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Jones, Cowper-Smith and Westwood. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Corrigendum: Spatial task context makes short-latency reaches prone to induced Roelofs illusion

# *Bahareh Taghizadeh1,2 and Alexander Gail 1,2,3\**

*<sup>1</sup> Sensorimotor Group, Cognitive Neuroscience Lab, German Primate Center, Goettingen, Germany*

*<sup>2</sup> Faculty of Biology and Psychology, Georg-August-Universität, Göttingen, Germany*

*<sup>3</sup> Bernstein Center for Computational Neuroscience, Göttingen, Germany*

*\*Correspondence: agail@gwdg.de*

#### *Edited by:*

*J. Douglas Crawford, York University, Canada*

#### *Reviewed by:*

*Lore Thaler, Durham University, UK Thomas Schenk, Universitätsklinikum Erlangen, Germany*

**Keywords: reach movement, induced Roelofs effect, Illusion, reference frame, allocentric, object-centered**

# **A corrigendum on**

**Spatial task context makes short-latency reaches prone to induced Roelofs illusion** *by Taghizadeh B., and Gail A. (2014). Front. Hum. Neurosci. 8:673. doi: 10.3389/fnhum. 2014.00673*

The authors regret errors in the reported subject numbers and values for the illusion size in the data for individual subjects. This affects experiments Ia, Ib, II, and IIa. This mistake does not affect any of the conclusions of the paper since the reported mean values and standard errors were all correct. Correct values for the illusion size in the *N* individual subjects for each experiment are as follows:

E Ia (*N* = 11): 4.13◦ 4.69◦ 0.30◦ 2.08◦ 4.70◦ 1.11◦ 4.15◦ 1.93◦ 4.37◦ 1.27◦ 4.16◦

E Ib (*N* = 9): 0.17◦ −0.03◦ 0.13◦ 0.42◦ 0.25◦ 0.09◦ −0.04◦ −0.14◦ 0.06◦

E II (*N* = 10): 3.83◦ 3.16◦ 1.91◦ 4.76◦ 4.59◦ 4.33◦ 4.45◦ 4.71◦ 5.07◦ 0.26◦

E IIa (*N* = 10): 4.12◦ 3.33◦ 0.43◦ 4.69◦ 4.26◦ 4.91◦ 4.33◦ 4.36◦ 3.36◦ 0.55◦

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 27 September 2014; accepted: 29 October 2014; published online: 14 November 2014.*

*Citation: Taghizadeh B and Gail A (2014) Corrigendum: Spatial task context makes short-latency reaches prone to induced Roelofs illusion. Front. Hum. Neurosci. 8:923. doi: 10.3389/fnhum.2014.00923*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Taghizadeh and Gail. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Binocular advantage for prehension movements performed in visually enriched environments requiring visual search

# *Roshani Gnanaseelan , Dave A. Gonzalez and Ewa Niechwiej-Szwedo\**

*Visuomotor Neuroscience Lab, Department of Kinesiology, University of Waterloo, Waterloo, ON, Canada*

#### *Edited by:*

*Irene Sperandio, University of East Anglia, UK*

#### *Reviewed by:*

*Martin Lages, University of Glasgow, UK Robert Volcic, Istituto Italiano di Tecnologia, Italy*

## *\*Correspondence:*

*Ewa Niechwiej-Szwedo, Department of Kinesiology, University of Waterloo, 200 University Avenue West, Waterloo, ON N2L 5G1, Canada e-mail: eniechwi@uwaterloo.ca*

The purpose of this study was to examine the role of binocular vision during a prehension task performed in a visually enriched environment where the target object was surrounded by distractors/obstacles. Fifteen adults reached and grasped for a cylindrical peg while eye movements and upper limb kinematics were recorded. The complexity of the visual environment was manipulated by varying the number of distractors and by varying the saliency of the target. Gaze behavior (i.e., the latency of the primary gaze shift and frequency of gaze shifts prior to reach initiation) was comparable between viewing conditions. In contrast, a binocular advantage was evident in performance accuracy. Specifically, participants picked up the wrong object twice as often during monocular viewing when the complexity of the environment increased. Reach performance was more efficient during binocular viewing, which was demonstrated by shorter reach reaction time and overall movement time. Reaching movements during the approach phase had higher peak velocity during binocular viewing. During monocular viewing reach trajectories exhibited a direction bias during the acceleration phase, which was leftward during left eye viewing and rightward during right eye viewing. This bias can be explained by the presence of esophoria in the covered eye. The grasping interval was also extended by ∼20% during monocular viewing; however, the duration of the return phase after the target was picked up was comparable across viewing conditions. In conclusion, binocular vision provides important input for planning and execution of prehension movements in visually enriched environments. Binocular advantage was evident, regardless of set size or target saliency, indicating that adults plan their movements more cautiously during monocular viewing, even in relatively simple environments with a highly salient target. Nevertheless, in visually-normal adults monocular input provides sufficient information to engage in online control to correct the initial errors in movement planning.

**Keywords: reaching and grasping movements, eye-hand coordination, binocular vision, visual search, phoria**

# **INTRODUCTION**

Vision provides important sensory input during performance of upper limb movements, such as reaching and grasping for objects or when using tools (Jeannerod et al., 1995; Elliott et al., 2001, 2010; Goodale and Westwood, 2004). Even seemingly simple motor behaviors require several stages of information processing involving a complex interaction between the cognitive, perceptual, sensory, and motor systems. For example, the act of picking up one's favorite coffee mug located among other mugs can be characterized by at least 3 stages of information processing: (1) visual search to find the mug, (2) localization of the mug in three dimensional space in order to plan reaching and grasping, and (3) online control during movement execution. Historically, the inquiry into these components of information processing has been conducted separately (Hayhoe and Rothkoph, 2011). However, acting in the real world depends on the coordinated interaction among the perceptual, sensory, and motor systems. Thus, the main goal of our research is to examine goal-directed movements performed in visually stimulating, three-dimensional (3D) environments. Since optimal movement control requires depth perception, the specific aim of the current study was to examine the contribution of binocular vision during execution of prehension movements in a visually rich environment containing multiple objects.

Over the years, studies from different disciplines have examined the contribution of binocular vision to the performance of perceptual and motor tasks, for example, discrimination of camouflaged objects, object recognition, and upper limb prehension movements (Howard, 2012). Benefits associated with two frontally placed eyes with overlapping visual fields can arise from two separate mechanisms: binocular summation (i.e., the similarities between the images) and binocular disparity (i.e., the differences in the retinal images between the two eyes) (Howard and Rogers, 2002). It has been shown that binocular summation is an important mechanism that contributes to more efficient performance of complex motor tasks, such as bead threading and water pouring (Jones and Lee, 1981). The second mechanism that can contribute to a binocular advantage is binocular disparity, which is the basis of stereopsis and provides information about relative depth and object structure/shape. Several studies used dichoptic viewing to examine the contribution of binocular disparity to object recognition and scene processing (Edelman and Bulthoff, 1992; Bennett and Vuong, 2006; Burke, 2006). Overall, results are in agreement and show reduced errors and shorter reaction time when objects are presented stereoscopically rather than on a flat, two-dimensional (2D) surface. In addition, this advantage seems to be greater when subjects are asked to recognize objects presented from a different viewpoint. In natural environments objects are often seen from different viewpoints and might be partially occluded by other objects; thus, binocular vision might facilitate the recognition of the target object and reduce visual search time.

The first set of studies into the role of binocular vision during prehension movements were conducted by Servos and Goodale (Servos et al., 1992; Servos and Goodale, 1994). They showed that removal of binocular vision resulted in a longer latency to initiate the movement, lower peak velocity, longer movement time, especially in the deceleration phase, and smaller peak aperture. It was concluded that binocular vision provides important sensory input for both reach planning and execution. It is important to note that binocular viewing does not always provide a significant advantage during motor task performance. For example, Coull and colleagues (Coull et al., 2000) found that the kinematics of aiming movements were comparable during monocular and binocular viewing. However, binocular advantage was found in a task where the localization difficulty was increased by varying target position on a trial-by-trial basis, and the opportunity to use online or terminal feedback was also eliminated by removing the target from view upon movement initiation. Although the authors did not examine the source of the aiming errors in the monocular condition, it is possible that subjects mislocalized targets due to phoria (i.e., the deviation of the covered eye). Previous studies with visually-normal people have shown that phoria has a significant effect on direction judgments (Ono and Gonda, 1978; Ono and Weber, 1981; Park and Shebilske, 1991), thus, it would not be surprising that aiming movements executed without visual or tactile feedback exhibit phoria-related errors. On the other hand, experiments where visual feedback is provided during movement execution found no significant end-point errors (Niechwiej-Szwedo et al., 2012). Collectively, these studies indicate that the planning errors due to phoria during monocular viewing must be corrected using online feedback. To our knowledge, no previous studies have considered the temporal dynamics of this correction process. Thus, one of the aims of our study is to examine the effect of phoria on trajectory corrections during our prehension task.

Over the last 20 years, research from different laboratories has extended the initial findings and showed that binocular viewing provides a greater advantage in more complex environments, for example, when multiple objects are present (Jackson et al., 1991), when reaching unfamiliar/novel objects (Marotta and Goodale, 2001; Keefe and Watt, 2009), or when online corrections are required (Bradshaw and Elliott, 2003; Hu and Knill, 2011). Furthermore, programming of the grasping component of a prehension movement is disrupted to a greater extent in comparison to the transport phase during monocular viewing (Watt and Bradshaw, 2000; Melmoth and Grant, 2006). In short, the literature indicates that the benefits of binocular vision during planning and execution of prehension movements may be greater in visually-rich environments, and thus, it is important to investigate the significance of binocular vision using naturalistic paradigms.

Most everyday prehension movements are performed in cluttered environments; however, only few researchers have examined prehension toward targets presented among other objects (Mon-Williams et al., 2001; Biegstraaten et al., 2003; Tresilian et al., 2005; Verheij et al., 2014). When participants were asked to reach for a block of wood with an obstacle placed at various distances from the target (3, 6, 9 cm), the influence of the obstacle depended on the target-to-obstacle distance (Mon-Williams et al., 2001). Specifically, when the obstacle was placed closer to the target, participants' reaching movements had reduced velocity and smaller peak grip aperture. In contrast, obstacles located 9 cm away from the target had no effect on reach kinematics. The authors concluded that placing obstacles near the desired object affects how a person will reach for that desired object (e.g., placement of the finger between obstacle and desired target). A recent study by Verheij and colleagues demonstrated that obstacles placed underneath the movement path seem to have little effect on the kinematics compared to those that are to the side of the desired object.(Verheij et al., 2014) Therefore, obstacles change the kinematics of reaching and grasping, but the effect is dependent on the location of the obstacles.

Natural goal-directed movements are performed in a variety of environments ranging from relatively simple (i.e., a single coffee mug on a table) to complex (i.e., coffee mug placed among other objects on a table). In the second case, the observer must find the target object, while filtering out irrelevant information. This process is referred to as visual search and requires attentional resources (Eckstein, 2011; Eimer, 2014). The level of difficulty in a visual search task has been manipulated using 2D displays of various complexities. Two factors have been shown to influence the efficiency of visual search: target saliency and the number of stimuli presented in the display. Searching for a salient target defined by a unique feature is referred to as "pop-out" search, because this type of target is easily detected even in displays that contain multiple items. In contrast, searching for a target that shares features with the distractors, such as color or shape, is called "conjunction" search. This task is more difficult and the time to find the target depends on the number of items in the display.

Most natural behaviors require visual search, that is, finding and localizing the target is necessary for the subsequent planning of goal directed movements. Furthermore, eye movements are crucial for guiding upper limb manipulation actions in 3D environments. However, there are only a few studies that examined prehension movements in visually rich environments containing multiple objects, and none of these studies examined the contribution of binocular vision. Our study was conducted to examine the contribution of binocular vision during a prehension task in the context of a visual search paradigm. To manipulate the difficulty of the visual search we manipulated the set size (i.e., the target was among 2 or 5 distractors) and target salience. Specifically, subjects were asked to reach toward a target defined by single, salient feature—color (i.e., pop-out target) or toward a conjunction target, which had the same color as the distractors. To further increase the difficulty of the visual search, we also introduced a condition where the target was presented with a salient, red-colored distractor. It was hypothesized that binocular viewing would facilitate visual search and provide more reliable cues for reach planning and execution in comparison to monocular viewing. In particular, we expected that during binocular viewing participants will demonstrate: (1) more efficient search pattern characterized by fewer gaze shifts; (2) faster reach reaction time; (3) higher peak velocity and shorter movement time. We also hypothesized that the advantage associated with binocular viewing will be most evident in the larger set size and when target's salience is reduced.

# **METHODOLOGY**

#### **PARTICIPANTS**

Fifteen healthy, right- handed adults (age: mean = 22.1 ± 4.6 years; 10 males) participated. Handedness was determined using the Edinburgh Handedness Inventory. One volunteer was excluded because he was left-handed. All participants had selfreported normal or corrected-to-normal vision and no history of visual or ocular problems. Stereoacuity was assessed using the Randot SO-002 test, and all participants had stereoacutiy of ≤50 s of arc. All volunteers who were screened for stereoacuity achieved at least 50 s of arc and no one was excluded. Eye dominance was determined using Dolman's "hole-in-card" test. The study was approved by the Research Ethics Board at the University of Waterloo and all protocols adhered to the guidelines of the Declaration of Helsinki. Informed consent was obtained from each participant.

## **APPARATUS**

The 3D visual environment consisted of cylindrical pegs (height: 4.0 cm, diameter: 1.2, 1.6, 2.0 cm), which were arranged on a 24 flat screen LCD monitor (Dell Professional P2312H, 1920 X 1020 @ 60 Hz). The LCD monitor was positioned horizontally and securely clamped to the table. The center of the monitor was aligned with participant's midline. The LCD display was controlled by DataPixx (VPixx Technologies, Montreal, Canada) and a VPixx script was used to randomize the placement of the pegs on the display on each trial (schematic diagram of the workspace is shown in **Figure 1**).

Upper limb reach kinematics were recorded with the Optotrak 3D Investigator motion capture system (Northern Digital, Waterloo, Canada) at a sampling rate of 250 Hz (spatial accuracy 0.4 mm, resolution 0.01 mm). Infrared markers were affixed to the tip of the index finger and the thumb of participant's right hand. A head-mounted EyeLink II (SR Research, Mississauga, Canada) eyetracker was used to record eye position at a sampling rate of 250 Hz (spatial accuracy 0.5◦; RMS resolution 0.01◦). The MotionMonitor software (Innovative Sports Technology, Chicago, USA) was used to synchronize the recording of eye and limb kinematics and to integrate the position data from the Optotrak and EyeLink into a common reference frame.

The Optotrak system was calibrated using a three-marker digitizing probe. A Cartesian coordinate system was used and defined relative to the workspace (i.e., the LCD display) used for placing the pegs. The origin was located at the left, bottom corner of the display (**Figure 1**). The three-dimensional system with respect to the observer was defined as follows: x-axis, horizontal plane (azimuth); y-axis, vertical plane (elevation); z-axis, median plane (depth). Calibration for the eye tracker was performed with binocular viewing using a standard 9-point grid. Validation was performed to ensure the reliability of the calibration was <1◦ error. Targets for eyetracker calibration were presented on a 19 CRT monitor (Viewsonic P95f+, 1600 × 1200 @ 87 Hz) at a viewing distance of 80 cm.

Participants were instructed to reach and grasp the target as quickly as

possible, and to place it on top of the criterion.

# **EXPERIMENTAL PROCEDURE**

Participants were seated at a table and executed prehension movements with their right arm. Each trial began with the participant's eyes closed and their index in a standard initial position, which was aligned with their midline. The initial posture for the thumb was not standardized, that is, participants placed their thumb in a position that was comfortable. While the participant's eyes were closed, the experimenter placed the pegs on the display in a unique grid that was provided by the VPixx script. Participants were instructed to open their eyes when they heard a warning tone and then to fixate on a fixation point presented on the LCD display for 120 ms. The fixation point was located 7.1 cm in front of their initial hand position. Subsequently, the fixation point disappeared and a criterion target was shown at the same location. The criterion was a circle which varied in diameter (1.2, 1.6, 2.0 cm). The participant's task was to find the peg that corresponded to the size of the criterion, to pick up the peg and place it on top of the criterion. On each trial the target peg was embedded among distractor pegs and there was only one peg that matched the criterion's size. Each participant was instructed to complete the task as quickly as possible.

The complexity of the visual environment was manipulated in two ways. First, the set size was either small (3 pegs) or large (6 pegs). Second, the salience of the target was either high (i.e., the target peg had a different color, which was red—highsalience condition) or low (i.e., target was the same color as the distractor—low-salience condition). On a small number of trials (8/56), a salient distractor (i.e., red-colored peg which was not the target) was presented on the display (invalid condition). The salient distractor was always presented in the same row as the target and at the farthest location from the target peg along the azimuth.

There were three viewing conditions, which were randomized in blocks among participants: (1) binocular, (2) monocular with right eye, and (3) monocular with left eye. During monocular viewing, an infrared (IR) long-pass filter was placed in front of the eye. The IR filter blocked visual input to the covered eye, but allowed the eye tracker to record its position. Each viewing condition consisted of 56 trials. On each trial the pegs were arranged in a different grid with two repetitions of each grid per viewing condition.

#### **DATA ANALYSIS**

First, reaching performance was quantified by calculating the number of errors (i.e., picking up the wrong peg). The frequency of errors was compared between each viewing condition using Pearson's Chi-square statistic. The effect of set size and target salience was also examined within each viewing condition using Pearson's Chi-square statistic.

Analysis of limb and gaze kinematic data was conducted offline using a custom-written Matlab script (Matlab, MathWorks, Natick, USA). Eye and hand position data were filtered using a second-order, dual-pass Butterworth filter with a cut-off frequency of 15 Hz for the hand data and 80 Hz for the eye data. Eye and reach velocity were obtained using a 2-point differentiation method using instantaneous velocity (adjacent data points) on the cumulative distance traveled (Hansen et al., 2008). The hand velocity data were used to obtain acceleration, also using 2-point differentiation. Reach initiation was identified when the velocity of the reach vector *exceeded* 0.02 m/s for 20 Consecutive milliseconds. The end of the movement was identified when the vector velocity *fell below* 0.1 m/s for 20 consecutive milliseconds (Elliott et al., 2006; Glazebrook et al., 2009). All trials were inspected visually to ensure that movement initiation and termination were identified correctly by the software. Movement kinematics presented in this paper (i.e., peak acceleration, peak velocity, duration of acceleration and deceleration phase) were calculated on position data obtained in the primary axis of the reaching movement (i.e., the z-axis). Due to a technical difficulty with a trigger, reach reaction time could not be obtained for 5 out of the 15 participants. Therefore, the analysis of 2 outcome measures (reach reaction time and primary gaze shift latency) is based on data obtained from 10 participants.

The total prehension movement consisted of 3 phases: (1) the reach approach phase (i.e., transport toward the target), (2) the grasping phase, and (3) the return phase. The approach phase, defined here as the interval from reach initiation to when the velocity *fell below* 0.1 m/s for at least 20 ms along the primary direction of movement (i.e., the z-axis), the grasping phase was defined as the interval from the end of the approach phase to when the velocity *exceeded* 0.02 m/s for at least 20 ms, and the return phase was defined as the interval from the end of the grasping phase to the end of movement (**Figure 2**).

Gaze shifts were detected using the eye velocity data. Data were plotted and examined visually to identify saccades, which were marked manually. Saccade initiation was identified when the velocity exceeded 20◦/s for a minimum duration of 20 ms. In the case of blinks, data obtained 100 ms before and after the pupil's occlusion were excluded from analysis. During binocular viewing the eye chosen for analysis was the one that provided the less noisy data. The seeing eye was used for eye movement analysis during monocular viewing.

Statistical analyses were performed using the SAS 9.2 software package. Descriptive statistics are reported as the mean and its corresponding standard deviation. All continuous measures were submitted to a repeated-measures ANOVA with 3 within-subject factors: Viewing Condition (binocular, monocular left eye, and monocular right), Set Size (set 3, set 6), and Target Salience (high, low, and invalid). Main effects and interactions were analyzed further using Tukey-Kramer *post-hoc* tests to adjust for multiple comparisons. The results section below reports the significant effects related to our hypotheses. The complete output of the analysis is presented in a Supplementary Table.

# **RESULTS**

## **PERFORMANCE ERRORS**

Participants picked up the wrong peg 4.2% of all trials. There were significantly fewer errors [χ(df <sup>=</sup> 2) = 6.6, *p* = 0.037] during binocular viewing (2.7%) in comparison to monocular viewing with the right eye (4.5%) or the left eye (5.3%). **Table 1** shows the distribution of errors across experimental conditions. These data clearly show that participants were more likely to pick up the wrong peg during monocular viewing when the target was embedded in a larger set size and when the target was more difficult to discriminate (low salience and invalid condition).

Since errors picking up the wrong peg were made on a relatively small percentage of trials, the analysis presented in the subsequent section is based on the kinematic data obtained in the correct trials.

#### **GAZE SHIFTS DURING THE ACQUISITION PHASE**

The participant's task was to discriminate the size of the criterion stimulus presented after fixation and to execute a reaching movement to pick up the peg whose diameter matched it. We defined the acquisition phase from the onset of the criterion to the initiation of the reaching movement. Information processing during this phase was quantified by examining the latency of the primary gaze shift and the number of scanning eye movements executed

**Table 1 | Frequency (%) and total number (***n***) of errors (picking up the wrong peg).**


*\*p* < *0.05 (obtained from Chi-square analysis within each viewing condition).*

prior to reach initiation. The latency of the primary gaze shift is indicative of the time that it takes to discriminate the size of the criterion, which will be referred to as the discrimination interval. There was no significant difference between viewing conditions for the duration of the discrimination interval. The latency of primary gaze shift was 328 ± 70 ms during binocular viewing, 350 ± 73 ms during right eye viewing, and 345 ± 73 ms during left eye viewing. The fixation duration before the initiation of the secondary gaze shift was not significantly different among viewing conditions (binocular: 170 ± 73 ms; right eye: 198 ± 99 ms; left eye: 185 ± 87 ms. Primary gaze shift latency was significantly longer in the larger set size [351 ± 78 ms vs. 332 ± 73 ms; *<sup>F</sup>*(1, 9) <sup>=</sup> <sup>7</sup>.87, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.021; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.64]. Fixation duration before initiation a secondary shift was also longer in the larger set size [192 ± 92 ms vs. 172 ± 100 ms; *F*(1, 14) = 18.31, *p* < 0.001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.72].

A secondary gaze shift prior to reach initiation was present on 22% of all trials. Chi-square analysis showed no significant difference between viewing conditions for the frequency of secondary gaze shifts [χ(*df* <sup>=</sup> 2) = 0.02, *ns*]. The effect of set size and target salience was also examined within each viewing condition. **Table 2** shows the frequency of secondary gaze shifts across experimental conditions. Across all viewing conditions, the frequency of a secondary gaze shifts increased when the target could not be easily discriminated (i.e., low salience and invalid condition), which was evident for displays with a larger set size in all viewing conditions, as well as during right eye monocular viewing in set size 3, invalid condition.

Tertiary gaze shifts were executed on 3.9% of all trials. The frequency of these gaze shifts was similar across viewing conditions [χ(*df* <sup>=</sup> 2) = 1.92, *ns*].

Temporal eye-hand coordination during the acquisition phase was examined by calculating the interval between the first gaze shift and reach initiation, which represents the time that was available for reach planning after the eyes were in the vicinity of the target. Analysis showed no significant difference between


**Table 2 | Frequency (%) of trials with a secondary gaze shift prior to reach initiation across viewing conditions.**

*\*p* < *0.05.*

viewing conditions for the saccade-to-reach initiation interval. Regardless of viewing condition, participants spent a longer time planning the reaching movement after the initial gaze shift when they were presented with a larger set size [*F*(1, 14) = 93.90, *p* < <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.93] and when the target couldn't be easily discriminated [*F*(2, 28) <sup>=</sup> <sup>3</sup>.80, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.037; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.21].

# **REACH AND GRASP PLANNING AND EXECUTION** *Temporal performance measures*

As illustrated in **Figure 3A**, mean reach reaction time was influenced by viewing condition, set size and target salience. The shortest response times were found during binocular viewing for most experimental conditions [*F*(2, 18) <sup>=</sup> <sup>3</sup>.58, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.049; <sup>η</sup><sup>2</sup> <sup>=</sup> 0.28]. Increasing the difficulty of the search task by increasing the number of distractors or by reducing target salience had a similar effect across viewing conditions. Specifically, reaction time was longer for set size 6 in comparison to set size 3 [*F*(1, 9) = <sup>88</sup>.00, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.95]. Similarly, reducing target salience resulted in longer reach reaction times [*F*(2, 18) = 4.05, *p* = 0.035; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.31].

The total movement time, which included the hand transport toward the target during the approach phase, the grasping phase and the return phase, was significantly shorter during binocular viewing [*F*(2, 28) <sup>=</sup> <sup>7</sup>.88, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.002; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.36; **Figure 3B**] and for the smaller set size [*F*(2, 28) <sup>=</sup> <sup>94</sup>.62, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.93]. However, the effect of cue or the interactions between viewing condition and set size or viewing condition and cue were not significant. Thus, the analysis of overall performance measures supports a significant binocular advantage for movement initiation and execution. The analysis presented next focused on determining the extent of this advantage during the approach phase, the grasping phase, and the return phase.

# *Approach phase*

Movement duration during the approach phase was not significantly different between the viewing conditions. However, movement time was significantly affected by set size [*F*(1, 14) = 48.24, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.87] and target saliency [*F*(2, 28) <sup>=</sup> <sup>12</sup>.65, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.47]. The interaction between set size and target salience was also significant [*F*(2, 28) <sup>=</sup> <sup>11</sup>.99, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.0002; <sup>η</sup><sup>2</sup> <sup>=</sup> 0.46]. *Post-hoc* tests revealed that movement times were significantly longer for the larger set size but only when a salient distractor was present.

**Figure 4** shows mean peak velocity during the approach phase. Peak velocity was significantly higher during binocular viewing [*F*(2, 28) <sup>=</sup> <sup>4</sup>.16, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.026; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.23] in comparison to monocular viewing with either eye. Peak velocities were also higher

salience (*p* < 0.05). **(B)** Mean total movement time across the experimental conditions. There was a significant main effect of viewing condition and set size (*p* < 0.05). Error bars show ±1 standard error of the mean.

in the smaller set size [*F*(1, 14) <sup>=</sup> <sup>38</sup>.03, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.84]. However, the main effect of cue or the interaction between viewing condition and set size or cue did not reach significance.

There were no significant differences between viewing conditions for the other reach kinematic measures during the approach phase: peak acceleration, duration of acceleration interval, peak deceleration or duration of deceleration interval. Peak acceleration and the duration of acceleration interval were influenced by set size. Specifically, peak acceleration was higher [*F*(1, 14) <sup>=</sup> <sup>19</sup>.00, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.0007; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.73] and the acceleration interval shorter [*F*(1, 14) <sup>=</sup> <sup>41</sup>.36, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.85] in the small set size condition. The acceleration interval was also shorter in the high salience target condition [*F*(2, 28) = 11.94, *p* = 0.0002; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.46].

Reach trajectory was examined by calculating the reach direction at peak velocity, during the acceleration phase (150, 100, and 50 ms before peak velocity) and during the deceleration phase (50, 100, and 150 ms after peak velocity). Reach direction was obtained using the finger position data along the azimuth and depth direction (reach direction = atan (x-position/z-position). In order to determine if monocular viewing introduces a bias in the reaching trajectory, we analyzed the data by subtracting the mean reach direction to each target location during binocular viewing from both monocular viewing conditions. The difference in reach direction between binocular and monocular

viewing was then analyzed. Results showed that reach direction was significantly influenced by viewing condition during the acceleration phase at 100 ms before peak velocity [*F*(1, 14) = 4.64, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.049; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.40], and 50 ms before peak velocity [*F*(1, 14) <sup>=</sup> <sup>7</sup>.98, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.014; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.53]. In contrast, reach direction was not reliably different between viewing conditions at peak velocity and during the deceleration phase. As shown in **Figure 5**, during the acceleration phase reaching trajectory had a leftward bias during left eye viewing and a rightward bias during right eye viewing.

#### *Grasping phase*

Grip aperture, defined as the separation between the finger and thumb, was examined at the initiation of the grasping phase. There was a significant effect of target size [*F*(2, 28) = 84.61, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.86] and set size [*F*(2, 28) <sup>=</sup> <sup>28</sup>.19, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.80]; however, there was no significant difference between viewing conditions, and the interaction between viewing condition and target size, set size, or cue was also not significant. Regardless of viewing condition, the mean grip aperture was smaller in set size 6 in comparison to set size 3 (11.9 ± 6.5 vs.19.5 ± 9.2 mm).

Data showed a significant effect of viewing condition for the duration of time spent in the grasping phase [*F*(2, 28) = 13.55, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.49]. The effect of set size was also significant [*F*(1, 14) <sup>=</sup> <sup>128</sup>.93, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.95]; however, the main effect of cue and the interaction did not reach significance. As shown in **Figure 6**, the grasping phase was prolonged during monocular viewing and in the larger set size condition.

Eye-hand coordination was assessed during grasping by calculating the interval between the end of the reaching movement and when gaze shifted away from the target. This interval represents the duration of time that participants spend fixating on the target as they were executing the grasping movement. There was a significant effect of viewing condition [*F*(2, 28) = 12.63, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.0001; <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.47]. Participants spent a shorter time fixating on the target prior to shifting their gaze during binocular viewing

(72 ± 177 ms) in comparison to monocular viewing (right eye 197 ± 232 ms; left eye 173 ± 194 ms). No other effects were significant.

## *Return phase*

There was no significant effect of viewing condition for the duration or reach kinematic during the return phase.

# **DISCUSSION**

To our knowledge, this is the first study that has examined the *role of binocular vision* during prehension movements in the context of a visual search paradigm. We chose this paradigm as most reaching movements performed in natural environments are not performed toward isolated objects. Rather, one of the major requirements associated with execution of goal-directed movements is the selection of relevant objects. Moreover, the demands associated with selection of behaviorally relevant sensory information can vary substantially depending on the number of objects in the environment and the similarity among these objects. Once the object is selected, successful reaching depends on the ability to plan and execute the movement while avoiding collisions with potential obstacles. Since viewing with both eyes has been associated with performance benefits during various perceptual and motor tasks, we hypothesized that binocular vision would provide significant performance advantages during the selection process, for movement planning and execution. The main findings from this study support our hypothesis and show that reach performance is more efficient during binocular viewing, which was demonstrated by fewer errors, shorter reach reaction time, higher peak velocity and shorter grasping interval. In contrast to our hypothesis, our data showed that the advantage associated with binocular viewing did not increase in the more complex environments.

Advantages associated with binocular viewing can arise at different stages of information processing and most likely have a cumulative effect contributing to overall improvement in the performance of fine motor skills. We used a complex experimental paradigm where participants had to match the size of a threedimensional target placed among distractors to a criterion, and then plan and execute a prehension movement. Our results showed no significant difference between viewing conditions for the latency of the primary gaze shift, which indicates that the time taken to process the visual information related to the criterion was not affected by viewing condition. This was not surprising because the criterion was a 2D shape presented at the same location as the initial fixation. However, participants made twice as many errors in picking up the wrong peg when viewing monocularly suggesting that the sensory information obtained from a 2D image was less reliable for finding a matching 3D shape, especially when the target had low salience and was embedded in a larger set size. These results are consistent with previous studies which found that subjects were significantly more accurate when asked to recognize objects presented stereoscopically in comparison to objects presented on two-dimensional displays (Edelman and Bulthoff, 1992; Bennett and Vuong, 2006; Burke, 2006). Furthermore, these studies showed that the differences between viewing conditions were greater when objects were rotated and presented from different viewpoints.

Previous studies that examined visual search using complex two-dimensional displays found that target salience and set size influence search time (Henderson et al., 2009; Beck et al., 2010; Neider and Zelinksky, 2011). That is, search times increase significantly with increasing set size and when the target shares feature(s) with the surrounding objects. We used a well-known manipulation (pop-out vs. conjunction targets) to increase the demands placed on visual processing during the selection process. Our data are consistent with the literature on visual search. We found that the frequency of gaze shifts and the time to reach initiation were both influenced by set size and target salience; however, there was no significant difference between viewing conditions. These results are in agreement with a recent study by Pomplun and colleagues (Pomplun et al., 2013). They examined visual search in a virtual 3D environment and showed that binocular disparity did not influence the search strategy. On the other hand, previous research has shown that binocular vision improves performance on tasks that require detection/discrimination of camouflaged objects (Jones and Lee, 1981; Schneider and Morglia, 1994; McKee et al., 1997). Thus, binocular advantage during search tasks is most likely dependent on the complexity of the display and would be expected in environments where objects are more difficult to discriminate. We used a visually enriched environment with multiple objects that varied in saliency, but our environment may not have been complex enough to show a binocular advantage during visual search.

Although we found no significant differences between viewing conditions in gaze behavior during the search process, the fact that reach reaction times were longer during monocular viewing indicates that binocular viewing facilitates the acquisition of sensory information for planning prehension movements in visually stimulating environments. Binocular summation is one factor that may explain faster reaction time when viewing with both eyes. Studies that used electroencephalography reported shorter latency of visual evoked potentials during binocular viewing as compared to monocular viewing (Woodman et al., 1990). Reduced latency at the first stage of information processing most likely extends to further stages of processing which include target identification and movement planning. Binocular viewing in 3D environments also activates disparity sensitive neurons in the primary visual cortex which project preferentially to parietal regions involved in reach planning and execution (Sakata et al., 1997; Fattori et al., 2004; Hajidimitrakis et al., 2011). Thus, binocular viewing might activate a more direct cortical path for planning reaching and grasping movements. Similar conclusions were also reached by a recent study which found a significant priming effect during binocular, but not during monocular viewing (Makris et al., 2013). The authors used an elegant paradigm where subjects were trained to respond with either a precision or power grip depending on the frequency of an auditory stimulus. During the test subjects were shown a real object and they were asked to respond by indicating the appropriate grip (i.e., precision or power), which was either congruent or incongruent with the priming auditory stimulus. Responses were significantly faster (20–35 ms) for congruent trials during binocular viewing, whereas a priming effect was not elicited during monocular viewing.

Most previous studies that consider the role of binocular vision during reaching and grasping have examined upper limb kinematics toward targets presented in isolation (Servos et al., 1992; Servos and Goodale, 1994; Watt and Bradshaw, 2000; Bradshaw et al., 2004; Melmoth and Grant, 2006; Keefe and Watt, 2009). The only exception is the study by Jackson et al. which examined movement kinematics in a prehension task when the target was presented with a single flanker object (Jackson et al., 1991). These authors demonstrated that the advantage associated with binocular viewing (i.e., shorter deceleration phase and smaller grip aperture) was greater in the presence of a flanker. The current study was conducted to extend the previous literature by examining prehension movements in more complex visual environments. Reaching in environments that contain multiple objects is more difficult because it requires the ability to plan a trajectory to pick up the target while avoiding the obstacles around it. Several studies have shown that reaching movements in cluttered environments are slower, have lower peak velocities, and larger grip aperture (Jackson et al., 1995; Tresilian, 1998; Mon-Williams et al., 2001; Biegstraaten et al., 2003). Binocular vision could facilitate the planning of an optimal trajectory path and online corrections in a cluttered environment. In particular, stereopsis provides unparalleled resolution of relative depth, which provides critical information about the target's shape and orientation, as well as its spatial relation with respect to the obstacles. During binocular viewing stereopsis can be combined with ocular vergence to provide the central nervous system with more accurate and more precise absolute depth information. Studies have also shown that binocular viewing provides an advantage during reach execution by facilitating online corrections (Bradshaw and Elliott, 2003; Hu and Knill, 2011). The need for online control might be increased when reaching in the presence of multiple objects because errors can arise due to mislocalization of a target in relation to the obstacles. Thus, it was expected that binocular vision would provide a greater benefit in an environment with a larger set size. Instead, our results showed that viewing condition and set size had an independent effect on prehension kinematics. Specifically, binocular vision and smaller set size were both associated with a shorter reaction time, higher peak velocity during the approach phase, and shorter total movement time. The finding that a binocular advantage was evident in both set size conditions indicates that monocular viewing provides less reliable information for planning and execution of prehension movements, even in a relatively simple environment with a target surrounded by two obstacles.

The current study provides a novel insight on the effect of monocular viewing on reach planning. Our results show that reach trajectory was biased during monocular viewing during the approach phase. Specifically, the initial direction of the reach trajectory (i.e., during the acceleration phase) was biased toward the left during left eye viewing, and toward the right during right eye viewing. This bias is consistent with the presence of esophoria during monocular viewing. Most visually-normal people experience a phoria (i.e., eye deviation) in the occluded eye, which can vary in direction and extent (Hrynchak et al., 2010). Esophoria is present when the occluded eye deviates medially and exophoria is present when the occluded eye shifts temporally. Phoria has been shown to affect the apparent direction of visual targets during monocular viewing (Park and Shebilske, 1991), during changes in accommodative vergence (Ono and Gonda, 1978), as well as during pointing tasks performed without visual feedback (Ono and Weber, 1981). Specifically, when the occluded left eye deviates medially (i.e., esophoria), the target's perceived direction will shift to the right. On the other hand, if the occluded left eye deviates temporally (i.e., exophoria), the target's apparent direction will shift to the left. Our data showed a bias in initial reaching direction that is consistent with the presence of esophoria during monocular viewing. One caveat in our current work is that the direction and extent of the phoria were not assessed in each participant. Despite this limitation, our results are in agreement with previous studies showing that target location is not perceived veridically during monocular viewing. The mislocalization of the target has a significant effect on motor planning, which is less accurate during monocular viewing. Importantly, the bias in initial trajectory was corrected shortly around the time of peak velocity which indicates that early online control was used to amend the initial reach plan. Importantly, there was no significant difference in movement time during the approach phase between viewing conditions which indicates that monocular viewing provided sufficient information to guide these early online trajectory corrections.

Previous prehension studies have shown that viewing with one eye leads to a greater grip aperture (Watt and Bradshaw, 2000; Bradshaw et al., 2004; Melmoth and Grant, 2006; Melmoth et al., 2009). In addition, Melmoth and Grant conducted a detailed analysis of the grasping phase and reported significantly greater number of errors, including larger apertures at object contact time and more adjustments of the grip aperture during monocular viewing (Melmoth and Grant, 2006). Consistent with previous literature, we found that monocular viewing had the largest effect on the grasping component of prehension, which was extended by ∼20% during monocular viewing in comparison to binocular viewing. This was also accompanied by a longer fixation on the target. It is possible that monocular viewing provides less reliable cues about the object's shape or structure, and subjects might have to rely to haptic feedback to a greater extent once they contact the object. On the other hand, object features are extracted more reliably during binocular viewing which facilitates the planning and execution of reaching and grasping. Keefe and colleagues provided an alternative explanation for increased grip aperture during monocular viewing (Keefe et al., 2011). These authors used an elegant experimental paradigm to manipulate the reliability of binocular and monocular depth cues and showed that grip apertures increased when either the binocular or the monocular cues were less reliable. Optimal performance (i.e., greatest precision of size estimates and lowest grip aperture) was found when both, binocular and monocular, depth cues were available. Results from that study support that the CNS integrates multiple cues for grasp programming; however, the authors did not examine the duration of the grasping phase or gaze behavior, thus, the question that remains outstanding is whether *visuohaptic integration* is affected differentially by the reliability of depth cues during monocular and binocular viewing.

Finally, in contrast to previous studies, our results showed no reliable differences between viewing conditions for grip aperture at the initiation of the grasping phase. Instead, regardless of viewing condition, grip aperture was smaller when there were more objects in the workspace. These data are consistent with previous studies that examined reaching and grasping when obstacles are present in a workspace. For example, Mon-Williams and colleagues examined grip aperture when subjects reached toward a target presented in isolation, and with one or two obstacles (Mon-Williams et al., 2001). Their results showed that in comparison to target only condition, grip aperture decreased by 10% when one obstacle was present, and by 20% when two obstacles were present in the workspace. Importantly, the extent of the reduction was dependent on the placement of the obstacles. Our study extends the previous literature by showing that, regardless of viewing condition, subjects adopt a cautious strategy in cluttered environments by reducing their grip aperture in order to reduce the possibility of a collision.

In conclusion, we examined prehension movements in a visually rich environment where the target was embedded among distractors and reaching the target required avoiding obstacles. We found that binocular vision provides advantages during information acquisition and for reach planning and grasp execution. Furthermore, the benefit associated with binocular viewing is consistent across environments of various complexities. Overall, this study provides an important contribution to our understanding of the role of binocular vision in movement control in complex environments. This knowledge is important for developing a comprehensive neural model of motor control, and ultimately, for establishing appropriate visuomotor training protocols for people with abnormal binocular vision.

#### **ACKNOWLEDGMENTS**

Supported by grants from the Canadian Foundation for Innovation (CFI), the Ontario Research Fund, and the Lyle Hallman Undergraduate Fellowship.

# **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnhum. 2014.00959/abstract

#### **REFERENCES**


Howard, I. P., and Rogers, B. (2002). *Seeing in Depth.* Toronto, ON: Porteus.

Hrynchak, P. K., Herriot, C., and Irving, E. L. (2010). Comparison of alternate cover test reliability at near in non-strabismus between experienced and novice examiners. *Ophthalmic. Physiol. Opt.* 30, 304–309. doi: 10.1111/j.1475- 1313.2010.00723.x


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 11 September 2014; accepted: 11 November 2014; published online: 28 November 2014.*

*Citation: Gnanaseelan R, Gonzalez DA and Niechwiej-Szwedo E (2014) Binocular advantage for prehension movements performed in visually enriched environments requiring visual search. Front. Hum. Neurosci. 8:959. doi: 10.3389/fnhum.2014.00959 This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Gnanaseelan, Gonzalez and Niechwiej-Szwedo. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# An allocentric exception confirms an egocentric rule: a comment on Taghizadeh and Gail (2014)

#### *Paul Dassonville1 \*, Benjamin D. Lester <sup>2</sup> and Scott A. Reed1*

*<sup>1</sup> Department of Psychology and Institute of Neuroscience, University of Oregon, Eugene, OR, USA*

*<sup>2</sup> Department of Neurology, University of Iowa Hospitals and Clinics, Iowa City, IA, USA*

*\*Correspondence: prd@uoregon.edu*

#### *Edited by:*

*J. Douglas Crawford, York University, Canada*

#### *Reviewed by:*

*Thomas Schenk, Universitätsklinikum Erlangen, Germany David Peter Carey, Bangor University, UK*

**Keywords: visual perception, illusion, egocentric frame of reference, allocentric frame of reference, motor control**

### **A commentary on**

**Spatial task context makes short-latency reaches prone to induced Roelofs illusion** *by Taghizadeh, B., and Gail, A. (2014). Front. Hum. Neurosci. 8:673. doi: 10.3389/ fnhum.2014.00673*

When a visual cue is presented in the context of a large rectangle shifted laterally from an observer's midline, its location is perceived to be shifted in the opposite direction (a phenomenon known as the induced Roelofs effect, a variant of an illusion first discovered by Roelofs, 1936). However, movements made immediately to the cue are accurate (Bridgeman et al., 1997; Dassonville and Bala, 2004a). We have shown evidence that the perceptual effect of the illusion is brought about by a distortion of the observer's egocentric reference frame—the offset rectangle attracts the observer's subjective straight-ahead (SSA), causing the cue to appear to be shifted in the opposite direction. However, if an action aimed at the cue is then guided within this same distorted egocentric reference frame, the error of motor guidance will cancel with the error of perceptual encoding, allowing the movement to be accurate (Dassonville and Bala, 2004a,b; Dassonville et al., 2004). We have begun to refer to this cancelation of errors, which allows for accurate actions in spite of the illusion, as the Two-Wrongs model, since, according to the model, two wrongs *do* make a right (Dassonville and Reed, under review). However, in a recent exploration of the induced Roelofs effect (IRE) on allocentrically-guided movements, Taghizadeh and Gail (2014) purport to show evidence against the Two-Wrongs model. A closer examination, though, reveals flaws in their assumptions, leading us to conclude that the Two-Wrongs model is, in fact, completely supported by their data. Here, we critically assess each of the three pieces of evidence used to argue against the Two-Wrongs model.

In Experiment II of Taghizadeh and Gail (2014), participants were first shown a reference array of possible cue locations (positioned to the left or right of the mid-sagittal plane), followed by a cue presented within a Roelofs-inducing rectangle. Participants were required to note the location of the cue within the previouslypresented reference array, and then point to the same allocentric location in a subsequent decision array. In certain critical trial types, the authors found errors that were in the opposite direction of those typically seen with the IRE. Based on their assumption that a distortion of the egocentric reference frame could only cause an error in the direction opposite that of the inducing rectangle, the authors concluded that the illusion must not be caused by such a distortion. However, their assumption is patently incorrect, since they fail to account for the initial influence of the *reference array itself* on the SSA. After all, there is nothing special about the typical Roelofs-inducing rectangle, other than its lateralized location—any lateralized stimulus would be expected to induce a similar distortion (e.g., Wapner et al., 1953; Walter and Dassonville, 2006; Lester and Dassonville, 2013), although its magnitude might be modulated by salience, attention, etc. (Lester and Dassonville, 2011). In the paradigm of Taghizadeh and Gail, when the reference array appears in the left hemifield, its presence would cause the SSA to be pulled to the left (**Figure 1A**), and the perceived location of the array would be encoded within this distorted reference frame. When the large inducing rectangle is later presented, it would exert its own influence on the SSA, but, since it is not as lateralized as the reference array, it would drag the SSA (and the memory of the reference array) *rightward* from where it had been at the time of the reference array presentation (**Figure 1B**). Accordingly, a cue presented at the center of the reference array would be reported as being to the left of center in the remembered array, even though the absolute position of the inducing rectangle was to the participant's left. Thus, an account of the IRE based on a distorted egocentric reference frame fully predicts that the resulting errors will depend on the relative displacement of the SSA between the occurrence of the reference array and cue/rectangle, not the rectangle's absolute position in space.

The authors also argue that the illusion's effect on immediate movements in their paradigm provides evidence against the Two-Wrongs model. However, the model specifically predicts that accurate movements will occur *only* when they are aimed at the *egocentric* location of the cue (Dassonville and Bala, 2004a; Dassonville et al., 2004). In contrast, the task of Taghizadeh and Gail required participants to guide their response to the *allocentric* location of the cue, and therefore the cancelation of errors described by

**FIGURE 1 | Distortions of the SSA in Experiment II of Taghizadeh and Gail (2014), according to the predictions of the Two-Wrongs model of visual processing. (A)** The initial step of stimulus presentation, showing the central fixation point (small square), and a reference array shifted to the left of the participant's mid-sagittal plane (solid gray line). Like the offset rectangle in the typical IRE, the offset reference array would serve to attract the participant's SSA (dashed gray line). **(B)** The subsequent appearance of the cue and Roelofs-inducing rectangle, after the reference array has been extinguished. In spite of the inducing rectangle's bias toward the left hemifield, its center of gravity is not as lateralized as that of the earlier reference array. This would cause the SSA to move rightward, pulling the memory of the reference array with it, and causing the cue to be mislocalized toward the left end of the remembered reference array. Although the image shows the SSA as moving directly between the distorted positions caused by the sequential presentation of the reference array and the later Roelofs-inducing rectangle, this change in the SSA need not be direct (for example, the SSA may drift back toward the objective midline during the delay period between reference array and inducing rectangle presentations, only to be pulled leftward again when the inducing rectangle is presented; see Dassonville and Bala, 2004a). Importantly, the direction and magnitude of the IRE would depend only on the relative locations of the distorted SSA during reference array and target presentations, regardless of its possible meanderings between those events.

the Two-Wrongs model would not occur. Given this, the data of Taghizadeh and Gail do not provide evidence against the Two-Wrongs model, but instead confirm the model's prediction that accurate movements will only occur in the face of the IRE when they are aimed at the cue's location within an egocentric reference frame.

Finally, the authors argue against an egocentric account of the IRE by pointing to their analysis that seems to suggest that the presence or absence of a fixation point has no effect on the illusion, claiming that a fixation point should provide an anchor that would stabilize the reference frame and eliminate the illusion. We agree that a fixation point *could* have a stabilizing effect, but there is no *a priori* reason to expect that it *must*, especially since the illusion is modulated by the salience of, and amount of attention directed toward, the inducing stimulus (Lester and Dassonville, 2011). Thus, it could have been anticipated that the effects of the large, salient inducing rectangle would largely overcome any stabilizing effects of the small fixation point or dimly lit laboratory.

Contrary to the conclusions of Taghizadeh and Gail (2014), their results are fully compatible with the hypothesis that the IRE is caused by a distortion in the observer's egocentric reference frame. Moreover, they provide confirmatory evidence for the Two-Wrongs model and its prediction that movements made in the context of the illusion will be accurate only when they are guided within the same distorted egocentric frame that is used to encode the cue's location (Dassonville and Bala, 2004a; Dassonville et al., 2004; Dassonville and Reed, under review).

# **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 17 September 2014; accepted: 04 November 2014; published online: 02 December 2014.*

*Citation: Dassonville P, Lester BD and Reed SA (2014) An allocentric exception confirms an egocentric rule: a comment on Taghizadeh and Gail (2014). Front. Hum. Neurosci. 8:942. doi: 10.3389/fnhum.2014.00942 This article was submitted to the journal Frontiers in*

*Human Neuroscience.*

*Copyright © 2014 Dassonville, Lester and Reed. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# The development of trunk control and its relation to reaching in infancy: a longitudinal study

#### *Jaya Rachwani <sup>1</sup> \*, Victor Santamaria1, Sandra L. Saavedra2 and Marjorie H. Woollacott <sup>1</sup>*

*<sup>1</sup> Human Physiology and Institute of Neuroscience, University of Oregon, Eugene, OR, USA*

*<sup>2</sup> Department of Rehabilitation Sciences, University of Hartford, West Hartford, CT, USA*

#### *Edited by:*

*Gavin Buckingham, Heriot-Watt University, UK*

#### *Reviewed by:*

*Klaus Libertus, University of Pittsburgh, USA Karen E. Adolph, New York University, USA*

#### *\*Correspondence:*

*Jaya Rachwani, Department of Human Physiology and Institute of Neuroscience, University of Oregon, 122 Esslinger Hall, Eugene, OR 97403, USA e-mail: rachwani@uoregon.edu*

The development of reaching is crucially dependent on the progressive control of the trunk, yet their interrelation has not been addressed in detail. Previous studies on seated reaching evaluated infants during fully supported or unsupported conditions; however, trunk control is progressively developed, starting from the cervical/thoracic followed by the lumbar/pelvic regions for the acquisition of independent sitting. Providing external trunk support at different levels to test the effects of controlling the upper and lower regions of the trunk on reaching provides insight into the mechanisms by which trunk control impacts reaching in infants. Ten healthy infants were recruited at 2.5 months of age and tested longitudinally, until 8 months. During the reaching test, infants were placed in an upright seated position and an adjustable support device provided trunk fixation at pelvic and thoracic levels. Kinematic and electromyographic data were collected. Results showed that prior to independent sitting, postural instability was higher when infants were provided with pelvic compared to thoracic support. Associated reaches were more circuitous, less smooth and less efficient. In response to the instability, there was increased postural muscle activity and arm muscle co-activation. Differences between levels of support were not observed once infants acquired independent sitting. These results suggest that trunk control is acquired in a segmental sequence across the development of upright sitting, and it is tightly correlated with reaching performance.

**Keywords: infant development, independent sitting, posture, trunk control, reaching, EMG**

# **INTRODUCTION**

Sitting postural control and reaching are distinguishable yet interrelated motor milestones, which are progressively acquired during the first years of life. When tasks require reaching while sitting, a simple reach toward an object involves complex interaction of musculoskeletal and neural systems to optimize the movement. Moreover, the emergence of posture and reaching skills is critical to subsequent perceptual, cognitive and social development (Sommerville et al., 2005; Soska et al., 2010; Lobo and Galloway, 2012).

The relation between posture and arm movements is evident in neonates. When newborns are fully supported, either in a reclined or upright sitting position, their usually chaotic arm movements are more coordinated and directed, indicating that pre-reaching movements are influenced by posture (Von Hofsten, 1982; Amiel-Tison and Grenier, 1983). However, newborns may not actually want to access the toy but rather pre-reaching movements may function to orient the infants' attention to the goal (Von Hofsten, 1982; Campos et al., 2008). In addition, the hands also attract considerable amount of attention and newborns will work to keep their hands in view (Van der Meer, 1997).

Beginning at 3 months, arm extensions are gradually replaced by goal directed reaches that are mainly unsuccessful in grasping the object. Grasping is typically achieved at the age of 4 months (Van der Fits et al., 1999a; De Graaf-Peters et al., 2007); however, arm movements are jerky with non-linear trajectories and have many movement units (defined as the number of accelerations and decelerations within the velocity profile of the reach (Von Hofsten, 1991). From this age there is an improvement in reaching kinematics and 6 month-old infants develop a straight arm trajectory accompanied by fewer movement units (Von Hofsten, 1991). During this phase of reaching development, there are many factors that influence arm trajectory, including visual perception, neuromuscular forces, biomechanical factors and proprioceptive information. However, the development of postural control for maintaining stability during reaching is indispensable (Bertenthal and Von Hofsten, 1998).

In this regard, 4 month old infants show a functional preference for stabilizing the head while reaching by first activating the neck muscles followed by trunk muscles. Control of the head enables infants to maintain stable vision of the target while reaching (Thelen and Spencer, 1998). Adults use a combination of strategies to attenuate head movement during dynamic tasks (Assaiante and Amblard, 1995; Keshner et al., 1999). However, infants must learn to coordinate head stability with arm movements. At 2 months of age head movements and arm movements are highly coupled (Von Hofsten, 1984; Von Hofsten and Rönnqvist, 1993). From 2 through 4 months of age there is an increased uncoupling of head and arm, allowing more flexibility of eye, head and hand coordination. This uncoupling of head and arm is important for environmental exploration (Hadders-Algra, 2008) and is a precursor of successful reaching (De Graaf-Peters et al., 2007; Van Balen et al., 2012).

Reaching and exploratory behaviors also depend on biomechanical and gravitational forces. Lying supine or prone limits manual exploration whereas sitting creates an advantageous setting for exploring objects (Out et al., 1998; Soska and Adolph, 2014). Within a sitting posture, the inability to sit independently reduces the amount of time the infant invests in exploring the toy because infants need their hands for stability (Harbourne et al., 2013). Nevertheless, when non-sitters are provided with pelvic support, reaching coordination and arm kinematics are significantly improved (Rochat and Goubet, 1995; Hopkins and Rönnqvist, 2002).

In summary, because reaching requires "whole body engagement" (Rochat and Goubet, 1995), its behavior is highly dependent on posture. At about 3 months, when arm extensions are being replaced by goal directed reaches but upright sitting is not mastered, infant reaching is better with external support. As infants generate the ability to sit independently, reaching becomes more coordinated. These observations are also evident in children with cerebral palsy who have deficits in postural control (Van der Heide et al., 2005). Although the progression of postural control is integral to development of reaching, the nature of this interrelation is unknown.

Postural control develops following a cranial-caudal progression, starting with head stabilization on the trunk, occurring at about 2 or 3 months of age. This provides a stable frame of reference for reaching (Assaiante, 1998; Thelen and Spencer, 1998). Control of shoulder and thoracic musculature around 4–5 months enables infants to maintain stability and counteract the reactive forces generated by the forward extension of the arm to successfully reach (Hopkins and Rönnqvist, 2002). As infants gain increasing control of the head and upper trunk, they progress from prop sitting to sitting without support (Harbourne et al., 2013). Subsequently, the control of the lower trunk, pelvis and leg muscles allows them to maintain the center of mass within a stable base of support (Von Hofsten and Woollacott, 1989; Assaiante, 1998; Van der Fits et al., 1999a; Harbourne et al., 2013). Thus, there is a cephalo-caudal development of control of an increasing number of trunk segments for sitting (Butler et al., 2010; Saavedra et al., 2012; Rachwani et al., 2013). However, previous studies have used supine (Van der Fits et al., 1999a; De Graaf-Peters et al., 2007), fully supported (Thelen et al., 1996; Thelen and Spencer, 1998; Van Balen et al., 2012) or unsupported (Van der Fits et al., 1999b; Harbourne et al., 2013) sitting conditions and thus have not examined the effect of trunk support on reaching in the upright position. Here we apply a systematic approach to examine the influence of segmental progression of trunk control on reaching.

In a previous cross-sectional study, we tested upright sitting conditions with trunk support at thoracic and pelvic levels to address contributions of higher and lower regions of the trunk to reaching (Rachwani et al., 2013). We showed that in non-sitters, postural and reaching kinematics depended on the external level of trunk support provided. With thoracic support, postural stability and reaching in non-sitters and sitters did not differ. On the contrary, with pelvic support, sitters outperformed non-sitters. Thus, reaching movement coordination depends on the extent of sitting control. However, the cross-sectional design of the study limited explanation of the causal effects of sitting posture on reaching. Cross sectional studies do not inform us about the mechanisms and trajectory of change and thus results cannot be translated into rehabilitation efforts.

To address this knowledge gap we applied the same experimental paradigm in a longitudinal design, examining intra-individual behavioral and kinematic changes of posture and reaching in conjunction with electromyography (EMG) recordings. As in our previous study we provided two levels of external support during an upright seated reaching task. With the use of video-coding software, we differentiated goal-directed, successful reaches from early pre-reaching movements. Quality of motor performance was assessed with kinematic variables including postural sway and reaching characteristics. Better reaches are more direct (i.e., a straighter reach), smoother (i.e., fewer movement units), more efficient (i.e., lower jerk score), and less reliant on on-line feedback for movement correction (i.e., the peak velocity of the reach occurs closer to the end of the reach) (Wu et al., 2000; Berthier and Keen, 2006).

Postural neuromuscular patterns for reaching include both anticipatory and compensatory adjustments. The role of anticipatory adjustments is to produce a preparatory muscular contraction to stabilize the body in advance (Aruin and Latash, 1995). Compensatory adjustments restore stability after a postural perturbation has occurred (Macpherson et al., 1989). To understand the mechanisms of change for infants, we recorded EMG from postural and arm muscles and documented anticipatory and compensatory activations.

We hypothesized that prior to independent sitting: (1) infants would demonstrate postural instability while reaching with pelvic support compared to thoracic support; (2) postural instability would be accompanied by inefficient and inaccurate reaching; and (3) frequency of anticipatory and compensatory postural reactions would increase as infants acquired independent sitting (Van der Fits et al., 1999a,b) but during pre-sitting stages, activation would be greater with more challenging postural conditions. As infants acquired independent sitting, we expected them to demonstrate invariable reaching and neuromuscular patterns irrespective of the level of support.

# **MATERIAL AND METHODS PARTICIPANTS**

Eleven infants were recruited for this study, 1 dropped out after the first session, and 10 infants completed the full protocol. All infants were born at term (5 males and 5 females) and had no known sensory or motor problems. All parents were fluent in English and most of the parents attended ongoing community based parenting groups. Infants began the study at a mean age of 2.5 months (± SD: 0.5 months) and were tested twice a month until the age of 8 months. Infants participated in 10– 12 sessions depending on age at entry to the study. If an infant missed an appointment or was fussy they were asked to make up that appointment the following week. Thus, most appointments were 2 weeks apart; however some were 1 or 3 weeks apart. The recruitment was carried out by using flyers in different child care centers in Eugene and Springfield (Oregon, USA). All procedures were approved by the Institutional Review Board for Human Subjects Research at the University of Oregon.

# **MATERIALS AND PROCEDURES**

Subjects were asked to come to the laboratory for 120 min sessions. At the first visit, parents were asked to respond to a health questionnaire about their infant, they were informed about the experimental procedure and were asked to sign the informed consent. During each visit, in addition to the reaching test, infants were clinically tested with the Segmental Assessment of Trunk Control (SATCo; Butler et al., 2010) to determine the level of intrinsic trunk control acquired, the Alberta Infant Motor Scale (AIMS; Piper and Darrah, 1994) and the motor subscales of the Bayley Scales of Infant and Toddler Development, 3rd edition (Bayley, 2005) to verify the typical trajectory of gross and fine motor functions. All infants were video recorded during each assessment. The AIMS test was used to determine the onset of independent sitting. The first two data sets when the infant was able to sit without arm support (item 8 from the sitting subscale of the AIMS test) served as the reference point for the developmental timeline. In this item, a specific duration of sitting without arm support was not needed but instead infants were required to demonstrate the ability to be left alone in the sitting position and to move their arms freely or play with a toy. Each infant's data were adjusted to this reference point (time in months of sitting onset = 0). Comparisons were made during the months prior to and after the month of sitting onset. In addition, parents were asked to do the Timed Sitting test twice per week at home to corroborate the onset of independent sitting ability. In this test parents placed the child in sitting with legs in front and timed how long they could stay upright with hands free. **Table 1** shows the clinical scores of all subjects (collapsed across 2 sessions at each month with respect to sitting onset).

## *Segmental Assessment of Trunk Control (SATCo)*

The SATCo is a clinical measure that examines balance control of the trunk while the evaluator manually supports the trunk at various levels, following a top-down sequence. The evaluator starts by supporting the trunk at a high level, at the shoulder girdle to assess cervical (head) control, through support at the axillae (upper thoracic control), inferior scapula (mid-thoracic control), lower ribs (lower thoracic control), below ribs (upper lumbar control), pelvis (lower lumbar control), and finally, no support, in order to measure full trunk control. During each level of manual support, the test is designed to assess: (1) static control (maintaining a neutral trunk posture) (2) active or anticipatory control (maintaining a neutral posture during head turning or reaching) and (3) reactive control (maintaining or regaining trunk control following a threat to balance, produced by a brisk nudge). The infant's ability to maintain or quickly regain a vertical position of the free region of the trunk in all planes during the assessment of static, active and reactive testing is scored as present or absent. The score reflects the region where infants lose control of posture: a score of 1 = loss of control at the head level, 2 = upper thoracic, 3 = mid-thoracic, 4 = lower thoracic, 5 = upper lumbar, 6 = lower lumbar, 7 = pelvis, 8 = no loss of trunk control (Butler et al., 2010). Thus, the SATCo follows a Guttman scaling, meaning that if an infant has a SATCo score of 4, he/she loses control of posture in static, active or reactive tests when the evaluator supports the lower thoracic region of the trunk but does not lose control of posture when being supported at the levels above that region. This test has been shown to be a valid and reliable measure of the development of trunk control in infants (Butler et al., 2010).

#### *Reaching test*

The reaching test was conducted with support at thoracic and pelvic levels for every session. The support at the thoracic level was placed below the scapular girdle, and the pelvic level of support was surrounding the pelvis, corresponding to the middle thoracic level and lower lumbar level of the SATCo, respectively. The design of the study was counterbalanced for the first session and was evaluated using the same order throughout the longitudinal process for each infant, with half the infants first being provided with thoracic support, and half first being provided with pelvic support, to eliminate fatigue or training effects as confounding variables.


The reaching test involved the infant being placed in a seated position on a customized infant chair. The hips of the infant were secured to the chair with specially designed straps and Velcro: two straps were used to wrap each hip joint and the third surrounded both posterior superior iliac spines (Butler et al., 2010). A rigid U-shaped posterior support attached to the back of the chair circled the trunk and provided upright stability of the trunk below the level of interest. The reclined position of the infant chair was used as a safety device in the backwards direction, for securing the infants if they fell backwards. The posterior support was adjusted to allow evaluation of different trunk segments: thoracic and pelvic (**Figure 1**).

Once posture was stabilized, a colorful toy (colored ring) was presented at approximately the infant's arm length in front of their sternum. The toy was presented using a device placed over the infant's chair that consists of a horizontal brace made of fiberglass with an attachment for the toy. This attachment permits the measurement of the distance from the toy to the chest (anteroposterior axis) and calibration of the height of the toy at the sternum level (vertical axis). Once the exact distance was measured, a toy attached to a rod was placed in the device and was introduced and removed by the tester from the top to the infant's visual field for every trial. The toy was presented approximately 10 times per level of support, but there were occasions in which this number had to be reduced due to fussiness of the infant. If that was the case, the infant's maximum number of trials was noted and the rest of the trials were counted as missing data.

The reaching test was synchronized with the collection of kinematic data (sampling rate = 84 Hz) using magnetic tracking (Minibird system, Ascension Technology, Burlington, VT) and with a 16-channel electromyography (EMG) system (MA300, Motion Lab Systems, Baton Rouge, LA), (sampling rate = 1000 Hz) and video data (sampling rate = 60 Hz).

## *Kinematics*

To document the quality of motor performance, four magnetic tracking sensors were placed on the infant: one superficial to the styloid process of the radius on each wrist, one on the posterior and prominent part of the cervical vertebra 7 (c7) and one on a headband with the sensor centered on the forehead. These sensors

were used to track arm and head movements. Prior to starting the reaching test, the position of the left and right tragus, the medial/lateral and anterior/posterior points of the external support (pelvic or thoracic) and sternal notch were recorded. This allowed estimation of the location of the head center of mass using the center of the distance between the midpoint of the two tragus markers and the head sensor. The center of the trunk region being evaluated was estimated as the midpoint between the sternal notch and C7, and the center of the external support was calculated as the midpoint of the two vectors created by the anterior/posterior and medial/lateral markers of the external support. Position data of all four sensors were referenced to the center of the external trunk support.

# *Electromyography*

To determine the mechanisms used by infants to control movement, EMG was recorded via bipolar self-adhesive surface electrodes with poles placed 2–3 cm apart. EMG signals were preamplified (gain × 20), band-pass filtered (10–375 Hz), and then further amplified, sampled at a rate of 1000 Hz per channel, and time-synched with position data. Two dorsal muscle groups and three arm muscle groups were recorded bilaterally (paraspinal muscles at the thoracic spine (T7-8) and lumbar spine (L3-4), at the belly of anterior deltoid, triceps and biceps muscles) in addition to the heart beat (over the 7th intercostal space, below pectoralis major, and over the sternal angle), used during analysis to subtract any heart beat artifacts from the EMGs.

# **DATA REDUCTION AND ANALYSIS**

## *Video analysis*

The video recordings served three purposes. First, the video was used to differentiate between non-directed arm movements and visually guided intentional reaching movements toward the toy. Second, the video was used for classification of the behavior of the movements of the arm during toy presentation. Third, initiation and end of reach were visually analyzed using computerized video-coding software (www*.*datavyu*.*org) for further evaluation of the kinematic and EMG parameters. Movements were classified as either (1) pre-reaching movements, also called "spontaneous arm movements," i.e., oscillating movements of the extended arms or forward directed arm movements (Van der Fits et al., 1999a), (2) unsuccessful reaches: reaching movements not ending in toy contact, associated with a loss of stability and/or requiring support while reaching, and (3) successful reaches: reaching movements ending in toy contact or grasp (De Graaf-Peters et al., 2007). The following types of reaches were not included: (1) the infant initiated a reaching movement toward the toy and lost interest during the trajectory by stopping and looking away; (2) the infant hit the toy; (3) the infant reached with full trunk support, i.e., the infant leaned back against the infant seat prior to reaching; (4) the infant used compensatory strategies like reaching with the head or dragging the toy with the forearm.

All reaches were coded as unimanual or bimanual. We defined bimanual reaches as those in which we visually saw the infant touch the toy with both hands and which also had an onset time difference between both arms of less than 1000 ms. If infants began unimanually and then switched to the other arm before reaching the toy, for the kinematic data analysis, only one arm, considered as the dominant arm, was selected. This selection was the same for bimanual reaches. Arm dominance was determined based on the hand that manipulated the object once it was held.

It is not easy to distinctly determine the start of a goal-directed reaching movement in infants, because one cannot instruct them to start from a defined position or at a given time. Thus, the computerized video-coding program allowed us to determine the onset and offset of all reaches. A light emitting diode (LED), placed on the corner of the visual field, was used to synchronize video and kinematic data during each reaching trial. With this, we were able to select reaches within the trial test time. We defined the onset of a reach as the moment when the infant initiated a movement of the upper extremity toward the toy accompanied by a visual fixation of the target. The offset of the reach was determined when the infant intentionally touched the toy.

To evaluate inter-rater reliability, a second coder scored approximately 25% of the video data. Coders agreed 85.9% of the time on the occurrence of a reach, its type (pre-reach, unimanual or bimanual), κ = 0*.*87, and whether it was successful or unsuccessful, κ = 0*.*67. Intra-class correlation coefficient between primary and secondary coders for reach onset and offset times was above 0.90.

After video-coding all reaches, reaching onsets was verified and adjusted, if necessary, by using an interactive cursor display, by simultaneously plotting the XYZ resultant of velocity and position data of the corresponding wrist sensor with the time frame selected with the video. A minimum velocity profile immediately preceding the initiation of the reach, identified from the videocoding software, was then verified. All dependent variables were then calculated from the selected time duration of each reach sequence. Kinematic and EMG data were digitized for off-line analysis with custom MATLAB programs.

#### *Kinematic analysis*

Kinematic data were filtered with a zero-lag fourth-order low-pass Butterworth filter with a cut-off frequency of 6 Hz to smooth the data and avoid possible jerky movements registered during the reaching sequence. We examined the following variables for each reach: angular trunk displacement and variability of trunk angle during a reach, straightness score, number of movement units, normalized jerk score and time when peak velocity occurred.

The time when peak velocity occurred was calculated as a percentage of time between the onset and end of the reach. A movement unit was defined according to Grönqvist et al. (2011) as the portion of the arm movement between two velocity minima with a velocity peak that should be greater than 2.3 cm/s. If the difference between the highest minima of one movement unit and the peak velocity of another movement unit was less than 8 cm/s, they were considered as one movement unit. Straightness was determined by measuring the trajectory of a straight line from the beginning of the trial to the moment when the infant touched the toy, which is the shortest distance to the target, considered as the baseline path length with a value of one. The amount that the arm movements deviated from this trajectory was then determined as the proportional increase in trajectory compared to this baseline path. Using this method, values greater than one

meant a more devious arm movement (Von Hofsten, 1991). The smoothness of the reach was quantified by calculating a time and distance normalized jerk score measured in cm/ms3. Time and amplitude were used to normalize the jerk score to eliminate dramatic increases with movement time. The following formula was applied to calculate normalized jerk score,

$$\text{normalized jerk score} = \sqrt{\frac{1}{2} \cdot \int \left(r^{\prime\prime\prime}\right)^2 \text{dt} \cdot \left(t^5/l^2\right)}$$

where *r*-- is the third time derivative of position data, *t* is movement time, and *l* is movement amplitude (Chang et al., 2005).

In terms of postural control, the angular displacement of the trunk was calculated as the angular summation during a reach in the anterior-posterior and medio-lateral planes. The trunk angle was calculated using a vector between the trunk center and the center of external support with respect to the vertical axis. With this, we were able to calculate the standard deviation of the trunk angle during a reach in the anterior-posterior plane. An increase in angular displacement and variability indicates that posture is in disequilibrium.

# *EMG analysis*

A frequency domain and Welch's power analyses on randomly selected sessions of the raw EMG signal were used to identify the most appropriate range of EMG signal frequency across the different muscles. Once we identified the most common frequency range, a modified version of the protocol used by Spencer and Thelen (2000) was applied: band-pass filter with cut-off frequencies at 20 and 160 Hz, demean, full-wave rectification and BoxCar averaging with a windows size of 7 data points in order to remove high-frequency components. In addition to this filtering process, a customized algorithm was applied for identifying and subtracting the cardiac QRS-complex signal from each channel of raw EMG before rectification.

Because this study was a within-subject design, the approach used for normalization and identification of EMG bursts was done relative to baseline EMG. This accounts for changes in baseline EMG magnitude and noise within-trials and across conditions for individual participants (William and Adam, 2012). For this purpose, EMG integrals of 10 ms bins were calculated across each muscle signal. A continuous 3 s time window of EMG-baseline signal for each muscle across the entire session was identified and the average integrated EMG of a bin was obtained during this baseline time window ( *EMGBaseline*). Each EMG integral ( *EMGIntegral*) of a bin was then normalized relative to EMGbaseline bin, *EMGNorm.Integral* = *EMGIntegral*− *EMGBaseline EMGBaseline* where *EMGNorm.Integral* greater than 1 would indicate an increase in EMG activity and less than 1 would indicate inhibition of activity. Thus, for determining significant bursts onsets and offsets, we applied an automatic onset and offset selection: 8 consecutive bins had to have a normalized value of 1.5 or greater (for determining onsets) or smaller (for determining offsets), prior to or during a reach. An interval of 80 ms was used because this time has been shown to be the minimal delay in postural muscle reactions (Horak et al., 1997; Shumway-Cook and Woollacott, 2012).

EMG analysis was structured in two main temporal windows: anticipatory stage, the 500 ms prior to the reaching onset; and compensatory stage which was variable depending on the movement time of the reach (Bigongiari et al., 2011). In comparison to previous studies, we decided to use a larger window size for the pre-defined anticipatory stage because infants, especially during early development, could activate postural muscles well in advance of the reach onset. Frequency of muscle activation during the compensatory stage was calculated as the number of times the EMG signal was active after the reach onset (%EMGACTIVATION in the compensatory stage). Frequency of muscle activation during the anticipatory stage was calculated as the percentage of times the EMG signal initiated its activation within the 500 ms preceding the reach onset and when its offset occurred at or after the reach onset (%EMGACTIVATION in the anticipatory stage). For postural muscles, frequency of activation during the compensatory and anticipatory stages was calculated as the percentage of time in which either the thoracic or lumbar muscle was activated. Lastly, we calculated the co-activation rates of the agonist and antagonist muscles of the arm. This was determined as the percentage of trials in which biceps and triceps muscles were simultaneously active with an onset difference of less than 40 ms (Van der Heide et al., 2003).

# *Statistical analysis*

Mixed models, in comparison to traditional analysis that do averaging, provide much more flexibility by taking the full data set into account and allowing subjects to have missing time points. Therefore, SPSS 22.0 for Windows (SPSS Inc., Chicago, IL, USA) was used to perform a Generalized Linear Mixed Model (GLMM) analysis of the interrelation between reach outcomes across developmental time and levels of external trunk support. GLMM is an extension of the LMM which allows fitting binary outcomes in addition to continuous outcomes into the model. As fixed effects, we entered developmental time of sitting ability, level of external support (thoracic and pelvic) and also their interaction into the model. As random effects, we had intercepts for infants and for sessions within infants, accounting for by-infant variability and by-session-within-infant variability in overall reach outcomes. Visual inspection of residual plots did not reveal any obvious deviations from homoscedasticity and normality. *Post-Hoc* comparisons using GLMM provided the ability to obtain *post-hoc* pairwise comparisons of the estimated marginal means for different levels of the fixed factors, such as level of external support across developmental time. *P*-values were obtained from *post-hoc* analysis after applying Bonferroni's sequential adjustment procedure that accounted for the multiple comparisons of the model.

# **RESULTS**

A total of 1730 reaches met the selection criteria. Out of this number, 1587 reaches were successful and were pooled for further kinematic and EMG analysis. Reaching onset occurred between 3 and 4 months of age (*M* = 3.26) when infants were placed in the supine position and were able to successfully contact a graspable toy placed at midline. Sitting onset occurred between 4 and 8 months of age; mean age of sitting onset as defined by item 8 on the AIMS was 5.95 months (**Table 1**).

# **VALIDITY OF THE SATCo**

Pearson's product moment correlation coefficient showed high correlation of SATCo scores with: developmental time (*r* = 0*.*91), AIMS test (*r* = 0*.*86), Bayley Scales of Infant and Toddler Development test (*r* = 0*.*83) and age (*r* = 0*.*90). According to the SATCo test, all infants achieved head control at 3 months prior to sitting onset and eight out of the ten infants achieved complete trunk control at the time of sitting onset (**Figure 2**).

# **DIFFERENCES IN REACHING SUCCESS AND TYPE OF REACH**

Four months prior to independent sitting, we were able to examine 7 out of the 10 infants. All attempted to reach toward the toy with thoracic support. The number of attempts was small (*M* = 5 reaches per infant) and the majority were unsuccessful or were classified as pre-reaches (*M* = 3 unsuccessful reaches out of 7 trials). With pelvic support, only 3 out of the 7 infants attempted to reach toward the toy. Most infants could not balance with this level of support and were continuously falling backwards. One infant was unsuccessful during all attempts and the other two were unsuccessful 50% of the time. Thus, for further analysis, infant reaches corresponding to 4 months prior to sitting were not included, due to the limited number of reaching attempts that infants were able to make with the external support at pelvic level.

Then, 3 months prior to sitting, 9 out of the 10 infants attempted to reach with thoracic support, and 8 out of the 10 infants attempted to reach with pelvic support. Infants were still less successful in reaching the toy with pelvic (47% of the time) in comparison to thoracic support (67% of the times), *t*(152) = 2*.*04, *p* ≤ 0*.*05, *d* = 0*.*17. Two months prior to sitting, infants were successful in reaching during approximately all attempts (96%

**FIGURE 2 | Graph showing SATCo scores (1–8) across sitting developmental time for each infant.** Vertical dashed line represents time of sitting onset. The developmental time period prior to sitting onset, corresponds to SATCo scores 1 through 5 which was when infants were learning to control progressively the upper and lower trunk regions. Once they acquired the ability to sit independently, it corresponded to SATCo scores 6, 7, and 8, indicating that they had control of almost all trunk segments.

with thoracic support and 86% with mid-rib support) and it was not until 1 month prior to sitting when they were completely successful (100% for both levels of support; see **Table 2**).

The type of reach (bimanual vs. unimanual) was variable across developmental time and not related to the level of support provided.

In summary, through the use of video-coding analysis, we were able to clearly distinguish the time when goal-directed reaches started to appear. The number of successful attempts increased with sitting age, but this increase in reaching performance was earlier for thoracic support compared to pelvic support.

# **DIFFERENCES IN POSTURAL AND REACHING KINEMATICS ACROSS DEVELOPMENT**

Major differences in reach outcomes between levels of external support were observed during the months prior to sitting onset. The graphs from **Figure 3** are examples of a reach at the thoracic and pelvic level of support of an infant 3 months before and 1 month after sitting onset. A photographic image is shown of the infant reaching with each level of support at 3 months before sitting onset. The 3-dimensional visual representation of the arm trajectory shows how the infant displayed a more circuitous reach and was more unstable with pelvic support compared to thoracic support prior to the development of independent sitting ability, and this difference was not observed once this milestone was acquired.

These observations were further corroborated with the kinematic variables (**Figure 4**). With pelvic support, compared to thoracic support, infants showed an increase in angular trunk displacement at 3 months, *t*(94) = 1*.*96, *p* ≤ 0*.*05, *d* = 0*.*20, 2 months, *t*(256) = 3*.*78, *p <* 0*.*01, *d* = 0*.*24, and 1 month, *t*(310) = 3*.*41, *p <* 0*.*01, *d* = 0*.*19, prior to sitting and at the time of sitting onset, *t*(344) = 2*.*02, *p <* 0*.*05, *d* = 0*.*11. Variability of trunk angle was greater for pelvic support at 3 months, *t*(94) = 3*.*00, *p* ≤ 0*.*01, *d* = 0*.*33, 2 months, *t*(256) = 3*.*00, *p <* 0*.*01, *d* = 0*.*19, and 1 month, *t*(310) = 3*.*50, *p <* 0*.*01, *d* = 0*.*20, prior to sitting.

Reaching kinematics also showed differences between levels of support, being worse with pelvic support. With pelvic support infants showed an increase in: straightness score at 3 months, *t*(94) = 1*.*92, *p* ≤ 0*.*05, *d* = 0*.*20, 2 months, *t*(256) = 3*.*79, *p <* 0*.*01, *d* = 0*.*24, and 1 month, *t*(310) = 2*.*83, *p <* 0*.*01, *d* = 0*.*16, prior to sitting; in movement units at 2 months prior to sitting, *t*(256) = 2.32, *p <* 0*.*05, *d* = 0*.*15; and in normalized jerk score at 2 months prior to sitting, *t*(256) = 2.76, *p <* 0*.*01,

**Table 2 | Summary of trial data across development.**

*d* = 0*.*18. Time at which peak velocity occurred was shorter for pelvic support compared to thoracic support at 3 months, *t*(94) = −3*.*00, *p <* 0*.*01, *d* = 0*.*31, and 2 months, *t*(256) = −2*.*12, *p <* 0*.*05, *d* = 0*.*13, prior to sitting.

**FIGURE 3 | Graphs above showing examples of the 3D trajectory of a single reach from onset (circular shape) to offset (diamond shape), of one infant with thoracic and pelvic support during (A) 3 months prior to sitting onset and (B) 1 month after sitting onset.** Photographic images show infant reaching toward the toy with thoracic and pelvic support at 3 months prior to sitting onset. Arrows indicate location of kinematic sensors.


*Success rate is based on the average number of times infants were able to successfully touch the toy.*

These kinematic results describe the quality of the motor task and show that with pelvic support compared to thoracic support, maintaining stability of the trunk (measured by angular trunk displacement and variability of trunk angle) was more challenging for infants. However this was only during the period when infants had not yet acquired the ability to independently sit. During the same time period their reaching behavior was worse with pelvic support, as indicated by their straightness score, movement units, normalized jerk score and time to peak velocity.

# **DIFFERENCES IN POSTURAL AND ARM EMG** *Frequency of postural muscle activation*

Differences between levels of external support in frequency of activation of postural muscles were mainly observed during months prior to sitting onset. In general, postural muscles were more frequently activated when infants were supported at pelvic vs. thoracic level and this was not observed once infants acquired independent sitting ability (**Figure 5**).

In comparison to thoracic support, with the support at pelvic level, infants showed an increased frequency of activation of postural muscles during the compensatory stage at 2 months, *t*(246) = 2*.*03, *p <* 0*.*05, *d* = 0*.*13, and 1 month, *t*(310) = 2.69, *p <* 0*.*01, *d* = 0*.*16, prior to sitting. When sitting onset occurred, compensatory adjustments of postural muscles substantially increased with both levels of support.

Similar to the results obtained for frequency of compensatory adjustments, we found that anticipatory postural adjustments were also more often present with pelvic support compared to thoracic support at 2 months, *t*(246) = 2.01, *p <* 0*.*05, *d* = 0*.*13, and 1 month, *t*(310) = 4.19, *p <* 0*.*01, *d* = 0*.*24, prior to sitting. By the time sitting was achieved, the percentage of anticipatory adjustments had reached similar values for both levels of support. The average onset time of anticipatory adjustments was approximately −285 ms across sitting development, irrespective of support and developmental time.

## *Arm muscle activity*

Frequency of activation for the arm muscles was characterized as being highly variable between levels of support and across developmental time; however, in general, results showed that all arm

∗∗*p <* 0*.*01.

muscles were consistently active during a reach, biceps activity being the most predominant.

**(B)** anticipatory postural adjustment stage. X-axis displays

Co-activation rates for biceps-triceps activity were substantially higher for pelvic support compared to thoracic support 3 months prior to sitting, *t*(46) = 2.00, *p <* 0*.*05, *d* = 0*.*29 (**Figure 6**).

Overall, through the use of EMG we were able to document the mechanisms of change in seated reaching across development. As with kinematics, results indicate that pelvic support was more challenging than thoracic support during the period when infants have not yet acquired independent sitting. This was determined by the increase in activation frequency of postural muscles, during both anticipatory and compensatory stages, and increase in coactivation rates of the arm muscles at 3 months prior to sitting onset.

# **DISCUSSION**

This study was motivated by the hypothesis that the development of sitting postural control and reaching behavior are highly interdependent functions. Full attempts were made to tease out the causal effects of postural control on reaching. First, having applied measures across a broad range of ages in a longitudinal design, we were able to explore a critical window of postural development prior to independent sitting. Second, with the use of experimental manipulations, we had the means to model the type of postural control that infants progressively generated for themselves. We provided external support to the thoracic and pelvic regions of the trunk to compare the effects of increased vs. decreased postural support on reaching.

With support at the thoracic level, we confirm and expand previous results by showing that reaching movements during pre-sitting stages were smoother, more coordinated and more mature than when support was limited to the pelvic level (Rochat and Goubet, 1995; Hopkins and Rönnqvist, 2002; De Graaf-Peters et al., 2007; Rachwani et al., 2013). Increased postural support had a direct impact on reaching performance and neuromuscular responses of the trunk. As infants developed trunk control,

they no longer required higher support to produce coordinated reaching. The data indicate that postural control is a primary factor contributing to reaching proficiency, regardless of whether posture improves naturally across age or with the help of an experimental set-up (Adolph and Berger, 2005). This information creates the basis for future studies that can be applied in assessment and rehabilitative protocols in children with postural dysfunctions.

## **REACHING SUCCESS**

Infants at early stages of sitting development (4 months prior to sitting onset), showed minimal ability to remain stable in the sitting position when provided with thoracic support. This ability was completely absent in most infants with pelvic support, in which only 3 infants were able to maintain stability part of the time. Similar results were seen with respect to the number of reaching attempts that the infants made with the two levels of support. Thus, even though both motor milestones, upright sitting and reaching, were still immature during this developmental time period, better support of the trunk was associated with the ability to maintain stability and to perform more reaches, as has been observed in previous studies (Von Hofsten, 1982; Amiel-Tison and Grenier, 1983). This suggests that postural control significantly regulates the interaction of the infant with the surrounding environment during development, facilitating new actions, like reaching, which can promote the emergence of cognitive skills and social behaviors (Gibson, 1988).

According to the AIMS and SATCo scores, infants began to master head control at 3 months prior to sitting onset. With this mastery, infants increased their ability to touch/grasp the toy, highlighting the importance of head control for successful reaching (Thelen and Spencer, 1998). In order to lift the arm and successfully touch the toy infants must fixate the visual target, which requires both strength and control of the head in space. We hypothesized that reaching abilities would be reduced when postural stability was reduced (i.e., with pelvic support) and, as predicted, infants were more successful in reaching with thoracic compared to pelvic support. Then, at 2 and 1 month prior to sitting onset, the success rate was similar between conditions, despite the challenging postural demands derived from trunk support at the pelvic level. Harbourne et al. (2013) showed a similar effect in that non-independent sitters persistently and successfully reached in spite of subsequent falls, disorganized muscle onsets and erratic trunk movements.

#### **POSTURAL AND REACHING KINEMATICS**

The effect of external support on the control of posture while reaching was evident in that infants showed reduced stability with pelvic support when they had not yet mastered the ability to sit independently. Postural sway was quantified as total displacement of the trunk and angular variability while reaching. With thoracic support, posture was more controlled and subsequent reaching performance was better during pre-sitting stages. Reaching movements performed under the high support (thoracic) condition were straighter, (smaller straightness score), smoother (fewer movement units), more efficient (less jerk score) and used a more refined program (greater percentage of reach time when peak velocity occurs) than those performed under the low support (pelvic) condition.

The point in time at which differences between support levels disappeared depended on the kinematic variable measured. For instance, differences between support levels in smoothness, efficiency and programming of a reach were seen only at 2 months prior to sitting and then disappeared. This indicates that other factors may contribute to further kinematic improvement. The straightness of a reach on the contrary, was persistently affected by support level during all pre-sitting ages with infants generating more circuitous reaches with pelvic support. Being able to independently sit marked the hallmark for performing straight, linear reaches regardless of the support level. Taken together, these results suggest that the ability to produce efficient and accurate reaching in sitting is due to maturation of trunk control.

## **POSTURAL AND REACHING EMG PATTERNS**

On numerous occasions, researchers showed that postural muscle activity accompanying reaching movements increases with age (De Graaf-Peters et al., 2007; Van Balen et al., 2012; Harbourne et al., 2013). Results from the current study show that postural muscle activity can be present even in early stages of sitting development, but it is dependent on the constraints of the task. Compensatory postural muscle activity was more frequent when infants were provided with pelvic support 2 and 1 month prior to sitting onset. This implies that during sitting development, infants were able to recruit postural muscles while reaching, and to increase recruitment frequency when the postural task was more demanding. Thus, postural muscle recruitment was situation-specific and depended on the degree of instability (Hadders-Algra, 2008). Then, with increased age and maturation of sitting ability, the activation frequency of postural muscles increased and infants showed similar values between levels of trunk support, implying that pelvic support no longer produced instability. Previous research showed that once independent sitting was mastered, postural muscle activity accompanying reaching movements while sitting was consistently present (Van der Heide et al., 2003) and thus became embedded in the task, although it could be further enhanced if the risk of losing balance was high (Hadders-Algra, 2005; Van Balen et al., 2012).

Similarly, though anticipatory adjustments were just emerging in pre-sitting stages, infants displayed a higher percentage of anticipatory postural adjustments when they were provided with pelvic support compared to thoracic support, indicating they were anticipating the disequilibrium the reaching created when they had not yet acquired full trunk control. After the time of sitting onset, anticipatory adjustments were more consistently present (more than 50% of the time) and were independent of the type of support, suggesting this was related to the onset of independent sitting. The study by Van der Fits et al. (1999b) examining anticipatory postural adjustments under conditions of upright sitting concluded that anticipatory postural adjustments were present only inconsistently (20% of trials) at 6 months of age and became more regular at 13–14 months. These differing results might be explained by methodological differences related to the time period that was selected for analysis, because Van der Fits et al. (1999b) evaluated 200 ms prior to prime mover activation whereas in the current study we evaluated 500 ms prior to reach onset. To test our theory, we re-evaluated our data with a 200 ms window and found a 20% frequency of anticipatory postural adjustments across development and support level. The smaller window replicates the findings of Van der Fits et al. (1999b) and eliminates the ability to see the developmental pattern, thus demonstrating the critical importance of window size when making conclusions about developmental trajectories.

In short, during early developmental stages of sitting, anticipatory postural adjustments accompanying reaching movements are present to some degree, especially when the postural task is more demanding. However, they are characterized by immature temporal features. Anticipatory postural adjustments start to play a major role in the postural mechanisms for seated reaching once independent sitting has been established. At this point their activation is not dependent on the level of postural stability but they are consistently activated well in advance (approximately 285 ms prior to reach onset).

The activation of both agonist and antagonist muscles at a joint often occurs when the individual has lower skill levels because coactivation stiffens the entire limb. In this study the co-activation rates of biceps and triceps muscles were significantly enhanced with pelvic support during early stages of sitting control, which was also associated with the onset of reaching (Thelen et al., 1993). This outcome implies the need to maintain arm stability when seated conditions had increased postural requirements. Nevertheless, our findings related to arm muscles indicate that frequency of activation was highly consistent and was not dependent on the level of support. This could be explained by the following reasons: first, infant arm movements were seldom at rest and therefore arm muscles were often active even prior to the start of the reach. Second, because infants were in upright sitting conditions, they showed arm movements that were not related to the reach but were used as compensatory strategies to maintain balance. For these reasons, the starting point of the reach was not identical across trials despite the attempts made to avoid this.

To conclude, results reinforce and further expand previous findings showing that improvements in sitting control have direct consequences on the development of reaching. There is a craniocaudal acquisition of trunk control for independent sitting. The extent of sitting control acquired has an impact on the kinematic quality of reaching movements and accompanying postural muscle patterns, attributed to frequency of activation. However, with additional support, infants experience improvements in their reaching skills and subsequent muscular parameters during the development of upright sitting. Further research should examine differences in compensatory balance strategies and muscle response patterns used to recover from seated perturbations with different levels of trunk support. Moreover, the interrelation of reaching and sitting postural control should also be examined in children with cerebral palsy to determine if they might benefit from external trunk support and consequently implement more efficient therapeutic strategies. This paradigm offers the foundation for future exploration both in typical development and in children with neurological deficits.

# **ACKNOWLEDGMENTS**

This work was supported by the National Institutes of Health Grant 1R01HD062745-01, Marjorie Woollacott, principal investigator, and by the Eugene Evonuk Memorial Graduate Fellowship for Jaya Rachwani. The authors would like to thank Wayne Manselle for his valuable help with computer programming and technical assistance.

# **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnhum. 2015.00094/abstract

# **REFERENCES**

Adolph, K., and Berger, S. (2005). "Physical and motor development," in *Developmental Science,* eds M. Bornstein and M. Lamb (Mahwah, NJ: Lawrence Erlbaum Associates), 223–281.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 13 October 2014; accepted: 29 January 2015; published online: 24 February 2015.*

*Citation: Rachwani J, Santamaria V, Saavedra SL and Woollacott MH (2015) The development of trunk control and its relation to reaching in infancy: a longitudinal study. Front. Hum. Neurosci. 9:94. doi: 10.3389/fnhum.2015.00094*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2015 Rachwani, Santamaria, Saavedra and Woollacott. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Real-time vision, tactile cues, and visual form agnosia: removing haptic feedback from a "natural" grasping task induces pantomime-like grasps

Robert L. Whitwell 1,2,3\*, Tzvi Ganel <sup>4</sup> , Caitlin M. Byrne<sup>2</sup> and Melvyn A. Goodale2,3,5

<sup>1</sup> Graduate Program in Neuroscience, The University of Western Ontario, London, ON, Canada, <sup>2</sup> Department of Psychology, The University of Western Ontario, London, ON, Canada, <sup>3</sup> The Brain and Mind Institute, The University of Western Ontario, London, ON, Canada, <sup>4</sup> Department of Psychology, Ben-Gurion University of the Negev, Beer-Sheva, Israel, <sup>5</sup> Department of Physiology and Pharmacology, The University of Western Ontario, London, ON, Canada

Investigators study the kinematics of grasping movements (prehension) under a variety of conditions to probe visuomotor function in normal and brain-damaged individuals. "Natural" prehensile acts are directed at the goal object and are executed using realtime vision. Typically, they also entail the use of tactile, proprioceptive, and kinesthetic sources of haptic feedback about the object ("haptics-based object information") once contact with the object has been made. Natural and simulated (pantomimed) forms of prehension are thought to recruit different cortical structures: patient DF, who has visual form agnosia following bilateral damage to her temporal-occipital cortex, loses her ability to scale her grasp aperture to the size of targets ("grip scaling") when her prehensile movements are based on a memory of a target previewed 2 s before the cue to respond or when her grasps are directed towards a visible virtual target but she is denied haptics-based information about the target. In the first of two experiments, we show that when DF performs real-time pantomimed grasps towards a 7.5 cm displaced imagined copy of a visible object such that her fingers make contact with the surface of the table, her grip scaling is in fact quite normal. This finding suggests that realtime vision and terminal tactile feedback are sufficient to preserve DF's grip scaling slopes. In the second experiment, we examined an "unnatural" grasping task variant in which a tangible target (along with any proxy such as the surface of the table) is denied (i.e., no terminal tactile feedback). To do this, we used a mirror-apparatus to present virtual targets with and without a spatially coincident copy for the participants to grasp. We compared the grasp kinematics from trials with and without terminal tactile feedback to a real-time-pantomimed grasping task (one without tactile feedback) in which participants visualized a copy of the visible target as instructed in our laboratory in the past. Compared to natural grasps, removing tactile feedback increased RT, slowed the velocity of the reach, reduced in-flight grip aperture, increased the slopes relating grip aperture to target width, and reduced the final grip aperture (FGA). All of these effects were also observed in the real time-pantomime grasping task. These effects seem to be independent of those that arise from using the mirror in general as we also compared grasps directed towards virtual targets to those directed at real ones viewed directly

#### Edited by:

Simona Monaco, University of Trento, Italy

Reviewed by: Claudio Galletti, University of Bologna, Italy David A. Westwood, Dalhousie University, Canada

#### \*Correspondence:

Robert L. Whitwell, Graduate Program in Neuroscience, Department of Psychology, The Brain and Mind Institute, The University of Western Ontario, Room 245, London, ON N6A 5B7, Canada rwhitwel@gmail.com

> Received: 15 October 2014 Accepted: 02 April 2015 Published: 06 May 2015

#### Citation:

Whitwell RL, Ganel T, Byrne CM and Goodale MA (2015) Real-time vision, tactile cues, and visual form agnosia: removing haptic feedback from a "natural" grasping task induces pantomime-like grasps. Front. Hum. Neurosci. 9:216. doi: 10.3389/fnhum.2015.00216 through a pane of glass. These comparisons showed that the grasps directed at virtual targets increased grip aperture, slowed the velocity of the reach, and reduced the slopes relating grip aperture to the widths of the target. Thus, using the mirror has real consequences on grasp kinematics, reflecting the importance of task-relevant sources of online visual information for the programming and updating of natural prehensile movements. Taken together, these results provide compelling support for the view that removing terminal tactile feedback, even when the grasps are target-directed, induces a switch from real-time visual control towards one that depends more on visual perception and cognitive supervision. Providing terminal tactile feedback and real-time visual information can evidently keep the dorsal visuomotor system operating normally for prehensile acts.

Keywords: grasping, pantomime grasps, haptic feedback, visual feedback, visual form agnosia

# Introduction

Being able to reach out and grasp objects with considerable skill is one of the defining features of primates. The act itself typically involves the use of real-time visual information and is directed at a visible object. It also results in contact with the object, manipulation, and haptic feedback. Detailed analysis of movements of the fingers, hand, and wrist show that the posture and orientation of the moving hand reflect the geometric properties of the goal object (e.g., Jeannerod, 1988; Jakobson and Goodale, 1991; Paulignan et al., 1991a,b; Gentilucci et al., 1996). The visually-mediated control of grasping is thought to involve the dorsal stream of visuomotor pathways in the posterior parietal cortex (PPC) and their interconnections with premotor areas of the frontal lobe (for review see: Culham and Valyear, 2006; Grafton, 2010; Davare et al., 2011). In line with this view, disrupting the activity of the anterior areas of the intraparietal sulcus of the PPC with transcranial magnetic stimulation (TMS) affects the grasp kinematics in neurologically healthy individuals (e.g., Glover et al., 2005; Tunik et al., 2005; Rice et al., 2006, 2007). Furthermore, damage to dorsal-stream structures in the PPC can result in selective visuomotor deficits involving misreaching and/or poor grasp formation (Jeannerod, 1986; Perenin and Vighetto, 1988; Jakobson et al., 1991; Goodale et al., 1994a; Jeannerod et al., 1994; Binkofski et al., 1998; Milner et al., 2001; Karnath and Perenin, 2005; Cavina-Pratesi et al., 2010). Despite their deficits in real-time visuomotor control, however, some patients with dorsal-stream lesions show relatively preserved visual perceptual abilities on comparable tasks that require object form processing (Jakobson et al., 1991; Goodale et al., 1994a; Jeannerod et al., 1994; Milner et al., 2001).

In contrast to the effects of lesions to the dorsal stream, lesions that are largely restricted to the ventral stream often produce gross deficits in the ability to report the features of visual stimuli, such as color, visual texture, and form. A deficit in form vision is typically referred to as ''visual form agnosia'' (for review, see Goodale and Milner, 2013). One of the best known examples of such a patient is DF (Milner et al., 1991; for review see Whitwell et al., 2014b; but see also patients JS and MC; Wolf et al., 2008; Karnath et al., 2009, respectively). DF and other similar patients had sustained bilateral lesions of varying extent to occipitotemporal cortex and, as a result, were left with a persistent deficit in visual form perception. Nevertheless, when these patients reached out and grasped objects, that they failed to discriminate amongst, the online configuration of their grasping hand reflected the spatial and geometric properties of those objects (Goodale et al., 1991, 1994a,b; Marotta et al., 1997; Westwood et al., 2002; Wolf et al., 2008; Karnath et al., 2009; Whitwell et al., 2014a, in press). Their relatively normal performance is made all the more remarkable by the fact that these patients were all demonstrably at chance when asked to manually indicate the widths of exemplars from a set of socalled ''Efron blocks'' (Efron, 1969) placed directly in front of them. The Efron blocks vary in length and width but, critically, are matched for cues that these patients, including DF, can perceive such as weight, texture, color, and overall surface area. In other words, despite gross deficits in visual object perception, these patients were capable of making relatively normal-looking visually guided target-directed actions, such as reaching and grasping, presumably by virtue of having spared visuomotor networks in the dorsal stream. These studies, together with the complementary neuropsychological studies of patients with dorsal-stream lesions described above, as well as demonstrations of dissociations between perceptual report and visually guided actions in normally-sighted individuals, (e.g., Ganel et al., 2008; Stöttinger et al., 2010, 2012) have provided compelling support for the Two Visual Systems Hypothesis (TVSH; Goodale and Milner, 1992; Milner and Goodale, 2006), which in turn has influenced subsequent and expanded proposals on the functional organization of the primate visual system (Rizzolatti and Matelli, 2003; Kravitz et al., 2011, 2013).

In a seminal investigation, Goodale et al. (1994b) explored the dependence of the dorsal stream on real-time visual control by examining how normal DF's grasps looked when she was forced to rely on a memory of a recently previewed target. To do this, the authors compared natural grasps to a variant Milner et al. (2001) later-called ''delayed-pantomimed grasping'' (DPG) in which the participants, including DF, executed grasps to the remembered location of targets viewed as recently as 2 s before the cue to respond occurred. In this task, the participants' view of the workspace was restored following the delay period. Critically, however, the experimenter removed the object during the delay period and so it was no longer present when the participants were cued to reach out and pretend to pick up the remembered object ''as if it was still physically present'' (p. 1165). The DPG task therefore differed from the natural grasping task in two respects: (1) online visual input about the target was not available when the response was cued; and (2) no haptics-based object information was available at the end of the movement. The results showed that all of the participants, including DF, moved their hand towards the previewed location of the target. Nevertheless, there were some clear differences in the hand kinematics of the two grasping tasks. Compared to natural grasps, the DPGs of the participants, including DF, took longer to complete, exhibited slower peak hand velocities, and showed smaller anticipatory grip aperture. The measure on which DF's performance differed most-drastically from that of the controls was the in-flight, anticipatory adjustments in grip aperture to the widths of the remembered targets (grip scaling). Whereas the controls showed no change in their grip scaling slope (relating grip aperture to target width) moving from natural grasps to DPGs, DF's slope bore no relationship whatsoever to target width. Goodale et al. argued that DF's failure in the DPG task was due to her inability to form a visual percept of the target and extract its width. Their reasoning was based on two assumptions: (1) that the DPG task required participants to use a remembered percept of the target's width; and (2) that the creation of this percept required an intact object processor housed in the occipital-temporal cortex. Thus, their argument runs, DF's failure in grip scaling was a direct result of the damage to her ventral stream, preventing her from forming a visual percept in the first place to store in memory.

Importantly, Goodale et al. (1994b) also tested DF and the controls in an additional variant of the ''natural'' grasping task. In this new task, the participants, including DF, were presented with a visible Efron block and were asked to imagine an identical version of that object displaced to the right of it (7.5 cm), and then to reach out to grasp this imagined object ''as if it were physically present'' (Goodale et al. p. 1171–1172). Unlike the DPG task, this real-time displacedpantomime grasping (RPG) task allowed the participants a full view of the workspace throughout the trial which included the Efron block and the hand and limb. Thus, the availability of real-time visual input about the object was equivalent across the natural and the RPG tasks, even though the target-directedness of the two tasks along with the availability of haptics-based object information clearly differed. Nevertheless, the results showed that, compared to natural grasps, the RPGs took longer to complete, exhibited slower peak hand velocities, and showed smaller anticipatory grip apertures. Thus, regardless of whether the pantomime grasps of neurologically-intact individuals are planned using online or remembered visual information about the object, removing haptics-based object information slows the hand movement, increases the movement time, and reduces the overall grip aperture. Noting an increase in the variability in DF's anticipatory grip aperture for the RPG task, Goodale et al. ultimately concluded that both the DPG and RPG tasks produced catastrophic results for her grip scaling. Interestingly, however, in stark contrast to an absence of grip scaling in DF's DPGs, DF's grip aperture in the RPG task actually appears to be linearly related to the width of the target.

Common to both of Goodale et al.'s (1994b) pantomime tasks is an obvious requirement to pretend to pick up either the remembered or imagined target as if it were actually there and an absence of haptics-based object information. As we have already pointed out, the availability of real-time visual input following the cue to perform the grasp differed between the two tasks. Thus, this factor alone can reasonably account for any differences in DF's performance across the two pantomime grasping task variants. As such, DF's poor performance on the DPG task serves as a striking example of the dependence of some visuomotor tasks (pantomime grasps) on ventral stream processing, not only in DF but, presumably, in neurologically-intact individuals as well. One perhaps less obvious requirement of Goodale et al.'s tasks is the fact that the dimensions of the Efron blocks (only 1 cm in height) that were used in these experiments allowed the participants to receive tactile feedback from the surface of the table at the end of their reach. This was because the participants could not reasonably be expected to refrain from touching the surface of the table with their fingertips when simulating reaching out to pick up short rectangular blocks. Thus, the table may well act as a proxy when the grasps are directed next to the visible object. Importantly, haptics-based object information need not correlate with the visual size of targets for DF's grip scaling to be normal. Indeed, when the grasped object remains an intermediate size despite changes in the visual size from trial to trial, DF's grip aperture scales to the visual size (Whitwell et al., 2014a, in press). According to this view (see also Milner et al., 2012), both terminal tactile feedback and real-time visual input are critical for normal, dorsally-mediated prehension. Unfortunately, Goodale et al., did not compare DF's performance in the RPG task directly against the performance of the controls, presumably because there were differences between DF and the controls in terms of the stimulus set (six Efron blocks vs. three) and the presentation protocol (one target position vs. three). Determining whether DF's grip scaling in this task is in fact normal or abnormal would help rule out (or rule in) the importance of terminal tactile feedback for normal, real-time prehension. Therefore, in the first of our two experiments, we aimed to fill in this gap by revisiting DF's grip scaling in Goodale et al.'s RPG task. We tested a new group of control participants using the same stimulus set and protocol that was used by Goodale et al. to determine whether DF's real time-pantomime grasps were indeed as good as the controls and, more importantly, whether or not her grip scaling in this task would actually dissociate from that of her ''natural'' grasps as is commonly assumed.

# Experiment 1

# Methods Participants

Eight self-reported right-hand dominant age-appropriate and gender-matched control participants ranging from 31 to 46 years of age (M = 39.1, SD = 5.7), volunteered to take part in the experiment to compare DF's grip scaling in the natural grasping and RPG tasks. The controls provided written informed consent and were compensated \$20 for their time. All experiments were approved by the local ethics committee and were in accordance with the Declaration of Helsinki.

# Apparatus and Stimuli

Details of the apparatus and stimuli used to test the controls for patient DF's data set can be found in Goodale et al. (1994b). Briefly, the stimuli consisted of a set of Efron blocks that were 1 cm in height but varied in their lengths and widths as follows: l × w (in cm), 10 × 2.5, 8.3 × 3, 7.1 × 3.5, 6.3 × 4, 5.6 × 4.5, 5 × 5. The kinematic data was collected at 200 Hz using an optoelectronic recording system (OPTOTRAK 3020, Northern Digital, Waterloo, ON, Canada) that recorded the 3D spatial locations of three infrared emitting diodes (IREDs). The IREDs were attached with adhesive tape at three positions on the right (grasping) hand: the distal left corner of the nail of the index-finger, the distal right corner of the nail on the thumb, and the skin blanketing the metacarpophalangeal joint (MCP) of the index-finger. The experimenter ensured that the pads of skin on the tip of the thumb and index-finger were uncovered to ensure normal tactile feedback from the goal objects when grasped. The leads from the IREDs were taped to the right forearm to ensure complete freedom of movement. There was only one target position, 30 cm along a sagittal plane from the start position. The start position was a raised button located 5 cm from the edge of the table facing the participant (see **Figure 1**). Before the experiment began, the experimenter ensured that all of the participants were seated comfortably and positioned close enough to the table so that they could grasp the objects at the farthest position comfortably and without leaning forward.

# Procedure and Design

Details of the procedure and design used to test the controls can be found in Goodale et al. (1994b). Briefly, before each trial was initiated, the participants closed their eyes and held the tips of their right index-finger and thumb together while depressing the start button. The experimenter then gave a verbal prompt to the participant to open her eyes. The experimenter then waited approximately 2 s before giving a ''go'' signal for the participant to execute their response. For the natural grasping task, the participants were instructed to reach out, grasp across the width (near-far axis) of the Efron block, lift up, and put back down the Efron block using a precision grip (index-finger and thumb) as soon as they heard the go signal. At the beginning of the experiment, participants were asked to grasp the objects naturally: neither labored nor speeded. For the RPG task, the participants were instructed to imagine that the visible target to their left was positioned at the same distance along their midline (see **Figure 1**). They were further instructed to reach out to pick up the imaginary target as if it were physically there. The experimenter explained the procedure for the upcoming task before each block of trials. The experiment was comprised of 2 blocks of 36 trials each for a total of 72 trials. Each block

of trials was dedicated to a different task. The block of natural grasps were performed before the block of RPGs. As Goodale et al. (1994b) cautioned, this order was chosen to give DF the maximum likelihood of being able to use the experience of actually grasping the objects when performing the RPGs. The order of the blocks were the same across all of the participants, including DF. For each block of trials (i.e., for each task), each one of the six Efron blocks was presented 6 times in a pseudorandom order.

# Data Processing and Statistical Analysis

of the table at the end of the reach.

The data from the control participants were processed offline with custom software written in Matlab (Mathworks Inc., Natick, MA, USA). The positional information from the IREDs was lowpass filtered at 20 Hz using a 2nd order Butterworth digital filter. Grip aperture was computed as the Euclidean distance between the IRED placed on the thumb and the IRED placed on the index-finger, and the instantaneous velocities were computed for each of the three IREDs and for grip aperture. We analyzed three principal measures: peak grip aperture (PGA), the slope relating PGA to the target size, and the peak hand velocity (PHV). The PGA was defined as the largest grip aperture within a search window that was designed to capture the forward-reach component of the movement. The beginning of this window (the ''movement onset'') was operationally defined as the first of 30 consecutive sample frames (150 ms) in which the velocity of the MCP IRED exceeded a threshold of 50 mm/s. Normally, one could use the movement onset as a measure of reaction time. In this case, however, because the timing between the initiation of the data collection and the subsequent experimenter's verbal ''go'' command was free to vary (as was the case in Goodale et al., 1994b), reaction time (RT) could not be referenced to a fixed point in time. Thus, RT could not be computed reliably. Nevertheless, the end of the search window was defined as the first sample frame in which the velocity of the IRED fell below 75 mm/s. Linear regression of PGA on the widths of the Efron blocks was performed separately for each task and the resultant regression coefficient (i.e., the slope, b) relating the average increase in PGA (in mm) per incremental increase in Efron width (also in mm) was computed for DF and for the controls. The PHV was defined as the peak speed at which the MCP IRED travelled towards the target within the search window outlined above.

Notably, only DF's PGA was available from the data set reported by Goodale et al. (1994b). Thus, only PGA and the slopes relating PGA to target size could be compared against the control data set. The PHV of the control participants was analyzed to test Goodale et al.'s finding that the RPGs of the controls are executed more slowly than natural ones in this slightly modified version of that task (one target position and six target sizes). The comparisons of interest in the control data were the differences in the PHV, overall PGA, and grip scaling slopes between the natural grasps and the RPGs. The comparisons of interest that involved DF included those measures that were common to both DF and the controls: the difference in overall PGA between the natural grasps and RPGs and the grip scaling slopes. A comparison of the PGAs between DF and the controls for each of the natural grasping and RPG tasks was not carried out given that inter-individual differences in IRED positioning and hand anatomy could have yielded spurious results. Comparisons of intra-individual differences involving PGA should be far less susceptible to this influence (if at all). Accordingly, we used independent-samples t-tests to assess the normality/abnormality of (1) DF's slope on each of the two grasping tasks and (2) DF's difference scores for both the slope and the PGA between the two grasping tasks. Together, these contrasts constitute tests for ''strong/differential'' or ''classical'' dissociations (Crawford et al., 2003; Crawford and Garthwaite, 2005). For all statistical tests, the alpha criterion for statistical significance was set to 0.05.

# Results

# Peak Grip Aperture (PGA), Slopes, and the Peak Hand Velocity (PHV)

The controls' overall PGA was significantly larger when they performed natural grasps than when they performed RPGs, t(7) = 8.23, p < 8 × 10−<sup>5</sup> (see **Figure 2A**). A comparison of the difference in the overall PGA across the two tasks yielded no significant difference, t(7) = −0.02, p = 0.98. In other words, the switch from natural grasps to RPGs affected DF's overall PGA no differently than it did the controls' overall PGA.

The controls' slopes relating PGA to target size did not depend on whether they executed natural grasps or RPGs, t(7) = 1.29,

p = 0.24 (see **Figure 2B**). DF's PGA was positively related to the size of the target in the natural grasping task (t(28) = 6.01, p < 2 × 10−<sup>6</sup> ) and in the RPG task, t(28) = 2.98, p < 6 × 10−<sup>3</sup> . Importantly, DF's slopes did not differ significantly from those of the controls when she performed natural grasps (t(7) = −0.69, p = 0.61) or when she performed RPGs, t(7) = −1.53, p = 0.17. Moreover, the test for dissociation yielded a null result, t(7) = 0.5, p = 0.62 (see **Figure 2C**). In other words, DF's slopes fell within the normal range regardless of whether she performed natural grasps or RPGs. Notably, DF's slopes on the natural grasping task and the RPG task contrasts sharply with an absence of grip scaling on the DPG task in which her pantomimes were based on a memory of the previewed target (slope based on data reported in Goodale et al., 1994b) (p = 0.9; see **Figure 2B**).

The controls' PHV was significantly slower when performing the RPGs than it was when they performed natural grasps, t(7) = 2.79, p < 0.05.

Finally, the time-normalized grip aperture (**Figure 3A**) and velocity (**Figure 3B**) profiles for the controls reveals a noticeable distinction between the natural and displaced-pantomime grasps that converges with the findings of Goodale et al. (1994b).

# Discussion

In this experiment we re-examined DF's natural grasps and RPGs from an earlier study by Goodale et al. (1994b) by contrasting her performance on these two tasks with the performance of a new sample of normally-sighted control participants. When compared to natural grasps, the controls' RPGs yielded smaller overall PGAs and slower PHVs. Thus, we replicated Goodale et al.'s findings but in a version of the task that the authors had modified before testing DF by reducing the number of possible target positions from three to one and increasing the number of targets from three to six. Although we were unable to examine DF's PHV, we

FIGURE 3 | Normalized grand mean grip aperture and velocity profiles for the natural grasps (black) and real-time pantomime grasps. (A) Grip aperture normalized to 100 time bins for qualitative comparisons. Note that, overall, the real-time pantomime grasps lack a distinct peak and achieve lower grip aperture values than the natural grasps. The error bars reflect average within-participant standard deviations (B) Velocity of the wrist normalized to 100 time bins for qualitative comparisons. Note that, overall, the displaced-pantomime grasps appear to be executed more slowly than the natural grasps. The error bars reflect the between-participant standard deviations.

found that the RPG task reduced DF's overall PGA just as much as it did for the controls. We also examined DF's grip scaling in terms of the slope relating PGA to target size and for the controls. Somewhat surprisingly, we found that DF's slopes fell within the control range in both tasks. Her intact performance on this task contrasts sharply with her performance on the DPG task in which (quite unlike controls) she shows no evidence of grip scaling at all. As we pointed out in the Introduction, one evident difference between the two tasks is the availability of real-time visual input about the target in the RPG task. In other words, in the RPG task, information about the target can be used in real time to program the movement parameters, including grip aperture. This is obviously not the case in the DPG task. Indeed, because the movement is being programmed in real time in the RPG task, the relatively intact visuomotor networks in DF's dorsal stream could presumably mediate this programming. Although this line of argument is appealing, recent experiments suggest that real-time visual input is not sufficient for ''normal'' prehension (e.g., Bingham et al., 2007; Schenk, 2012a; Whitwell et al., 2014a,b).

Several years after Goodale et al.'s (1994b) investigation, Bingham et al. (2007) introduced a novel variant of a grasping task which was later adapted by Schenk (2012a) to re-test DF's grasps. Noting that movements that lack feedback are often more variable, Bingham et al. (2007) hypothesized that goal-directed movements, such as grasping, are precise because they can make use of haptic feedback (what we are referring to here as haptics-based object information) for calibrating each movement. Thus, Bingham et al. reasoned, the slower pantomime grasping movements that Goodale et al. (1994b) observed could be due to a decrease in precision and the lack of haptics-based object information in the DPG and RPG tasks. Bingham et al. set out to test how the provision of periodic haptic feedback about the target object would affect the grasps of normally-sighted individuals. To do this, Bingham et al. used an ingenious mirror apparatus that allowed the participants to view a virtual target in the mirror. This way, the participants could be instructed to reach out behind the mirror towards the apparent position of the virtual target to grasp it. An identical copy of the virtual target could be positioned behind the mirror such that the virtual and hidden targets were spatially coincident. Critically, the arrangement allowed the experimenter the choice to deny the participants an opportunity to grasp a real cylinder by refraining from positioning one behind the mirror. In short, this setup allowed Bingham et al. to preserve both the real-time visual information about the targets and the target-directedness of natural grasps in these new grasping task variants. Similar to Goodale et al.'s findings, Bingham et al. found that when participants were consistently denied an object to grasp, they showed slower hand velocities, longer movement times, and lower overall PGA.

Several years following Bingham et al.'s (2007) study, Schenk (2012a) used a similar mirror-apparatus to re-examine patient DF's grasps. He was motivated by the observation that the dissociation in grip scaling between DF's grasping and her explicit perceptual estimates of target size might be due to the difference in the availability of haptic feedback about the target between grasping and perceptual estimation tasks. As Bingham et al. has suggested, haptic feedback might normally be used to calibrate actions. Perhaps DF has developed some abnormal reliance on this source of information that allows her to calibrate the programming of her grasps (see also Schenk, 2012b). Rather than providing DF haptic feedback for her perceptual estimations of target size, however, Schenk opted to divorce it from the grasping task as Bingham et al. (2007) had done. Critically, he found that DF's grip scaling was abolished when haptic feedback was consistently denied and concluded that haptic feedback was required to calibrate DF's grasping movements. Curiously, however, he did not appeal to the same pantomime-based explanation as Bingham et al. and Goodale et al. (1994b) had done in the past. Instead, he argued that DF uses haptic feedback to ''compensate'' for her visual perceptual deficit when reaching out to grasp objects (Schenk, 2012a). According to this line of reasoning, no distinction between visual processes for perception and those for skilled goal-directed action is required, because DF's vision is merely degraded—haptics can help bootstrap her performance. As things turn out, this interpretation is incorrect, because DF's inaccurate perceptual estimates of Efron width show no improvement when haptic feedback is available to putatively calibrate her estimates: she was permitted to reach out to pick up the Efron blocks immediately after each of her explicit estimates (Whitwell et al., 2014a, in press). Thus, DF's dissociated performance on perceptual estimation and grasping tasks continues to support a fundamental distinction between dorsal and ventral stream object processing.

Nevertheless, one important factor was overlooked in both Schenk's and Bingham et al.'s experiments: the participants in the ''no haptic feedback'' tasks of both studies were unlikely to have encountered anything other than ''thin air'' at the end of their reaching movements. For example, in Schenk's study, the visual targets were vertically-standing cylinders 7 cm tall, requiring a horizontal grasping motion across the diameter of the visible target. In Bingham et al.'s investigation, the objects were shorter (though >3 cm in height), but the participants were explicitly instructed not to touch the surface of the table at the end of their reach and encouraged to adopt a particular approach that would minimize this possibility. At any rate, denying participants objects to grasp not only removed haptics-based object information in these studies but also any terminal tactile feedback about the end of the movement (Milner et al., 2012). This was not the case in Goodale et al.'s (1994b) study (and therefore in Experiment 1 of the present study) in which the participants, including DF, clearly made contact with the surface of the table next to the visible target. In fact, given DF's normal grip scaling, the results from Experiment 1 support an important distinction between hapticbased object information and the information derived from terminal tactile input. Adapting the term as it was used by Bingham et al. and Schenk, we hereafter use ''haptic feedback'' in an overarching sense to refer to the denial of an object or even a proxy at the end of the movement (i.e., terminal tactile/haptic feedback).

Notably, a critical role for terminal tactile feedback in maintaining DF's grip scaling is supported by the fact that DF scales her grip aperture to target size when she reaches out to ''grasp'' 2-D images of Efron blocks presented on a table top (Westwood et al., 2002). Furthermore, DF's normal grip scaling in this 2D-grasping task cannot reasonably be attributed to the availability of online visual feedback to update her movements as they unfold or to update the programming of subsequent movements or even some sort of ''visuo-manual matching'' strategy, because she continues to show grip scaling to Efron width in the absence of any online visual feedback whatsoever (Whitwell et al., in press). Additional support for the role of terminal tactile feedback in maintaining DF's grasps comes from the fact that her grip scaling is normal when she reaches out to grasp objects that vary in their visible (virtual) size but are always a constant, intermediate haptic size (Whitwell et al., 2014a, in press). In other words, hapticsbased object information need not provide veridical information about the target width or edges of the visible goal object to maintain normal dorsal-stream mediated grasping. Indeed, the results of Experiment 1 indicate that DF shows normal grip scaling when terminal tactile feedback from the table surface is available to her, even when she performs RPGs. Interestingly, the results of Experiment 1 promote the real-time nature of a natural grasping task over the target-directedness of it per se. Thus, the two critical factors underlying DF's grip scaling slope appear to be terminal tactile feedback and real-time visual input.

In the second experiment reported here, we addressed whether or not the task requiring DF and the control participants to reach out to a visible target that is not physically present results in grip scaling that resembles that of a more explicit pantomimed grasping task as Milner et al. (2012) suggest. After all, a desirable and novel feature of the grasping task used by Bingham et al. (2007) and Schenk (2012a) is that the resultant movements are programmed and executed in real-time and directed at the target—conditions under which the dorsal visuomotor system typically operates. Despite these similarities, there is some indication that the neurologically intact controls in Schenk's (2012a) experiment showed an increase in grip-scaling and inter-subject variability (Whitwell and Buckingham, 2013). DF's grip scaling to object size, as we pointed out earlier, was abolished in this task. Thus, the removal of haptic feedback appears to have changed DF's grip scaling and that of the controls, but in different ways. Unfortunately, however, the controls' grip-scaling with and without haptic feedback was never formally compared in that study. Thus, one aim of the second experiment reported here was to directly test whether removing haptic feedback from a target-directed grasping task influences grip scaling in neurologically-intact individuals. An additional aim (related to the first) was to directly contrast grasping in the target-directed task in which haptic feedback is removed against a variant of the RPG task in which the participants must imagine the visible target at a different location. This way, the responses when haptic feedback is denied in a target-directed grasping task could be compared to the responses in a task that quite obviously requires a pantomimed grasp. In order to implement these tasks, we adopted a mirror apparatus not unlike the one discussed above.

Finally, we took the opportunity that the mirror setup presented us to explore more systematically how the mirror itself might influence natural grasps. Although the mirror apparatus allows for the haptic and visual information about the target to differ, it has at least three possible drawbacks. (1) the mirror apparatus does not allow the participants to view their hand and limb throughout their grasping movement. The unavailability of any visual input about the hand and limb throughout the movement is of course quite different from what occurs with natural grasps. After all, normally when we reach out to pick things up, the hand and limb do not suddenly disappear from sight. A number of studies have shown that when vision is suppressed during the execution of a grasping movement in neurologically-intact individuals, grip aperture increases and, in many cases, the grip scaling slopes decrease (Fukui and Inui, 2006; Fukui et al., 2006; Whitwell et al., 2008, in press; Hesse and Franz, 2009, 2010; Whitwell and Goodale, 2009; Tang et al., 2014). In fact, DF shows similar changes in her grip aperture and grip scaling when vision is suppressed during the movement (Whitwell et al., in press). Presumably, these effects reflect an effort to ensure a sufficient margin of error in the absence of visual information that is normally used for online control. (2) When the participants make contact with the hidden object and pick it up, the virtual object remains stationary in the mirror. In short, there is a clear disconnect between what the participant sees in the mirror and what actually happens. (3) The mirror might be treated as an obstacle which has to be avoided. Any one or a combination of these three factors could have been responsible for reducing grip scaling in both normally-sighted individuals and in DF, because natural grasps that were directed at virtual targets in a mirror were contrasted against natural grasps that were directed at targets in plain view (Whitwell et al., 2014a). Thus, in an additional manipulation, we substituted a pane of glass in for the mirror to assess two effects of using a mirror: the removal of online visual input about the moving hand and limb, and the obvious disconnect between the behavior of viewed and hidden targets after contact. In total, therefore, we set out to test four tasks: grasping real targets (cylinders) viewed through a pane of glass (GG−H); grasping virtual targets viewed in a mirror with haptic feedback (GM−H); grasping virtual targets viewed in a mirror without any haptic feedback (i.e., no cylinder was present behind the mirror, GM−NH); and real-time pantomime grasps that were based on virtual targets viewed in a mirror but displaced to the side without any haptic feedback (RPGNH).

We grouped the task comparisons according to our apriori predictions: (1) that natural grasps directed at virtual targets (GM−H) would result in larger grip apertures than those directed at real targets viewed directly through glass (GG−H); and (2) in the absence of haptic feedback, target-directed grasping movements would resemble RPGNH grasps that are directed towards an an imagined copy of the virtual target.

# Experiment 2

# Methods

# Participants

Twenty-five self-reported right-hand dominant individuals (9 males) ranging from 17 to 33 years of age (M = 21.3, SD = 3.7), volunteered to take part in the second study. In a follow-up pair of control experiments that was prompted by some of our results, we tested an additional group of 18 self-reported righthand dominant individuals (6 males) ranging from 18 to 32 years of age (M = 21.4, SD = 3.5). The participants in both groups provided written informed consent and were compensated \$10 for their time. All experiments were approved by the local ethics committee and were in accordance with the Declaration of Helsinki.

# Apparatus and Stimuli

The apparatus and stimuli did not differ from that described in Experiment 1 except as noted below. The stimuli consisted of three pairs of black cylinders with diameters of 3.5 cm, 4.8 cm, and 6 cm and a height of 7 cm. Depending on the task, the workspace comprised either a mirror or a pane of glass positioned 45◦ from the edge of the table facing the participant. For all of the tasks that involved the mirror setup, the target cylinder was always positioned in front of the mirror. A vertically-standing occluding board was attached to the edge of the table that faced the participant. The occluding board was positioned to the left of the participants' midline so as to block them from viewing the target cylinder directly. This way, the participant could only see the reflection of the cylinder (i.e., its virtual image) placed in front of the mirror. The occluding board was left in place throughout the experiment. The cylinders could be placed at two different positions in front of and (at corresponding positions) behind the mirror. The ''near'' target position was located 14 cm away from the mirror along the participant's sagittal plane. The ''far'' position was located 10 cm farther away from the mirror along the same plane. The hand's resting start position was a small black button located 22 cm to the right and 7 cm in front of the nearest target position (see **Figure 4**). Before the experiment began, the experimenter ensured that all of the participants were positioned close enough to the table so that they could grasp the objects at the farthest distance comfortably and without leaning forward. The experimenter also ensured that the participants could see each of the target cylinders binocularly in the mirror.

# General Procedure and Design

Before each trial was initiated, the participants held the tips of their right index-finger and thumb together while depressing the start button. The participants were instructed to reach out, grasp, and lift up the cylinder using a precision grip (index-finger and

FIGURE 4 | A bird's eye view of the mirror setup used for Experiment 2. The cylinders (indicated by circles with a solid-line border) were placed in front of the mirror at one of two possible positions from trial to trial (the near position is indicated with a filled-in circle). The cylinder was hidden from direct view by an occluding board, and so the participants viewed a virtual cylinder in the mirror. An identical cylinder could be positioned behind the mirror (again indicated by circles with a solid-line border) such that it was spatially coincident with the apparent position of the virtual one. This way, haptic feedback about the object could be permitted (GM−H) or denied (GM−NH) by removing the cylinder from behind the mirror. In one of the tasks, the mirror was replaced with a pane of glass so that participants viewed the cylinder directly (GG−H). For the "real-time" pantomime grasping task (RPGNH), the participants imagined the virtual cylinder at the mirror-symmetrical position (dashed open circles) opposite a sagittal plane that was aligned with the start button. In a brief follow-up investigation, the RPGNH task was modified such that the target was imagined immediately next to the visible one (also indicated with dashed open circles).

thumb) as soon as the lenses of the goggles cleared. Participants were asked once they grasped and lifted the objects to simply move the objects to the center of the table. In all conditions, the lenses of the goggles remained transparent for 2.5 s following the participants' release of the start button before returning to their translucent state (i.e., visual closed-loop feedback). Participants were asked to grasp the objects naturally, neither labored nor speeded. The experimenter explained the procedure for the upcoming task before each block of trials. The experiment was comprised of 4 blocks of 24 trials each for a total of 96 trials. Each block was dedicated to a different task. For each block of trials (i.e., for each task), the six combinations of target-cylinder size and location were presented 4 times each. The block order (i.e., task order) was counterbalanced across participants.

# Grasping Real Targets Viewed Through a Pane of Glass

The participants viewed the cylinders through the pane of glass and were asked to reach out to pick them up as described in the previous section.

# Grasping Virtual Targets Viewed in a Mirror With Haptic Feedback

The participants viewed the cylinders in the mirror. The experimenter ensured that the cylinder behind the mirror matched the one that the participants viewed. The participants were asked to reach out behind the mirror to pick up the cylinder as described in the previous section. Note that the mirror blocked the participants' view of their hand during the movement. Thus, a comparison of this task with the one in which the participants grasped real targets viewed through a pane of glass tests the effect of online visual feedback of the hand and limb during the movement.

# Grasping Virtual Targets Viewed in a Mirror Without Haptic Feedback

This task was identical to the task described in the previous section in all respects, except that, after the matched cylinder was placed behind the mirror, it was immediately removed and the trial then initiated. Positioning a target behind the mirror was done simply to preserve the overall ''feel'' and timing of the events between trials. Neither haptics-based object information nor any terminal tactile feedback was available in this task. In accordance with the instruction to simulate a real grasp, the participants were asked to refrain from sending their fingers or hand through the imagined cylinder.

# Pantomime Grasping Visualized Copies of Virtual Targets Viewed in a Mirror

The participants viewed the cylinders in a mirror, but were asked to execute their grasps as if the cylinder was located to the right of where it appeared to be. This location was the right of the start button at a distance that equaled the distance from the visible cylinder to a sagittal plane aligned with the start button (see **Figure 4**). The experimenter explained this contingency to the participant and reinforced it by indicating the target locations for each of the two possible positions for the viewed cylinder. In accordance with the instruction to simulate a real grasp, the participants were asked to refrain from sending their fingers or hand through the imagined cylinder.

# Data Processing and Statistical Analysis

The data were processed offline with custom software written in Matlab (Mathworks Inc., Natick, MA, USA). The positional data from the IREDs was low-pass filtered at 20 Hz using a 2nd order Butterworth digital filter. Grip aperture was computed as the Euclidean distance between the IRED placed on the thumb and the IRED placed on the index-finger, and the instantaneous velocities were computed for each of the three IREDs and for grip aperture.

The PGA was defined as the largest grip aperture within a search window that was designed to capture the forwardreach component of the movement. The beginning of this window, the movement onset, was operationally defined as the first of 20 consecutive sample frames (100 ms) in which the velocity of the IRED attached to the knuckle of the indexfinger exceeded a threshold of 50 mm/s. The movement onset was also used to calculate the reaction time (RT). The end of the search window was defined as the first sample frame in which the velocity of the IRED fell below 150 mm/s. Linear regression of PGA on the widths of the cylinders was performed separately for each task and the resultant regression coefficient (slope, bPGA) relating the average increase in PGA (in mm) per incremental increase in cylinder width (also in mm) was computed. The PHV was defined as the peak velocity achieved by the knuckle IRED within the search window. One additional measure was operationally defined: the final grip aperture (FGA). The FGA was determined on the basis of grip stability (grip aperture velocity). Grip stability was used to identify the plateau phase of the grip aperture profile during which the participant holds the target (GG−<sup>H</sup> and GM−<sup>H</sup> tasks), pretends to hold a visible target (GM−<sup>H</sup> task), or pretends to hold an imagined copy of a visible target (in the case of the RPGNH). Linear regression of FGA on the widths of the cylinders was performed separately for each task and the resultant regression coefficient (slope, bFGA) relating the average increase in FGA (in mm) per incremental increase in cylinder width (also in mm) was computed. Note that the bFGA should be at or close to 1 for the natural grasps, and so the tests of this measure indicate how faithfully the participants reflected changes in target size from trial to trial in their FGA in the absence of haptic feedback.

To test for differences amongst the tasks, a one-way repeated measures Analysis of Variance (rmANOVA) was conducted separately for each of the dependent measures (RT, PHV, PGA, bPGA, FGA and bFGA) with Task as the main factor. The significant rmANOVAs were followed up with planned paired t-tests designed to test the specific effect of removing online visual feedback on the natural grasps and that of removing haptic feedback. The test of the former effect involved a comparison of the grasps directed at ''real'' cylinders viewed directly through a pane of glass (GG−H) and the grasps directed at ''virtual'' cylinders viewed in a mirror with haptic feedback (GM−H). The tests of the latter effect involved comparisons amongst the three tasks in which virtual cylinders were visible in the mirror: The GM−<sup>H</sup> task, the variant without haptic feedback (GM−NH), and the real-time pantomime grasps directed away from the virtual cylinders and towards imaged ones without haptic feedback (RPGM−NH). With respect to this set of contrasts, it should be noted that the RPGM−NH entailed online visual feedback. Therefore, we included a comparison of this task with the natural grasping task in which online visual feedback was available (i.e., RPGM−NH vs. GG−H). Greenhouse-Giesser epsilon multipliers were applied to the degrees of freedom to all ANOVAs to compensate for potential violations of sphericity of the variance-covariance matrices. The F-statistics which were adjusted in this way are reported in-text as Fadj. Violations of sphericity were assessed using Mauchley's test and assessed at a liberal alpha criterion of 0.15 as Kirk (1995) recommends for tests of underlying assumptions. For all other statistical tests, the alpha criterion for statistical significance was set to 0.05.

# Results

## Reaction Time

The rmANOVA of the reaction times (RTs) yielded a significant main effect of Task, F(3,72) = 26.7, p < 2 × 10−<sup>11</sup> , η 2 <sup>p</sup> = 0.53 (see **Figure 5A**). There was no significant difference in the RTs between GG−<sup>H</sup> and GM−<sup>H</sup> (t(24) = 1.75, p = 0.09), indicating no effect of online visual feedback on the velocity of the reach.

The RTs were slower for GM−NH than the RTs for GM−H, t(24) = 2.81, p < 0.01. In turn, the RTs for RPGNH were significantly slower than those for GM−H, t(24) = 6.52, p < 1 × 10−<sup>6</sup> . However, the RTs for RPGNH were significantly faster than the RTs for GM−NH, t(24) = 3.11, p < 5 × 10−<sup>3</sup> . Thus, the removal of haptic feedback induced a partial shift in the RTs towards pantomimed grasps. In other words, removing haptic feedback slowed the RTs and displacing the grasps slowed the RTs further still. Finally, the RTs for GG−<sup>H</sup> were significantly faster than the RTs for RPGNH, t(24) = 8.57, p < 1 × 10−<sup>8</sup> , suggesting that the slowing of RT that occurs when haptic feedback is denied occurs regardless of whether online visual feedback of the hand and limb is available or not.

#### Peak Hand Velocity (PHV)

The rmANOVA of PHV yielded a significant main effect of task, Fadj(2,43) = 21.2, p < 1 × 10−<sup>6</sup> , η 2 <sup>p</sup> = 0.47 (see **Figure 5B**). The PHV was significantly slower for GM−<sup>H</sup> than for GG−<sup>H</sup> (t(24) = 5.34, p < 2 × 10−<sup>5</sup> ), indicating a role for online visual feedback of the hand and limb in the velocity of the reach.

The PHV was significantly slower for GM−NH than the PHV for GM−H, t(24) = 5.87, p < 5 × 10−<sup>6</sup> . Furthermore, the PHV was significantly slower for RPGNH than the PHV for GM−H, t(24) = 2.29, p < 0.04. Finally, the PHV did not differ significantly between GM−NH and RPGNH, t(24) = 1.75, p = 0.09. Thus, the removal of haptic feedback resulted in a complete shift in the PHV towards pantomime grasps. In other words, regardless of whether the grasps were target-directed or not, the velocity of the reach was slower when haptic feedback was denied.

FIGURE 5 | Reaction time (RT) and peak hand velocity (PHV) across the four variants of the grasping task arranged (within each panel) from left to right as follows: grasps directed at real (i.e., viewed through a pane of glass) targets (GG−H) with haptic feedback, grasps directed at virtual targets (i.e., viewed in a mirror) with haptic feedback (GM−H), grasps directed at virtual targets with no haptic feedback (GM−NH), and the "real-time" pantomime grasps directed at imagined copies of the virtual targets (RPGNH). Note that the error bars reflect 95% confidence intervals extracted from the mean square error term from the rmANOVA (corrected for violations of sphericity where appropriate). (A) RT increased when a mirror was used rather than a pane of glass for target-directed grasps with haptic feedback. The RT increased further when haptic feedback was denied and increased further still when the participants performed displaced-pantomime grasps. (B) PHV slowed when a mirror was used rather than a pane of glass for target-directed grasps with haptic feedback. PHV slowed further when haptic feedback was denied and when the participants performed displaced-pantomime grasps.

The PHV was significantly faster for GG−<sup>H</sup> than it was for RPGNH (t(24) = 4.54, p < 2 × 10−<sup>4</sup> ), suggesting that the slowing of PHV when haptic feedback is denied occurs regardless of whether online visual feedback of the hand and limb is available or not.

# Peak Grip Aperture (PGA)

The rmANOVA of the mean PGA revealed a significant main effect of Task, Fadj(2,47) = 18.5, p < 2 × 10−<sup>6</sup> , η 2 <sup>p</sup> = 0.44 (see **Figure 6A**). The PGA for GM−<sup>H</sup> was significantly larger than the PGA for GG−<sup>H</sup> (t(24) = 5.16, p < 3 × 10−<sup>5</sup> ), indicating a significant effect of online visual feedback of the hand and limb in the offline and/or online updating of grip aperture in the natural grasping task.

The PGA was significantly smaller for GM−NH than the PGA for GM−H, t(24) = 3.4, p < 3 × 10−<sup>3</sup> . In turn, the PGA for RPGNH was significantly smaller than the PGA for GM−H, t(24) = 6.43,

FIGURE 6 | The PGA, slopes relating PGA to target size (bPGA), the final grip aperture (FGA), and the slopes relating the FGA to the target size (bFGA) across the four tasks. (A) The overall PGA increased when a mirror was used (GM−H) rather than a pane of glass (GG−H) for target-directed grasps with haptic feedback. For grasps directed a virtual targets, removing haptic feedback (GM−NH) reduced the PGA. The PGA was reduced further for the real-time pantomimed grasps (RPGNH). For each task the mean PGA for each target size is plotted for each participant. Evidently, denying haptic feedback increased the slopes. (B) The participants' bPGAs (open circles) and the mean bPGA (dashes) for each task. The bPGAs for GM−<sup>H</sup> were smaller than those for GG−H, indicating a significant role for online visual feedback of the hand and limb. The bPGAs were larger, however, whenever haptic feedback was denied, regardless of whether the grasps were target-directed (GM−NH) or not (RPGNH). (C) The overall FGA was reduced when haptic feedback was not available. Plotted for each task is the mean FGA for each target size for each participant. Even in the absence of haptic feedback, the FGAs were well-related to target size. (D) The bFGA (open circles) did not differ amongst the four tasks. Note that the error bars reflect 95% confidence intervals extracted from the mean square error term from the rmANOVA (corrected for violations of sphericity where appropriate).

p < 2 × 10−<sup>6</sup> . However, the PGA for RPGNH was significantly smaller than the PGA for GM−NH, t(24) = 4.01, p < 6 × 10−<sup>4</sup> . Thus, removing haptic feedback induced a partial shift in the PGA towards pantomimed grasps. In other words, removing haptic feedback reduced the PGA, but displacing the grasp reduced the PGA further still.

The PGA for the GG−<sup>H</sup> task was significantly larger than the PGA for the RPGNH task (t(24) = 4.36, p < 3 × 10−<sup>4</sup> ), suggesting that the reduction in PGA when haptic feedback is denied also occurs regardless of whether online visual feedback of the hand and limb is available or not.

# Regression Coefficients (Slopes) Relating PGA to Target Width

The rmANOVA performed on the slopes (bPGA) revealed a significant main effect of Task, Fadj(2,52) = 24.4, p < 2 × 10−<sup>8</sup> , η 2 <sup>p</sup> = 0.5 (see **Figure 6B**). The bPGA for GM−<sup>H</sup> was significantly smaller than the bPGA for GG−H, t(24) = 4.46, p < 2 × 10−<sup>4</sup> .

The bPGA for GM−NH was significantly larger than the bPGA for GM−H, t(24) = 7.31, p < 2 × 10−<sup>7</sup> . In turn, the bPGA for RPGNH was significantly larger than the bPGA for GM−H, t(24) = 6.33 p < 2 × 10−<sup>6</sup> . Finally, the bPGA did not differ significantly between GM−NH and RPGNH, t(24) = 1.79, p = 0.09. Thus, the removal of haptic feedback resulted in a complete shift in the grip scaling slopes toward pantomime grasps. In other words, regardless of whether the grasps were target-directed or not, the slopes were larger when haptic feedback was denied.

The bPGA for GG−<sup>H</sup> was significantly smaller than the bPGA for RPGNH (t(24) = 4.06, p < 5 × 10−<sup>4</sup> ), suggesting that the increase in bPGA when haptic feedback is denied also occurs regardless of whether online visual feedback of the hand and limb is available or not.

Finally, we opted to test for a difference in the bPGA between the controls' of Experiment 1 and the participants in the GG−<sup>H</sup> task of Experiment 2 using an independent samples t-tests with appropriate adjustments for violations of homogeneity where necessary. We found no significant difference in the bPGA across the two groups (p = 0.64), suggesting that the pane of glass did not affect the bPGA in Experiment 2. Interestingly, pooling the no haptic feedback conditions in Experiment 2 (i.e., GM−NH and RPGNH) to test for an effect of the absence of haptic feedback compared to terminal tactile feedback (i.e., the RPG task of Experiment 1) revealed an increase in the bPGA for the former, t(28) = 3.36, p < 3 × 10−<sup>3</sup> . Thus, the results of these additional tests support the findings of Experiment 1 that terminal tactile feedback helps ''normalize'' grip scaling slopes.

# Final Grip Aperture (FGA)—Grip Stability at the End of the Reach

The rmANOVA of FGA revealed a significant main effect of Task, Fadj(2,43) = 20.1, p < 2 × 10−<sup>6</sup> , η 2 <sup>p</sup> = 0.46 (see **Figure 6C**). Not surprisingly, the FGA for GM−<sup>H</sup> and GG−<sup>H</sup> did not differ significantly (t(24) = 1.41, p = 0.17.), presumably because this measure was constrained by the widths of the cylinders in these tasks. Thus, the removal of haptic feedback resulted in a complete shift in the FGA toward pantomime grasps. In other words, regardless of whether the grasps were target-directed or not, the FGA was smaller when haptic feedback was denied.

The FGA for GM−NH was smaller than the FGA for GM−H, t(24) = 5.3, p < 2 × 10−<sup>5</sup> . In turn, the FGA for RPGNH was significantly smaller than the FGA for GM−H, t(24) = 3.72, p < 2 × 10−<sup>3</sup> . However, the FGA for GM−NH did not differ significantly from the FGA RPGNH, t(24) = 1.69, p = 0.1. Thus, the removal of haptic feedback resulted in a complete shift in the FGA toward pantomime grasps. In other words, regardless of whether the grasps were target-directed or not, the FGA was smaller when haptic feedback was denied.

The FGA for GG−<sup>H</sup> was significantly larger than the FGA for RPGNH (t(24) = 4.19, p < 4 × 10−<sup>4</sup> ), suggesting that the reduction in FGA when haptic feedback is denied also occurs regardless of whether online visual feedback of the hand and limb is available or not.

# Regression Coefficients (Slopes) Relating FGA to Target Width

The rmANOVA performed on the slopes relating FGA to target size (bFGA) indicated no significant main effect of Task, Fadj(2,38) = 1.6, p = 0.22 (see **Figure 6D**), suggesting that, even in the absence of haptic feedback, participants on the whole took into account differences in the widths of the virtual cylinders when simulating their grip on them (in the case of GM−NH) or on imagined copies of the virtual cylinders (in the case of the RPGNH).

Finally, we examined the change in the slopes relating PGA to target size (bPGA) and those relating FGA to target size (bFGA) for each task (i.e., ∆b = bFGA − bPGA). This analysis provides an indication of how consistent the slope was from the point in the response at which PGA was achieved (i.e., while the hand was in-flight) to the point at which the FGA occurred (i.e., while the fingers held the object tor simulated holding one). A significant ∆b was observed for GG−<sup>H</sup> (M = 0.25, SD = 0.18, t(24) = 6.88, p < 5 × 10−<sup>7</sup> ) and GM−H, M = 0.4, SD = 0.21, t(24) = 9.7, p < 9 × 10−10. In contrast, the ∆b for GM−NH (M = −0.03, SD = 0.18, t(24) = 0.76, p = 0.47) and RPGNH (M = −0.06, SD = 0.28, t(24) = 1.13, p = 0.27) failed to differ significantly from zero. Thus, the ∆b appeared to be largely driven by the availability of haptic feedback. To confirm this, a rmANOVA performed on the ∆b indicated a main effect of Task, F(3,72) = 39.9, p < 3 × 10−<sup>15</sup> . Given the null findings amongst the tasks with respect to the bFGA, the differences in ∆b amongst the tasks are quite likely to have been driven by the differences in the bPGA we reported above. Indeed, follow up tests (not reported) showed that this was true. Thus, the analysis of the ∆b indicates that in the absence of haptic feedback, the participants grip aperture faithfully reflected differences in the widths of the targets while their hand was inflight and when it was simulating holding a visible or imagined cylinder.

# Testing for Possible Methodological Issues With Respect to the Use of the Mirror

Given the significant differences between GG−<sup>H</sup> (natural grasps directed at real targets viewed through a pane of glass) and GM−<sup>H</sup> (natural grasps directed at virtual targets viewed in a mirror) tasks across a number of measures, we tested an additional group of participants (see Section Participants) to test for factors other than the online visual feedback of the hand and limb that could be driving this effect. In this follow-up experiment, we employed the GG−<sup>H</sup> and GM−<sup>H</sup> tasks (see Sections Participants, Apparatus and Stimuli, General Procedure and Design, Grasping Real Targets Viewed through a Pane of Glass, Grasping Virtual Targets Viewed in a Mirror with Haptic Feedback) however, the grasps in this additional experiment were performed entirely in open loop. In other words, the lenses of the goggles switched from a transparent state to a translucent one as soon as the participants' fingers left the start button. Thus, the only difference between the tasks was that nature of the target image (one being virtual and the other real). If other methodological factors (e.g., subtle mismatch in the placement of the copy of the virtual target or differences in lighting) were responsible for the differences in grasping in the two tasks (rather than the differences in online sources of visual input) then we should replicate the pattern of results that we observed, because these differences would still be present despite the loss of online visual feedback throughout the movements.

The results were clear: in the absence of any visual input throughout the grasping movements, viewing virtual or real targets did not significantly affect the RTs (t(17) = 1.22, p = 0.24), PHVs (t(17) = 1.16, p = 0.26), PGAs (t(17) = 0.26, p = 0.8), or the bPGA, t(17) = 0.14, p = 0.89 (see **Figure 7A**). Thus, the differences in the PHVs, PGAs, and bPGA in Experiment 2 are unlikely to have been driven by methodological factors putatively introduced by using a mirror.

# Removing Online Visual Feedback from the "Real-Time" Pantomime Grasping Task

As we have seen, the PGA for GM−NH was smaller than the PGA for GM−<sup>H</sup> yet larger than the PGA for the RPGNH. A similar result was observed for the RTs. Specifically, the RTs for GM−NH were slower than those for GM−<sup>H</sup> yet faster than those for RPGNH. The partial shifts in these measures for GM−NH towards those observed in the pantomime grasping task (i.e., RPGNH) suggest that the target-directed nature of the GM−NH task might have partially compensated for the effect of removing haptic feedback. It also possible, however, that the availability of online visual feedback of the hand and limb or the added shift in gaze or attention that the RPGNH task demanded (as participants looked to towards the empty workspace to imagine a copy of the target) increased the RT. To test these possibilities, we carried out an additional experiment. We reasoned that altering RPGNH so that the grasps were directed to a location immediately next to the virtual target should minimize differences between the two tasks in terms of the availability of visual feedback, shifts in attention, and other factors such as a difference in biomechanical constraint. A difference in PGA or RT following a comparison of the GM−NH and modified RPGNH (i.e., the grasps were directed to the side of the virtual target) would support the suggestion that the target-directed nature of the GM−NH can at least partially compensate for an absence of haptic feedback.

Compared to GM−NH, the modified RPGNH showed slower PHVs (t(17) = 3.73, p < 2 × 10−<sup>3</sup> ) (attributable to the modest

grasps in the absence of any online visual feedback (visual open loop). Grasps were directed at "real" (i.e., viewed through a pane of glass) targets (GG−H) and virtual (i.e., viewed in a mirror) targets (GM−H) in visual open loop (vision was occluded at the start of the movement). In the case of GM−H, haptic feedback was available when participants made contact with a spatially coincident duplicate that was positioned behind the mirror. Whether the grasps were directed at virtual targets or real

overall reduction in distance the hand travelled in this task) and, importantly, a smaller PGA, t(17) = 2.75, p < 0.02. Thus, directing the hand away from the target and towards an imagined copy appears to reduce the PGA no matter how far away from the visible object the hand is directed. The results also indicated no significant differences in the RT (t(17) = 0.23, p = 0.82) or in the bPGA, t(17) = 0.14, p = 0.89 (see **Figure 7B**) between GM−NH and modified RPGNH.

## Between-Groups Tests of the Regression Coefficients (Slopes) Relating PGA to Target Width

Testing the additional group of participants also afforded us an opportunity to test for a replication of one of the critical finding of Experiment 2 concerning the grip scaling slopes (bPGA). In a series of independent–samples t-tests involved the bPGA of the GM−NH and RPGNH tasks from the first and second group of participants, and the GM−<sup>H</sup> task from the first group, and in the series of independent–samples t-tests for the tasks in which the targets were virtual (i.e., viewed in a mirror) and the goggles remained clear for the duration of the movement (i.e., closed-loop with respect to the target). We adjusted the multiple post hoc independent–samples ttests using Holm's step-down Bonferroni procedure (Holm, 1979). The results, again, showed that the critical factor for this measure was the absence of haptic feedback. The bPGA for GM−NH (p < 7 × 10−<sup>7</sup> ) and RPGNH (p < 5 × 10−<sup>8</sup> ) from the second group of participants were significantly steeper than the bPGA for the GM−<sup>H</sup> from the first group of participants. Furthermore, none of the tasks in which haptic feedback was denied differed between the two groups of participants (pmax = 0.14, uncorrected).

ones made no difference across any of the dependent measures, including PGA and the slopes. (B) The PGA and slopes for the GDVT task in which haptic feedback was denied and the displaced-pantomime grasping (RPGNH) task in which the grasps were directed immediately to the right of the visible location of the target towards an imagined copy. Whether the grasps were directed towards or beside the virtual target did not affect the slopes, which appear to be quite steep in both tasks. Sending the hand to a location right beside the object did, however, reduce the overall PGA, just as it did for displaced-pantomime grasps to locations more distant from the location of the virtual target.

# Discussion and Conclusions

One of the principal aims of Experiment 2 was to determine whether or not removing both haptics-based object information and terminal tactile feedback (together referred to here as ''haptic feedback'') from a target-directed grasping task shifts the response mode away from a natural one and towards a more pantomimed (i.e., simulated) kind as has been suggested by Milner et al. (2012). To do this, we compared target-directed grasps with (GM−H) and without haptic feedback (GM−NH) to pantomime grasps (RPGNH) in which the participants were asked to imagine a copy of the target in another location in the workspace and to grasp that imaginary object as if it were actually there (e.g., Goodale et al., 1994b; Holmes et al., 2013). We found that when participants reached out to grasp virtual targets, removing haptic feedback slowed RT and PHV, reduced PGA, increased the slopes relating PGA to the width of the target, and reduced the FGA. Just as important was the fact that the grasps directed at virtual targets (viewed in a mirror) without haptic feedback were statistically indistinguishable from the pantomime grasps in terms of the PHV, the slopes relating PGA to target size, the slopes relating FGA to target size, and the FGA, suggesting a complete shift across these measures towards pantomimed grasping following the removal of haptic feedback. The only measures that differed between the two ''no haptic feedback'' tasks were the RT and the magnitude of the PGA. It is important to acknowledge, however, that removing haptic feedback from grasps directed at virtual targets slowed the RTs and reduced the PGAs. In other words, both of these measures registered a shift in the direction away from natural grasps and towards the pantomimed ones.

An additional aim of Experiment 2 was to determine whether or not the mirror itself has an effect on the kinematics of target-directed grasps. After all, the mirror introduces three key differences when compared to natural grasps: First, the mirror blocks the participant's view of their hand and limb as soon as the participant reaches behind it (removing re-afferent online visual feedback). Second once the participants make contact with the hidden object and the virtual target, the mirror imposes a disconnect between the felt movements of the hidden object and the apparently stationary target visible in the mirror. Although this effect might startle the participants at first, it is reasonable to suggest that the participants acclimate to this situation, growing more comfortable on subsequent trials. This says nothing, however, about any possible effects all of this might have on the unconscious ''automatic'' online control mechanisms that normally mediate grasping. Third, the mirror might act as an obstacle that the participants attempt to avoid. Given these considerations, we implemented an additional task in which the participants reached out to grasp target they viewed through a pane of glass. The pane of glass was the same size as the mirror and was positioned in the same way with respect to the participant. Compared to natural grasps directed behind the pane of glass, the ones directed behind the mirror resulted in slower PHVs, larger PGAs, and shallower slopes. Nevertheless, it was possible that some other aspects of the mirror task may have played a role. We ruled these factors out in a control experiment in which we removed online visual feedback altogether for both tasks. In this control experiment, all the differences between the grasps directed behind the mirror and the grasps directed behind the pane of glass completely disappeared, strongly implicating a role for one or more of the sources of online visual feedback outlined above in the programming and updating of targetdirected grasps. Given Connolly and Goodale's (1999) null findings concerning the magnitude of the PGA and the fact that the participants in that study were permitted a view of the target and the hand making contact with the target, then the results of the current investigation suggest a significant role for vision during the contact and manipulation phase of the grasping movement in the programming of grip aperture on subsequent grasps.

Many of the additional findings in the present investigation can be explained through the changes in task demands and differences in the availability of visual and haptic input. For example, the overall reduction in PGA and FGA in the absence of haptic feedback (see also Bingham et al., 2007; Fukui and Inui, 2013) is likely due to the removal of the physical constraints that the object imposes on the fingers and, therefore, the minimum magnitude that the grip aperture would normally be required to achieve a suitable grasp. Without the physical constraints imposed on the fingers and hand by an actual object, there would be (1) no consequences for consistently under-sizing grip aperture, such as knocking the object away; and (2) less effort (and perhaps even more comfort) in opening the hand a smaller amount. The FGA, being a measure of grip stability when the target is being held, would necessarily be restricted by the sizes of the cylinders. We speculate that the selective removal of haptics-based object information might also lift this restriction and result in a similar reduction in FGA. Nevertheless, unlike the FGA, the PGA was affected by both haptic feedback and online visual feedback of the hand and limb. Specifically, providing online visual feedback and removing haptic feedback each effected reductions in the PGA. The effect of online visual feedback of the hand and limb on PGA observed in the present study is in line with previous findings following a comparable manipulation (Whitwell et al., 2014a,b) and is also in line with the broader literature on the effects of removing online visual feedback entirely (e.g., Jakobson and Goodale, 1991; Whitwell et al., 2008; Whitwell and Goodale, 2009; Hesse and Franz, 2010). The effect of removing haptic feedback on PGA observed in the present study is also in line with previous reports in which terminal tactile feedback was almost certainly denied (Bingham et al., 2007; Fukui and Inui, 2013). Interestingly, (although not always explicitly tested), a similar effect on PGA appears to occur in the absence of haptics-based object information when short (e.g., ∼1 cm in height) block-like stimuli (or even 2D images) are used in which the fingers are highly unlikely to avoid touching the surface of the table at the end of the reach (e.g., Westwood et al., 2002; Cavina-Pratesi et al., 2011; Holmes et al., 2013). If we assume an additive model of the effects of online visual feedback and haptic feedback, then consideration of the details of the tasks of the present study readily explain the findings involving FGA and PGA.

In keeping with an appeal to differences in task demands, we should point out that we required the participants to refrain from sending their hand and fingers through the visible or imagined target for the tasks in which haptic feedback was removed. We would argue that most (if not all) tasks in which the participants simulate grasps carry with them analogous instructions, regardless of whether such instructions are stated explicitly by the experimenter or are tacitly understood by the participant. Critical to this is (1) any consideration the participants might give to the sizes and positions of the target in a situation in which the target is not actually there; and (2) how well the participants understand what their hand does when they reach out to pick up a goal object. It seems likely that these factors account for the increase in RT when haptic feedback was denied. A similar appeal to differences in task demands can explain the additional increase in RT that occurred when the grasps were directed at an imagined copy of the visible object. Unlike the other grasping tasks, the instructions for the pantomime grasps required the participants to imagine a copy of the visible target at a different location. Presumably, participants would first look at the visible target and then look towards the location where they were to imagine a copy of that object before or shortly after they initiated their response. In contrast, in the target-directed grasping tasks (with and without haptic feedback), the target's viewed position and the location to which the participants sent their hand are one and the same. We suspect that the addition of a preparatory shift in gaze in the pantomime grasping task likely increased the RT relative to the targetdirected grasping task in which haptic feedback was denied. It is possible that the biomechanical difference in the direction that the participants sent their hand and limb in pantomime grasping task or the availability of online visual feedback might also play a role in the increase in RT. We should point out, however, that in the control experiment in which haptic feedback was denied, RT did not depend on whether the participants directed their hand towards the virtual target or beside it. In other words, the difference in RT between pantomimed and target-directed grasps without haptic feedback was abolished when the pantomime task was modified to minimize differences in shifts in gaze or attention, biomechanical constraints, and online visual feedback. Furthermore, we note that online visual feedback did not influence the RT of natural grasps in the current study—a finding consistent with previous investigations of natural grasps with and without online vision (e.g., Whitwell et al., 2008; Hesse and Franz, 2010). Thus, it seems unlikely that this factor can account for differences in RT in the absence of haptic feedback.

In contrast to the RTs and the PGA, the PHV, the slopes relating PGA to target size, the slopes relating FGA to target size, and the FGA were not affected by the added requirements of pantomime grasps when compared to the target-directed grasps without haptic feedback. In other words, for these measures of movement execution, the target-directedness of the response was not a critical factor. Instead, the removal of haptic feedback about the object appeared to dominate, independent of whether the grasp was directed to a visible or an imagined target. In line with Bingham et al.'s (2007) finding, without haptic feedback the PHVs were slower. The participants likely approached the targets more cautiously and deliberately, presumably because they were simulating what they would do if an object was actually there, making sure that their fingers did not go through the visible or imagined object. Importantly, the slopes increased relative to the slopes for grasps that received haptic feedback, approaching a 1:1 relationship between changes in the width of the target and changes in PGA. In fact, the slopes in these tasks resemble those observed during manual estimation tasks in which the participants indicate the width of a visible object by opening their thumb and index-finger a matching amount (e.g., Daprati and Gentilucci, 1997; Haffenden and Goodale, 1998; Pettypiece et al., 2010; Schenk, 2012a; Whitwell et al., 2014a, in press). Thus, the increase in the grip scaling slope when haptic feedback is not available would appear to reflect the deliberate consideration given to the sizes of the targets in these simulated grasps. As Whitwell and Buckingham (2013) noted, removing haptic feedback from a real-time grasping task appears to increase the grip-scaling slopes (Schenk, 2012a). In our experiment (see also Byrne et al., 2013) we explicitly tested this and found that, in the absence of haptic feedback, the slopes do, in fact, increase relative to natural grasping tasks. Interestingly, on a task that is not unlike the delayedpantomimed grasping task devised by Goodale et al. (1994b), the slopes appear to increase relative to those observed on a natural grasping task regardless of whether vision of the workspace is available at the time of the movement or not (see Fukui and Inui, 2013). Overall, it seems reasonable to conclude that in the absence of haptic feedback, the geometric properties of the target are taken into explicit consideration when planning and programming the grasp. Thus, DF's poor grip scaling slope when haptic feedback is consistently denied (Schenk, 2012a) can be attributed to a switch in the kind of response she provided towards a more pantomimed or simulated one as Milner et al. (2012) suggested. Interestingly, as we showed in Experiment 1, the provision of some proxy next to the visible target (in our case the surface of the table) has a normalizing influence on DF's and the controls' slopes. This finding adds to a growing body of work indicating that DF's slope remains normal provided that real-time visual input is available along with tactile feedback from a proxy of the target (Westwood et al., 2002; Whitwell et al., 2014a,b).

Additional support for a distinction between haptics-based object information from a real (3D) object and tactile feedback from a proxy comes from studies of the influence of a mismatch between the haptic and visual size of target objects. When normally-sighted participants reach out to grasp objects in which the apparent visual width of the objects differs from their felt width, they typically show some adaptation in their PGA to the actual (i.e., the felt) size of the target—even though they continue to scale their grip aperture to the visual width of the target (e.g., Gentilucci et al., 1995; Säfström and Edin, 2004, 2008; Pettypiece et al., 2010). In fact, DF responds in an identical manner, suggesting that (1) the ventral stream is not required for the updating of grip aperture to reflect the real size of a target and that (2) veridical haptics-based object information is not required for DF to maintain normal grip scaling to trial-to-trial changes in the visual sizes of targets (Whitwell et al., 2014a,b). Rather, DF's dorsal stream can exploit terminal tactile feedback to update her grip aperture on subsequent grasping movements and to maintain normal visuomotor processing of target shape to program movements parameters like grip aperture. Thus, it seems reasonable to conclude that (1) provided real-time visual input is available, tactile feedback from the surface of the table is sufficient to keep the visuomotor networks in DF's dorsal stream engaged; and that (2) the damaged areas of DF's ventral stream are not necessary for grip scaling for grasps that are directed towards the table surface next to a visible object.

Since Goodale et al.'s (1994a) study, pantomime grasps have been used in many kinematic investigations and is considered a tool to test the role that perception plays in the visual control of skilled actions. For example, the PGAs of pantomime grasps have been shown to be more susceptible to the Muller-Lyer illusion than natural grasps (Westwood et al., 2000). In addition, the within-subject variability of the PGAs of pantomimed grasps, but not natural ones, obeys Weber's Law (i.e., the variability of the PGA increases linearly with target width; Holmes et al., 2013; although see Foster and Franz, 2013). In fact, even the movement preparatory time for pantomimed grasps, but not for natural gasps, is increased by the holistic object-perception that is thought to underlie Garner interference (Ganel and Goodale, 2003, 2014). Moreover, patient IG, who suffers from optic ataxia following damage to her PPC shows a paradoxical improvement in the correlation between her PGA and target width when she executes pantomime grasps following a delay period compared to natural grasps (Milner et al., 2001). Finally, provided the object is visible, the hand kinematics of magicians (who routinely pantomime actions to deceive their audiences) look far more like those of natural grasps than they do those of non-magicians (Cavina-Pratesi et al., 2011). In all of these studies, haptic feedback about the object was denied but not terminal tactile feedback about the end of the movement. Thus, tactile feedback from the tabletop is not enough to preserve all of the kinematics of a real grasping movement. Indeed, when neurologically-intact individuals pretend to pick up 2D images, the variability of their grip aperture scales with target size as Weber's law would predict (Holmes and Heath, 2013), just as it does for pantomimed grasps (Holmes et al., 2013). Furthermore, grasps that are directed towards 2D objects invoke holistic processing (Freud and Ganel, 2015) in which the irrelevant and relevant target dimensions interact to influence processing times. This is not so for grasps that are directed at 3D objects (e.g., Janczyk and Kunde, 2012; Eloka et al., 2014; Freud and Ganel, 2015).

Importantly, it remains to be seen whether the cognitive or perceptual effects associated with pantomimed grasps are indeed mediated by ventral stream processing as is commonly assumed. An interesting future direction might be to test DF's pantomime grasps for evidence of holistic processing (e.g., Garner interference) and relative sensitivity to stimulus magnitude (e.g., Weber's law). Interestingly, pantomime grasps directed to the workspace next to a visible object fail to elicit preferential activity in the temporaloccipital areas in healthy participants, uniquely recruiting, instead, regions in the supramarginal gyrus, middle intraparietal sulcus, and supplementary motor area of the right hemisphere (Króliczak et al., 2008)—areas that remain intact in DF. These findings, combined with those of Goodale et al. (1994b) and the present study, suggest that a delayed pantomime grasping task would invoke preferential activity in areas of the occipito-temporal cortex of healthy individuals that

# References


are damaged in DF. Interestingly, these areas are in fact recruited when reach-to-grasp movements are based on a memory of the target, albeit in the context of a delayed grasp (as opposed to a delayed pantomime grasp) which received haptic feedback about the remembered object at the end of the reach (Singhal et al., 2013). Thus, although pantomime grasps with tactile feedback invoke cognitive and perceptual influences that are absent in natural grasps, some of these influences (e.g., the effects of holistic processing on movement preparation time, or of stimulus magnitude) might well emerge from a combination of visual processes in the ventral stream and the inferior parietal cortex of the right hemisphere.

In summary, the current study shows clear evidence that the removal of haptic feedback induces a shift from natural towards pantomimed (simulated) grasps, as suggested by Milner et al. (2012). The pattern of changes in the grasps kinematics, longer initiation times, slower movements, and steeper slopes were indicative of a more deliberate process of responding in which the participants explicitly took into account the metrics of the object, the location to which they were directing their hand, and the path that their hand and fingers would take. Furthermore, as Fukui and Inui (2013) have pointed out, the reduction in grip aperture that followed the removal of haptic feedback presumably reflects a natural consequence of the removal of a physical object, which, normally, would impose a constraint on the grip aperture of a natural grasp. Thus, the removal of haptic feedback also changes the task incentives. Without haptic feedback, there is no obvious consequence for an inaccurate grasp. These results and those of other investigations highlight the importance of haptics-based object information, or, at the least, terminal tactile feedback, in maintaining normal grasps which, we have shown here with patient DF, depend on intact dorsal pathways.


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Whitwell, Ganel, Byrne and Goodale. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Dissociable contribution of the parietal and frontal cortex to coding movement direction and amplitude

Marco Davare1,2,3† , Alexandre Zénon1† , Michel Desmurget <sup>4</sup> and Etienne Olivier <sup>1</sup> \*

1 Institute of Neuroscience (IoNS), School of Medicine, University of Louvain, Brussels, Belgium, <sup>2</sup> Sobell Department of Motor Neuroscience and Movement Disorders, UCL Institute of Neurology, University College London, London, UK, <sup>3</sup> Department of Kinesiology, Movement Control and Neuroplasticity Research Group, Biomedical Sciences Group, KU Leuven, Leuven, Belgium, <sup>4</sup> Centre de Neurosciences Cognitives, CNRS, UMR 5229, Bron, France

To reach for an object, we must convert its spatial location into an appropriate motor command, merging movement direction and amplitude. In humans, it has been suggested that this visuo-motor transformation occurs in a dorsomedial parieto-frontal pathway, although the causal contribution of the areas constituting the "reaching circuit" remains unknown. Here we used transcranial magnetic stimulation (TMS) in healthy volunteers to disrupt the function of either the medial intraparietal area (mIPS) or dorsal premotor cortex (PMd), in each hemisphere. The task consisted in performing steptracking movements with the right wrist towards targets located in different directions and eccentricities; targets were either visible for the whole trial (Target-ON) or flashed for 200 ms (Target-OFF). Left and right mIPS disruption led to errors in the initial direction of movements performed towards contralateral targets. These errors were corrected online in the Target-ON condition but when the target was flashed for 200 ms, mIPS TMS manifested as a larger endpoint spreading. In contrast, left PMd virtual lesions led to higher acceleration and velocity peaks—two parameters typically used to probe the planned movement amplitude—irrespective of the target position, hemifield and presentation condition; in the Target-OFF condition, left PMd TMS induced overshooting and increased the endpoint dispersion along the axis of the target direction. These results indicate that left PMd intervenes in coding amplitude during movement preparation. The critical TMS timings leading to errors in direction and amplitude were different, namely 160–100 ms before movement onset for mIPS and 100–40 ms for left PMd. TMS applied over right PMd had no significant effect. These results demonstrate that, during motor preparation, direction and amplitude of goal-directed movements are processed by different cortical areas, at distinct timings, and according to a specific hemispheric organization.

Keywords: action planning, prehension, goal-directed, step-tracking, sensorimotor transformation, posterior parietal cortex, reaching, transcranial magnetic stimulation

# Introduction

Visually-guided movements require sensory information about the target to be extracted and transformed into an appropriate motor command (Crawford et al., 2011; Vesia and Crawford, 2012). In the particular instance of arm movements aimed at grabbing an object, two types of

#### Edited by:

Simona Monaco, University of Trento, Italy

#### Reviewed by:

Patrizia Fattori, University of Bologna, Italy Neil Albert, Colgate University, USA

#### \*Correspondence:

Etienne Olivier, Institute of Neuroscience (IoNS), School of Medicine, University of Louvain, 53, Avenue Mounier COSY-B1.53.04, B-1200 Brussels, Belgium etienne.olivier@uclouvain.be

†These authors have contributed equally to this work.

> Received: 29 January 2015 Accepted: 14 April 2015 Published: 06 May 2015

#### Citation:

Davare M, Zénon A, Desmurget M and Olivier E (2015) Dissociable contribution of the parietal and frontal cortex to coding movement direction and amplitude. Front. Hum. Neurosci. 9:241. doi: 10.3389/fnhum.2015.00241 visual information related to object's extrinsic and intrinsic features need to be extracted to feed two independent transformation processes, which lead to two separate movement components i.e., a reaching (transport of the hand) and a grasping (pre-shaping of the hand posture) component (Jeannerod et al., 1995; Jeannerod, 1997). A large number of experiments in both human and non-human primates corroborated the view that two separate pathways, connecting the posterior parietal cortex (PPC) to the premotor cortex, subserve the two movement components underlying prehension, allowing primates to interact so skillfully with their environment (Jeannerod et al., 1995; Castiello, 2005; Culham et al., 2006). According to this classical view, the reaching component is subserved by a dorsomedial pathway, connecting the medial part of IPS (mIPS) and parieto-occipital junction (POJ) to the dorsal premotor cortex (PMd) and the grasping component relies on a dorsolateral circuit connecting the anterior intraparietal (AIP) and ventral premotor (PMv) cortex (Jeannerod et al., 1995; Castiello, 2005; Davare et al., 2011) although some findings challenge the view that these two circuits process reach and grasp independently (Raos et al., 2004; Fattori et al., 2012). A comparable organization has been evidenced in non-human primates although homologies between human and monkey PPC remain debated (Mars et al., 2011; Vesia and Crawford, 2012; Andersen et al., 2014; Turella and Lingnau, 2014).

However, the question of the causal contribution of individual areas belonging to this ''reaching pathway'' remains open. In humans, the technique of choice to address this issue is transcranial magnetic stimulation (TMS), and like pharmacological inactivation in monkeys, this approach has the advantage of having a relatively good spatial resolution and of precluding long-term compensation, which might hamper the conclusions of clinical studies. Additionally, TMS offers the unique possibility of establishing the time course of the contribution of the studied area to the task at hand, a piece of information normally unavailable from functional imaging studies (but see Gallivan et al., 2011a,b). We have already used TMS to investigate the grasping circuit in details, deciphering both the causal role of the anterior part of the intraparietal sulcus (aIPS) and PMv, and the time course of their respective contribution when planning a grasping movement (Davare et al., 2006, 2007a; Olivier et al., 2007). More recently we applied the same approach to mIPS, one of the key nodes of the reaching circuit, and found that, during motor preparation, this area encodes the direction of goal-directed movements performed towards contralateral targets (Davare et al., 2012).

However, in addition to movement direction, planning a reaching movement also requires specifying its amplitude. Interestingly during the last two decades, many behavioral studies have suggested that movement direction and amplitude are processed separately, by showing, for instance, that the variability and systematic biases of direction and amplitude errors are independent (Gordon et al., 1994; Messier and Kalaska, 1997, 1999; Vindras et al., 2005). However, whereas, in both humans and monkeys, the direction coding in the primary motor cortex (M1), premotor areas, and the PPC has been well documented (Caminiti et al., 1991; Georgopoulos, 1995; Kakei et al., 1999, 2001; Eisenberg et al., 2010; Fabbri et al., 2010), the neural correlates of amplitude coding are less clear (Riehle and Requin, 1989; Kurata, 1993; Riehle et al., 1994; Fu et al., 1995; Desmurget et al., 1998; Messier and Kalaska, 2000). In monkeys, it has been shown that most cells in PMd encode both movement direction and amplitude (Kurata, 1993), and that this coding possibly occurs serially at a single-cell level, with direction coding occurring first during movement preparation, followed by amplitude coding, spreading over movement execution (Fu et al., 1995). More recently, it has been confirmed that cells coding only for movement amplitude are very uncommon in monkey PMd and that most cells encode both direction and amplitude at some points during the performance of an instructed-delay reaching task (Messier and Kalaska, 2000). Similarly in the parietal cortex, recent studies have also found cells encoding amplitude when monkeys performed reaching movements in 3D requiring different depths (Bhattacharyya et al., 2009). Again, direction and amplitude jointly influenced cell-spiking activity in the earlier planning stages of the reach, while amplitude coding became stronger towards movement execution (Hadjidimitrakis et al., 2014). In humans, sensitivity to movement direction has been demonstrated in a large number of cortical areas including M1, PMd and the parietal reach region (PRR; Eisenberg et al., 2010; Fabbri et al., 2010, 2014), with the strongest directional selectivity in the right PRR, then decreasing in the frontal areas (Fabbri et al., 2010). In an attempt to identify cortical areas involved in amplitude coding, Fabbri and collaborators also investigated the sensitivity to movement amplitude in those parietal and frontal regions known to be tuned to movement direction (Fabbri et al., 2012). This study demonstrated that all PPC regions showing directional tuning for reaching movements (inferior parietal lobule (IPL), aIPS, posterior intraparietal sulcus (pIPS) and superior parieto-occipital cortex (SPOC)) are also sensitive to movement amplitude, in contrast with the conclusion of a TMS study showing no evidence for amplitude coding in mIPS (Davare et al., 2012). In addition, the authors reported that the frontal areas, including PMd, show only a partial transfer from the adapted to the non-adapted amplitude, suggesting that the amplitude might be processed differently in parietal and frontal areas (Fabbri et al., 2012). However, these approaches are correlative and a covariation of neural activity with the movement direction and/or amplitude does not prove that the area under investigation is causally involved in processing these parameters (Messier and Kalaska, 2000).

In order to establish the causal contribution of the dorsomedial pathway to the movement amplitude and/or direction coding in humans, we used TMS to interfere with the function of two of the key nodes belonging to the ''reaching'' circuit, namely mIPS and PMd, on both sides. TMS was applied in healthy volunteers while performing goal-directed wrist movements with a manipulandum (Hoffman and Strick, 1986) operated with the right hand. The movements were executed towards visual targets located at different direction and amplitude in both visual hemifields; the targets either remained visible for the whole movement duration (Target-ON) or were flashed for 200 ms before movement onset (Target-OFF). The latter condition was chosen to further validate our conclusion concerning the causal role of mIPS in coding movement direction, since in our previous study, movements were performed under constant visual feedback (Davare et al., 2012). As far as PMd is concerned, we reasoned that if, as suggested in monkeys, this area encodes both movement direction and amplitude, interfering with the functioning of this area should alter these two movement parameters, possibly at different delays during movement preparation or execution. TMS has already been shown to allow the identification of two different, temporally dissociated, movement parameters within the same cortical area (Davare et al., 2007a).

# Materials and Methods

# Subjects

Six healthy right-handed (Oldfield, 1971) subjects (range 23–29 years) with normal, or corrected to normal, vision gave their informed consent to participate in the present study. None had history of neurological disease. Potential risks of adverse reactions to TMS were evaluated by means of the TMS Adult Safety Screen questionnaire (Keel et al., 2001). The present experiment was approved by the local ethical committee of the Université catholique de Louvain.

# Experimental Setup

Subjects sat comfortably in front of a 19 inches computer screen located at a distance of 60 cm. Their right forearm was fastened midway between pronation and supination while the right hand grasped the handle of a two-axis manipulandum (Hoffman and Strick, 1986; Davare et al., 2007b, 2012). A potentiometer placed on each axis of the manipulandum allowed us to measure wrist displacements in the horizontal (flexion-extension (FE)) and vertical (radial-ulnar (RU)) planes. A yellow cursor (0.4◦ wide dot) displayed on the screen continuously indicated the manipulandum position. Eye position was monitored by means of an infrared camera (Thomas Recording GmbH) with a 4 ms temporal resolution. Trials were interrupted whenever a saccade occurred during the fixation period (frequency: 4.1 ± 1.2%, mean ± SD, n = 6). Those aborted trials were then replayed at the end of the experiment until all trials had been executed correctly.

# Transcranial Magnetic Stimulation

Single-pulse TMS was delivered through a 70 mm figure-ofeight coil connected to a Magstim 200 stimulator (Magstim, Whitland, UK). Before each experiment, the resting motor threshold—defined as the intensity for which single-pulse TMS applied over the primary motor cortex produced a wrist movement in 50% of the cases—was estimated while the subjects were comfortably seated with their hand relaxed on their lap. TMS intensity was then set at 120% of the resting motor threshold for the whole experimental session. Trials with and without TMS were randomly intermixed during each experimental block.

We used a neuronavigation technique (Noirhomme et al., 2004; Zosso et al., 2006) to place the TMS coil either over mIPS or PMd in either the left or right hemisphere, as identified in the literature (Connolly et al., 2003; Prado et al., 2005; Davare et al., 2012). Anatomical landmarks were used to guide coil placement: PMd was located as the most caudal portion of the superior frontal gyrus at the level of its intersection with the precentral gyrus; mIPS was located over the medial portion of the IPS, near the caudal part of the angular gyrus (see Davare et al., 2012). The mean normalized MNI coordinates of our stimulations points were −33, −47, 48 and 31, −45, 53 mm for left and right mIPS, respectively, and −21, −6, 68 and 22, −7, 65 for left and right PMd, respectively (x, y, z, n = 6) (**Figure 1A**).

# Experimental Procedure

Each trial started with the wrist in a neutral position, a condition fulfilled when the cursor indicating the position of the manipulandum (yellow circle) was at the center of the screen, indicated by a 15 mm (1.5◦ of field of view) blue square (**Figure 1B**). Subjects were instructed to fixate this central cue at the beginning of each trial. After a 700 ms delay, the central cue was turned off, replaced by a fixation cross, and a 15 mm red square target was randomly displayed in one out of the 20 possible locations. There were 5 possible target locations in each quadrant, corresponding either to a 45◦ direction and to a wrist movement amplitude of 10, 15 or 20◦ , or to a fixed 15◦ movement amplitude in a 33, 45 or 57◦ direction (see **Figure 1B**). We also varied the duration of visual feedback to disentangle online corrective mechanisms from movement planning processes. Targets were displayed either briefly, for 200 ms (Target-OFF condition) or during the whole trial duration (Target-ON condition). Subjects were instructed to move the cursor into the target as rapidly and as accurately as possible. In the Target-ON condition, subjects had to keep the cursor inside the target for at least 700 ms to complete the trial. In the Target-OFF condition, the end of the trial occurred when the cursor velocity dropped below 5% of the velocity peak and remained stable for at least 700 ms. Inter-trial interval varied randomly from 3.5 to 5 s. Throughout the trial, subjects had to maintain eye fixation on the central cross in order to prevent any confounding effects of TMS on eye movements, which could have had indirect consequences on the planning of hand reaching movements.

The experiment consisted of eight blocks of 200 trials in which TMS was applied either over the left or right mIPS or PMd (in separate blocks, 2 blocks each). The order of the 8 blocks was pseudorandomly distributed across subjects. In 80% of the cases, single pulse TMS was delivered either at 100 or 200 ms after target presentation and, in 20% of the cases, no TMS pulse was delivered but the coil remained in place over the current stimulation site.

# Data Acquisition and Analysis

The position signal from the 2 potentiometers was digitized (sampling rate: 1 kHz; PCI-6023E, National Instruments, Austin, TX), and stored on a personal computer for offline analysis. Then, these signals were low-pass filtered offline (16 Hz) with a

Frontiers in Human Neuroscience | www.frontiersin.org May 2015 | Volume 9 | Article 241 |

FIGURE 1 | (A) Location of transcranial magnetic stimulation (TMS) sites over mIPS and PMd in the left and right hemispheres after normalization into the MNI coordinate system (n = 6 for each site). The template used for rendering the cortical surface is Colin27 and each normalized stimulation point has been projected on the template cortical surface using the Brainsight stick tool (Rogue research, Montreal). (B) Location of the 20 possible target locations on the computer screen; only one target was randomly displayed during a given trial. The central square represents the starting point. The horizontal axis is the flexion-extension axis of the wrist, the vertical one the radial-ulnar deviation axis. In each quadrant, there were 5 possible target locations: 3 directions (45 ± 12◦ ) and for the target along the 45◦ axis, 3 possible eccentricities (10, 15 or 20◦ ). Targets were either displayed for the whole movement duration (Target-ON condition) or flashed only for 200 ms (Target-OFF condition).

fourth order, zero-phase-lag, Butterworth filter (see Davare et al., 2007b for details).

We measured the following movement variables: (1) the reaction time (RT), defined as the delay between target onset and the moment when wrist position first exceeded the mean baseline position by 2 SD or more; (2) The movement time (MT), defined as the delay between wrist movement onset and the time the cursor velocity dropped below 5% of the velocity peak and remained stable for at least 700 ms in the Target-OFF condition or inside the target for the Target-ON condition; (3) The displacement ratio (DR), measured by computing the ratio between the total distance traveled by the wrist and the distance between the start and end point of the trajectory. DR provides a reliable estimate of the length of movement trajectory; a DR value equal to 1 corresponds to a straight wrist displacement from the screen center to the target (Davare et al., 2007b, 2012); (4) The value of acceleration and velocity peaks, both known to be linearly related to movement amplitude (Hoffman and Strick, 1986); (5, 6) The mean value and standard deviation (SD) of the initial movement direction were computed to estimate, respectively, the constant (DIRCE) and variable direction errors (DIRVE). The initial movement direction was measured by computing the direction of the velocity vector at the peak acceleration. This parameter allowed us to determine the direction of the movement initially planned, before any visual feedback may take place (Prablanc and Martin, 1992; Desmurget et al., 2005), as indicated by the early occurrence of the peak acceleration (28.7 ± 8 ms in the present study, mean ± SD). DIRCE was computed by taking the difference between the initial movement direction and the direction of the target; (7) Endpoints of step-tracking movements were only computed for the Target-OFF condition. They were measured at the end of the movement and segregated into endpoint constant and variable errors (Desmurget et al., 1999). Endpoint constant errors were computed as the Euclidean distance between the target and endpoint locations. Endpoint variable errors were estimated by measuring the area of the isodensity ellipsoid in which 95% of the endpoints were located; and (8) The shape of the isodensity ellipsoid was determined by computing the ratio between the length of the long and short axes. In order to group the endpoints for the 10 target locations presented in each hemifield, endpoint coordinates were normalized to the 15◦ target amplitude and 45◦ target direction (see **Figure 1B**) by rotating and scaling the horizontal and vertical coordinates of the endpoints.

In order to determine more precisely the time course of the effects of TMS applied over mIPS and PMd on those different movement parameters, each trial was classified with respect to the actual delay between TMS and movement onset and was assigned to a 20 ms time bin (12 bins, ranging from 200 ms before movement onset to 40 ms after movement onset). For each subject and for each bin, an averaged value of the studied parameter was computed, provided that at least 3 data points were available; mean values in each bin were then averaged for all subjects (Davare et al., 2007b). **Figures 2**–**4** only show data for the specific time window TMS was effective (i.e., 100–160 ms bins for mIPS and 40–100 ms bins for PMd.

# Statistical Analysis

First, we performed repeated-measure analyses of variance (ANOVA-RM) for each TMS site separately (left mIPS, right

when no TMS effect was revealed. Only TMS applied over mIPS increased the variability in the initial movement direction, and only for movements directed towards contralateral targets, as shown in purple for left mIPS and in green for

movement onset; the gray shading area highlights data points significantly different from control (no TMS) conditions, shown in the right-hand side of the graph. Error bars illustrate ±1 SD. Results for right mIPS TMS were the same, but only for left targets (not shown).

mIPS, left PMd and right PMd), with TMS DELAY (13 levels: no-TMS and 12 bins), HEMIFIELD (ipsi- or contralateral) and target DISPLAY conditions (Target-ON or Target-OFF) as within-subject factors. In order to increase the number of data points per condition and per subject when analysing the movement variables, we pooled together all targets of

right mIPS (averages were only computed for the specific TMS timing showing a

different eccentricities and directions from the same hemifield. All movement parameters were averaged per condition within each subject. The ANOVA-RM analyses were performed with the SAS Enterprise Guide software, Version 5.1. (Copyright © 2012 SAS Institute Inc., Cary, NC, USA). We performed planned comparisons, investigating selectively the relevant contrasts,

and used Tukey correction for multiple comparisons. Then, for each movement parameter, we performed a global analysis including data from all TMS SITES, including them as additional factor.

We analyzed the effects of target eccentricity (10, 15 or 20◦ ) on the peaks of movement derivatives (velocity and acceleration) by means of multiple linear regressions including subjects dummy variables to take into account the correlation between repeated measures. For this analysis, we pooled all target directions together so as to find the global correlation between these movement derivatives and target eccentricity.

# Results

# Effects of TMS on Initial Movement Direction

TMS applied over the left or right mIPS, but not the left or right PMd, altered MT, DR and DIRVE, in agreement with our previous findings (Davare et al., 2012). For these parameters, we found a significant DELAY × HEMIFIELD interaction for both mIPS sites (all F(12,260) > 3.80, all p < 0.0001 for left mIPS TMS; all F(12,260) > 2.20, all p < 0.012 for right mIPS TMS). Planned comparisons of all bins against the no-TMS condition, for each hemifield, revealed that, TMS applied over the left mIPS, led to an increase in MT, DR and DIRVE only for movements

directed towards contralateral targets (**Figure 2A**), and only when TMS was delivered 160–100 ms before movement onset (all t(260) > 3.06, all p < 0.0025; **Figure 2B**). Identical results were found following right mIPS TMS: MT, DR and DIRVE increased only for movements performed towards contralateral targets (**Figure 2A**) when TMS was applied 160–100 ms before movement onset (all t(260) > 2.54, all p < 0.012). There was no significant main effect of DISPLAY (Target-ON vs. Target-OFF condition) nor DELAY × DISPLAY effect on these movement parameters (DISPLAY: all F(1,260) < 2.0, all p > 0.15; DELAY × DISPLAY: all F(12,260) < 1.6, all p > 0.1). Left or right PMd virtual lesions did not affect MT, DR and DIRVE (**Figure 2A**), as shown by an absence of significant main effect of DELAY or interaction between DELAY and HEMIFIELD on these parameters (all

mIPS and 100–40 ms for left PMd. The ellipses illustrate the 95% confidence interval of endpoint locations. The long axis of ellipses is shown to highlight the direction of the main component of endpoint variability. The black triangles represent DIRVE (as in Figure 2), extrapolated up to the target location (purple

p > 0.2). A global analysis including all TMS sites confirmed a significant SITE × HEMIFIELD × DELAY interaction (F(12,520) = 2.56, p = 0.0027), with a planned comparison confirming that the difference between the no-TMS condition and the 160–100 ms bins was significantly higher for both mIPS sites than for both PMd sites (t(520) = 4.03, p < 0.0001).

the targets, irrespective of their location in the right or left hemifield; this effect is illustrative of an inappropriate amplitude programing. (A) Representative subject, each dot represents an individual trial. (B) Average ellipses across all subjects (n = 6). The dot represents the center of each average ellipse.

# Effects of TMS on Movement Acceleration and Velocity

We found that only TMS applied over left PMd affected the acceleration and velocity peaks; TMS applied over right PMd and over either mIPS had no effect on these parameters. ANOVA-RM showed a significant main effect of DELAY for the acceleration and velocity peaks only when TMS was applied over left PMd (F(12,260) = 9.00, p < 0.001 and F(12,260) = 28.99, p < 0.0001, respectively; **Figure 3A**). Planned comparisons revealed that applying TMS over the left PMd 100–40 ms before movement onset led to an overall increase in the acceleration and velocity peaks (all t(260) > 3, all p < 0.0027, **Figure 3B**); this effect was observed for all target locations (main effect of HEMIFIELD and DELAY × HEMIFIELD interaction: all p > 0.2). TMS applied over left or right mIPS did not affect the peak of velocity or acceleration as denoted by a lack of main effect of DELAY and of interaction between DELAY and HEMIFIELD (all p > 0.18) on these parameters. A global analysis on all TMS sites confirmed a significant SITE × DELAY interaction for both velocity and acceleration peaks (F(36,1040) > 3.29, p < 0.0001), and the planned comparison showed that the difference between the no-TMS condition and the bins between 100 and 40 ms was significantly larger for left PMd than for all other stimulation sites (t(1144) = 2, p = 0.0455). There was no main effect of DISPLAY (Target-ON vs. Target-OFF condition) on acceleration and velocity peaks (F(1,260) < 2.0, p > 0.16).

In the no-TMS condition, linear regressions between acceleration or velocity peaks and target eccentricity showed the typical relationship between the peaks of movement derivatives and movement amplitude (see **Figure 3A**; Acceleration: Intercept = 0.83 × 10<sup>3</sup> , Slope = 5.32, p < 0.001; Velocity: Intercept = 105.15, Slope = 4.304, p < 0.001). When TMS was applied over the left PMd 100–40 ms before movement onset, the intercept of the linear regression increased significantly (acceleration: Intercept = 1.34 × 10<sup>3</sup> , p < 0.001; velocity: Intercept = 147.28, p < 0.001, **Figure 3A**), mainly due to the increased peak of movement derivatives after left PMd virtual lesions. This effect was also found irrespective of target location (HEMIFIELD main effect or SITE × HEMIFIELD interaction: all F < 1) and was comparable for both target display conditions ( DISPLAY main effect or SITE × DISPLAY interaction: all F < 1). In contrast, the slope of the regression line relationship between peak derivatives and target eccentricity was not significantly affected (Acceleration: Slope = 5.902; Velocity: Slope = 4.523, all p > 0.05; **Figure 3A**).

# Effects of TMS on Movement Endpoints

The Target-OFF condition allowed us to separate the effect of TMS on movement planning from the online corrective mechanisms (Prablanc et al., 2003). Indeed in the target-OFF condition, hand movements started well after the target was switched off, which prevented the hand trajectory to be updated based on visual feedback about target location, thus mostly relying on the initial movement plan. The endpoint constant error increased following left PMd virtual lesions only when TMS was delivered 100–40 ms before movement onset (main effect of DELAY : F(12,130) = 21, p < 0.0001, planned comparisons: all t(130) > 1.86, p < 0.0001), irrespective of target location ( HEMIFIELD main effect and DELAY × HEMIFIELD interaction, both F < 1.2, p > 0.3). **Figure 4** shows that following left PMd virtual lesions the step-tracking movements systematically overshot the target whereas the endpoint constant error was not affected by TMS applied over the right PMd nor over left mIPS or right mIPS (not shown). A global ANOVA-RM performed on all TMS sites showed a significant SITE × DELAY interaction (F(36,520) = 6.84, p < 0.0001). A planned comparison confirmed that the difference between the no-TMS condition and the 100–40 ms bins was significantly larger for left PMd than for the other stimulation sites (t(1144) = 4.85, p < 0.0001).

The endpoint variable error was affected when TMS was applied over left mIPS, right mIPS (not shown) or left PMd (all F(12,130) > 28, p < 0.0001; see **Figure 4**). Following left or right mIPS TMS, the endpoint variable errors increased when TMS was applied 160–100 ms before movement onset and for movements directed towards contralateral targets (DELAY × HEMIFIELD interaction: F(12,130) > 29, both p < 0.0001, all planned comparisons, t(130) > 10, p < 0.0001). Importantly, the shape of the isodensity ellipsoid (ratio between the lengths of long and short axes) was different for both left and right mIPS TMS conditions when compared to controls (both t(5) > 5.24, all p < 0.025). For both mIPS TMS conditions, the axis that explains the largest endpoint variability was nearly orthogonal (85 ± 17◦ , mean ± SD, n = 6) to the optimal movement direction (**Figure 4**). This peculiar ellipsoid shape was found only when TMS was delivered 160–100 ms before the onset of movements directed towards contralateral targets (all p < 0.043).

For left PMd virtual lesions, the endpoint variable error increased only when TMS was applied 100–40 ms before movement onset and for movements towards all targets (F(12,130) = 55.64, p < 0.0001; planned comparisons: all t(130) > 10, all p < 0.0001). In contrast to results gathered for mIPS, TMS applied over left PMd led to ellipsoid shapes that extended along the target direction (all t(5) > 6.76, all p < 0.004), with their long axis roughly aligned with the optimal movement path to reach the target (**Figure 4**, 3 ± 12◦ , mean ± SD, n = 6). Finally, a global ANOVA-RM on endpoint variable error confirmed a significant SITE × HEMIFIELD × DELAY effect (F(36,520) = 17.01, p < 0.0001). We performed a planned comparison highlighting the specific mIPS effect, comparing the no-TMS vs. 160–100 ms bins difference between mIPS and PMd (t(520) = 3.09, p = 0.0021) and another comparison focused on the specific effect obtained on left PMD (no-TMS vs. 100–40 ms bins, left PMD vs. other areas: t(520) = 10.19, p < 0.0001).

# Discussion

The present study indicates that direction and amplitude coding of goal-directed movements is performed by two distinct areas belonging to the dorsomedial ''reaching'' pathway, with direction being processed, first, in the posterior parietal cortex (mIPS) and amplitude being implemented later in the premotor cortex (PMd). In addition, we found that both mIPS are involved in coding the direction of right hand movements when performed in the contralateral hemifield, whereas left PMd processes the amplitude of all right hand displacements, whatever the location of the target, in the right or left hemifield. These results shed new light onto the cascade of visuo-motor transformations performed in the dorsomedial pathway of the prehension circuit.

The finding that both mIPS process the direction of goaldirected movements performed towards contralateral targets during movement preparation corroborates and extends our previous study (Davare et al., 2012), by showing that, when no visual feed-back was available during the task performance (Target-OFF condition), the movement endpoint distribution was biased along a direction orthogonal to the optimal movement path, and again only for movements directed towards contralateral targets and within a precise time window (160–100 ms) before movement onset. In our previous study, a series of control studies allowed us to conclude that the effect of mIPS TMS on movement direction could not be explained by an incorrect target spatial representation but was due to an inaccurate computation of the reach ''motor'' vector, or more precisely of the direction of this vector since our previous, and current, results have failed to reveal any evidence that mIPS is involved in coding the norm (i.e., amplitude) of this movement vector (Davare et al., 2012). Importantly, because we were able to demonstrate a deficit in coding the movement amplitude when TMS was applied over PMd (see below), the negative results we obtained following mIPS TMS do not results from a lack of accuracy or sensitivity of the measurements and/or analyses we performed.

As previously mentioned in the Introduction, the contribution of the PPC to movement direction coding has already been suggested in humans, in particular by using fMRI adaptation protocols (Fabbri et al., 2010). In this study, the same movement was repeated several times in one particular direction, followed by a test trial consisting of a movement executed in a different direction. The rationale of this approach is that, if the investigated area contains a directionally tuned cell population, its activation during the test trial should be proportional to the angular difference between the adapted and tested directions (Fabbri et al., 2010). Such an approach has allowed the authors to identify an extensive network of cortical areas sensitive to the direction of reaching movements, namely bilateral PMd, mIPS, aIPS and PRR (Fabbri et al., 2010), but also SMA and anterior precuneus (Fabbri et al., 2012); additionally, an interaction between reach direction and grip type has been found in a large number of areas belonging to the dorsolateral ''grasping'' circuit, including PMv (Fabbri et al., 2014). The large number of areas that this technique has revealed as being directionally sensitive, together with the fact that eye movements and/or attention allocation might have influenced these results, raises the issue of the causal role of all these areas in movement direction processing. Another possible drawback of these studies is that they did not try to disentangle movement direction from the other parameters that systematically covary with it, such as kinematics, EMG activity, or pattern of joint rotations. In any case, a modulation of the BOLD signal with movement direction alone cannot be viewed as evidence for a causal role in coding this parameter, as proved by the discrepancy between, for example, the lack of evidence for direction coding in PMd (current study) and the results of these fMRI adaptation studies, all showing consistently that this area contains directionally tuned cell populations.

Surprisingly, a lot fewer studies have investigated the role of cortical areas in coding movement amplitude. In monkeys, cells coding for amplitude have been mainly reported in PMd, but it has been repeatedly shown that most PMd cells encode both direction and amplitude of reaching movements (Kurata, 1993), in a sequential order, with movement direction being coded first, then followed by amplitude (Fu et al., 1995). The scarcity (around 2–4%) of cells coding only for movement amplitude in PMd has been confirmed by the group of Kalaska, supporting the view that most PMd cells in monkeys have an activity serially related to both direction and amplitude, during the performance of an instructed-delay reaching task (Messier and Kalaska, 2000). More recently, it has been shown in the monkey that the activity of most cells in V6A is also modulated by both depth and direction during reaching (Hadjidimitrakis et al., 2014), in contrast with the conclusion of the only study showing the existence of distinct cell populations coding the distance, azimuth and elevation in area 5 in monkey PPC (Lacquaniti et al., 1995).

Likewise, in humans, only very few studies have directly explored the coding of movement amplitude, possibly because of the even larger number of movement parameters that covary with amplitude (Messier and Kalaska, 2000). Recently, Fabbri and collaborators investigated the sensitivity to movement amplitude in those parietal and frontal regions already known to be tuned to movement direction (Fabbri et al., 2012). This study demonstrated that all PPC areas directionally tuned for direction (IPL, aIPS, pIPS and SPOC) are also sensitive to movement amplitude, suggesting they contain cell populations tuned to specific combinations of direction and amplitude. In addition these authors reported that the frontal areas, including PMd, PMv and SMA, show a partial transfer of adaptation to movement direction from the large to the small movement amplitude, but not the opposite, suggesting that, although this result remains puzzling, the amplitude is somehow processed by frontal areas, but in a different way than in the PPC areas (Fabbri et al., 2012).

The present study fails to find any evidence that mIPS is causally involved in amplitude coding, although of course we cannot exclude that another parietal area belonging to the ''reaching'' cortical circuit (Vesia and Crawford, 2012) codes movement amplitude. This finding corroborates the conclusion of our earlier TMS study showing no evidence for amplitude coding in mIPS (Davare et al., 2012). However, because this earlier conclusion was based on indirect evidence gathered by analyzing velocity and acceleration peaks, it remained questionable. To address this issue, in the current study, we tested an additional open-loop condition in which the target was flashed only for 200 ms, allowing us to prove that the distribution of endpoint errors after mIPS TMS was compatible with errors in coding direction, and not compatible with an error in movement amplitude, in contrast to what we found for PMd TMS.

From a more theoretical perspective, the current study reopens the long-lasting debate about a separate coding for amplitude and direction, a view mainly supported by behavioral studies (Desmurget et al., 1998), but for which neural evidence was still lacking, at least at the cortical level. At a behavioral level, a series of findings concur in suggesting independence of treatment for amplitude and direction: RT is decreased by prior information about either the direction or the distance of the target with respect to the hand (Rosenbaum, 1980; Bock and Arnold, 1992; Desmurget et al., 2004); rotation and gain learning occur at different paces and have different patterns of generalization (Pine et al., 1996; Krakauer et al., 2000; Vindras and Viviani, 2002); specification of movement amplitude and direction follow different time courses (Favilla et al., 1989; Ghez et al., 1997); variability and systematic biases of direction and amplitude errors are independent (Gordon et al., 1994; Messier and Kalaska, 1999; Vindras et al., 2005). The current TMS study demonstrates for the first time that such an independent coding for direction and amplitude exists at the cortical level. This finding echoes the results of previous studies that have linked amplitude coding with the basal ganglia (BG) network (Desmurget et al., 2003, 2004; Desmurget and Turner, 2008). Indeed, BG activity is known to modulate neural response in PMd (Grafton et al., 2006). It is thus tempting to speculate that BG inputs mediate the influence of PMd on movement amplitude (or any covariate of this parameter; e.g., velocity, acceleration, force). This view is compatible with recent evidence that the BG modulates movement performance according to non-motor motivational factors (Mazzoni et al., 2007; Turner and Desmurget, 2010; Baraduc et al., 2013). It is also interesting to note that the BG network has been involved in coding force amplitude during grasping movements, likely via BG connections to the dorsolateral grasping circuit including AIP and PMv (Wasson et al., 2010). This highlights that the BG are a key node for coding the amplitude not only for reaching movements via interactions with the dorsomedial circuit but also for force scaling via interactions with the dorsolateral grasping circuit (see Prodoehl et al., 2009 for review). In addition, to this point, and in agreement with electrophysiological experiments performed in monkeys (Fu et al., 1995; Messier and Kalaska, 2000; Hadjidimitrakis et al., 2014), our results also confirm that the amplitude is processed later than the direction information during reach movement preparation, suggesting that these two parameters are processed, or at least implemented in a serial order in the motor-related cortical areas. We can speculate that a serial encoding of direction and amplitude is likely to reflect a motor control strategy leading to smoother movement generation. First selecting a specific agonist muscle group (to reach a particular direction in space) before determining the exact amount of muscle activity required to reach a given distance seems a more cost-saving strategy. Interestingly, the current study also suggests that the processing of motor intention signals evolves along the parieto-frontal circuit: whereas the left or right mIPS only encodes preparatory signals for movements directed towards targets located in the contralateral hemifield,

# References


the coding in PMd appears more closely linked to the effector i.e., the contralateral hand, irrespective of the target location. Nevertheless, an additional study in which the left and right hands are systematically tested will be necessary to substantiate this conclusion.

To summarize, the present study provides, for the first time, evidence for a double-dissociation between direction and amplitude coding of reaching movements within the dorsomedial reaching circuit in humans. It is noteworthy that another candidate area for playing a causal role in encoding amplitude within the human dorsomedial circuit could be the human homolog of V6A (Pitzalis et al., 2013). Since Ciavarro et al. (2013) have found endpoint amplitude errors in reaching movements when TMS was applied over that area. Interestingly this effect seemed related more to a disruption of the visuospatial target representation rather than to the motor representation of target amplitude such as following PMd TMS. Further studies are required to substantiate the existence of a visuomotor gradient of amplitude encoding in parieto-frontal networks. Additional TMS experiments are also needed to investigate the possible interactions between the dorsomedial reaching and dorsolateral grasping circuits because, so far, these two components of prehension movements remain frequently investigated by using separate experimental paradigms.

# Acknowledgments

This work was performed at the Institute of Neuroscience (IoNS) of the Université catholique de Louvain (Brussels, Belgium); it was supported by grants to EO from the ARC (Actions de Recherche Concertées, Communauté Française de Belgique), from the Fondation Médicale Reine Elisabeth (FMRE) and from the Fonds de la Recherche Scientifique (FNRS–FDP). MD is funded by a BBSRC David Phillips fellowship (UK), the Royal Society (UK) and a FWO Odysseus project (Fonds Wetenschappelijk Onderzoek, Belgium). AZ is a Senior Research Associate supported by INNOVIRIS.


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Davare, Zénon, Desmurget and Olivier. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Corrigendum: The development of trunk control and its relation to reaching in infancy: a longitudinal study

Jaya Rachwani <sup>1</sup> \*, Victor Santamaria<sup>1</sup> , Sandra L. Saavedra<sup>2</sup> and Marjorie H. Woollacott <sup>1</sup>

*<sup>1</sup> Department of Human Physiology and Institute of Neuroscience, University of Oregon, Eugene, OR, USA, <sup>2</sup> Department of Rehabilitation Sciences, University of Hartford, West Hartford, CT, USA*

Keywords: infant development, independent sitting, posture, trunk control, reaching, EMG

# **A corrigendum on**

**The development of trunk control and its relation to reaching in infancy: a longitudinal study** by Rachwani, J., Santamaria, V., Saavedra, S. L., and Woollacott, M. H. (2015). Front. Hum. Neurosci. 9:94. doi: 10.3389/fnhum.2015.00094

**Figure 4** of the article by Rachwani et al. (2015) contained a minor error, which we hereby rectify. In the original figure, the graph displaying the number of movement units across sitting development is incorrect (bottom graph on the left column). We therefore re-submit **Figure 4** with the correct graph. We sincerely apologize for the inconvenience.

Edited and reviewed by: *Gavin Buckingham, Heriot-Watt University, UK*

> \*Correspondence: *Jaya Rachwani, rachwani@uoregon.edu*

Received: *27 June 2015* Accepted: *01 July 2015* Published: *21 July 2015*

#### Citation:

*Rachwani J, Santamaria V, Saavedra SL and Woollacott MH (2015) Corrigendum: The development of trunk control and its relation to reaching in infancy: a longitudinal study. Front. Hum. Neurosci. 9:406. doi: 10.3389/fnhum.2015.00406* Acknowledgments

This work was supported by the National Institutes of Health Grant 1R01HD062745-01, MW, principal investigator, and by the Eugene Evonuk Memorial Graduate Fellowship for JR.

# References

Rachwani, J., Santamaria, V., Saavedra, S. L., and Woollacott, M. H. (2015). The development of trunk control and its relation to reaching in infancy: a longitudinal study. Front. Hum. Neurosci. 9:94. doi: 10.3389/fnhum.2015.00094

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Rachwani, Santamaria, Saavedra and Woollacott. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Muscle co-activity tuning in Parkinsonian hand movement: disease-specific changes at behavioral and cerebral level

A. M. M. van der Stouwe<sup>1</sup> \*, C. M. Toxopeus <sup>1</sup> , B. M. de Jong<sup>1</sup> , P. Yavuz <sup>1</sup> , G. Valsan<sup>2</sup> , B. A. Conway <sup>2</sup> , K. L. Leenders <sup>1</sup> and N. M. Maurits <sup>1</sup>

*<sup>1</sup> Department of Neurology, University Medical Center Groningen, University of Groningen, Groningen, Netherlands, <sup>2</sup> Biomedical Engineering, University of Strathclyde, Glasgow, UK*

#### Edited by:

*Simona Monaco, University of Trento, Italy*

#### Reviewed by:

*Krystal Lynn Parker, University of Iowa, USA Elisabeth Huis in 't Veld, Maastricht University, Netherlands Ahmed Mostafa, York University, Canada*

#### \*Correspondence:

*A. M. M. van der Stouwe, Department of Neurology, University Medical Center Groningen, Hanzeplein 1, 9700 RB, Groningen, Netherlands a.m.m.van.der.stouwe@umcg.nl*

> Received: *06 October 2014* Accepted: *17 July 2015* Published: *05 August 2015*

#### Citation:

*van der Stouwe AMM, Toxopeus CM, de Jong BM, Yavuz P, Valsan G, Conway BA, Leenders KL and Maurits NM (2015) Muscle co-activity tuning in Parkinsonian hand movement: disease-specific changes at behavioral and cerebral level. Front. Hum. Neurosci. 9:437. doi: 10.3389/fnhum.2015.00437* We investigated simple directional hand movements based on different degrees of muscle co-activity, at behavioral and cerebral level in healthy subjects and Parkinson's disease (PD) patients. We compared "singular" movements, dominated by the activity of one agonist muscle, to "composite" movements, requiring conjoint activity of multiple muscles, in a center-out (right hand) step-tracking task. Behavioral parameters were obtained by EMG and kinematic recordings. fMRI was used to investigate differences in underlying brain activations between PD patients (*N* = 12) and healthy (age-matched) subjects (*N* = 18). In healthy subjects, composite movements recruited the striatum and cortical areas comprising bilaterally the supplementary motor area and premotor cortex, contralateral medial prefrontal cortex, primary motor cortex, primary visual cortex, and ipsilateral superior parietal cortex. Contrarily, the ipsilateral cerebellum was more involved in singular movements. This striking dichotomy between striatal and cortical recruitment vs. cerebellar involvement was considered to reflect the complementary roles of these areas in motor control, in which the basal ganglia are involved in movement selection and the cerebellum in movement optimization. Compared to healthy subjects, PD patients showed decreased activation of the striatum and cortical areas in composite movement, while performing worse at behavioral level. This implies that PD patients are especially impaired on tasks requiring highly tuned muscle co-activity. Singular movement, on the other hand, was characterized by a combination of increased activation of the ipsilateral parietal cortex and left cerebellum. As singular movement performance was only slightly compromised, we interpret this as a reflection of increased visuospatial processing, possibly as a compensational mechanism.

Keywords: Parkinson's disease, step-tracking, fMRI, EMG, kinematic parameters, muscle co-activity

**Abbreviations:** ANOVA, analysis of variance; AI, activation index; BA, Brodmann area; BG, basal ganglia; DQ, differentiation quotient; EMG, electromyography; fMRI, functional magnetic resonance imaging; M1, primary motor cortex; m. ECRB, (musculus) extensor carpi radialis brevis; m. ECRL, (musculus) extensor carpi radialis longus; m. ECU, (musculus) extensor carpi ulnaris; m. FCR, (musculus) flexor carpi radialis; PD, Parkinson's disease; PMC, premotor cortex; PV, peak velocity; ROI, region of interest; RT, reaction time; SMA, supplementary motor area; UPDRS, Unified Parkinson's Disease Rating Scale.

# Introduction

The direction of voluntary hand movement along the wrist originates from cerebrally encoded vectors, without a direct link to specific muscles to effectuate their contraction. Regarding the effector system, however, movement in some directions is dominated by activity of only one agonist muscle, while other directions require coordinated simultaneous activation, or, "co-activation" of multiple agonists. This directional tuning highlights the highly adaptable outflow structure of motor commands that underpin goal directed movements. The basal ganglia (BG) are known to modify the cortically generated motor plan by selecting appropriate muscles and inhibiting undesired motor activity (Alexander et al., 1986; Mink, 1996; Middleton and Strick, 2000; Rubchinsky et al., 2003; de Jong and Paans, 2007).

In studies addressing these aspects of direction tuning in motor control, center-out step-tracking tasks (Hoffman and Strick, 1999) are commonly used. In such tasks, subjects are required to make hand excursions into various directions. By combining this motor paradigm with functional brain imaging, we aimed to demonstrate that the BG play an important role in the organization of tuned muscle co-activity. We hypothesized that increased BG activity would be found in movement excursions requiring multiple muscles to be simultaneously active (co-active), which we defined as "composite movements," to be distinguished from "singular movements" requiring the activity of one dominant agonist. Given the association between pathophysiological BG changes and characteristic movement impairments in Parkinson's disease (PD) (DeLong and Wichmann, 2009), we included PD patients, expecting to find reduced BG activity during movements requiring highly tuned muscle co-activity, when compared to healthy subjects. This concept finds support from the observation that muscle tuning is indeed impaired in PD as patients show insufficient inhibition of antagonist muscles, which causes co-contraction of agonist and antagonist muscles (Meunier et al., 2000). To gain insight in the impaired selection of highly tuned muscle co-activity in PD patients, the here employed center-out step-tracking task thus enabled the comparison of movements executed with different degrees of muscle tuning between PD patients and healthy subjects both at behavioral level, using kinematic and electromyography (EMG) parameters, and at the cerebral level by using functional magnetic resonance imaging (fMRI).

The center-out step-tracking task employed a manipulandum that enabled measurement of hand movement along the wrist, made toward eight different targets. A priori, we made a distinction between movement directions requiring either more or less muscle co-activity based on what is known from previous work on step-tracking (Hoffman and Strick, 1999). Therefore, the first step of the present study was to validate the distinction between composite and singular movements in healthy subjects. In addition to the EMG data, we analyzed kinematic parameters to confirm that a pattern of composite muscle activity indeed results in a movement profile that differs from a singular muscle activity pattern. Finally, differences in brain activation patterns related to these tasks were assessed with fMRI using an eventrelated design. This entails, that we contrasted composite and singular movements to identify differences between PD patients and healthy subjects regarding the cerebral organization of movement with different degrees of muscle tuning. We expected to find impaired performance by PD patients, reflected in increased reaction times and more extensive muscle co-activity, while at the cerebral level, we hypothesized to find reduced brain activation in specifically the BG and interconnected circuitry. To our knowledge, this study is the first to use a center-out step-tracking task for the purpose of investigating muscle tuning organization on an output level (EMG and kinematics) as well as on brain (organizational) level in PD patients as well as in healthy subjects.

# Methods

# Subjects

The study was approved by the Medical Ethical Committee of the University Medical Center Groningen (UMCG). Patients were recruited at the outpatient clinic for movement disorders at the UMCG and healthy subjects were recruited by advertisements in local newspapers. Subjects participated after full explanation of the study's purpose, protocol and risks, and provided informed consent in accordance with the Declaration of Helsinki (2008). All subjects participated in two experimental sessions, the second of which included fMRI. Twelve patients with idiopathic PD experiencing mild to moderate clinical symptoms were recruited. Patients were assessed by the Unified Parkinson's Disease Rating Scale (UPDRS) (Fahn et al., 1987), and Hoehn–Yahr disability scale (Hoehn and Yahr, 1967). In addition, 18 healthy gender and age matched subjects were recruited. Patients had to have a stable response to medication, and to reduce medication effects, had to refrain from taking their morning dose of levodopa, or dopamine agonists (overnight withdrawal). All subjects had to be right handed as assessed by the Annett Handedness Scale (Annett, 1970). Exclusion criteria for both groups were a history of epileptic seizures, head injury, neurological diseases (for patients: other than PD), psychiatric diseases, or the use of any type of medication affecting the central nervous system. Also, during a brief neurological physical examination it was ensured that subjects had (corrected-to-) normal vision. Patients who could either not abstain from their levodopa use or had a Mini Mental State Examination (MMSE, Cockrell and Folstein, 1988) score <26 were excluded. Patients with Parkinsonism other than PD or the tremor-dominant type of PD, which might be regarded as a PD subtype (Josephs et al., 2006), were also excluded to obtain a maximally homogeneous group. Subjects came in for the behavioral and fMRI experiments on two separate days, with a maximum interval of two weeks. During the first visit, subjects performed the task in sitting position and additionally practiced one block of the task in the supine position inside a dummy MR scanner. During the second visit subjects practiced task performance prior to fMRI data collection, again for one block.

# Experimental Set-up

All subjects performed a visual step-tracking task with the right hand, using a magnetic resonance (MR) compatible manipulandum (**Figure 1**). The applied manipulandum is a

joystick-like device that can rotate in two perpendicular planes allowing all combinations of wrist flexion-extension and wrist ulnar-radial deviation. Subjects were comfortably positioned with the right arm supported by an armrest. The hand was positioned in a vertical plane and subjects grasped the manipulandum handle. The right wrist joint was positioned in the center of the two concentric rings composing the device. The fingers were taped to the thumb reminding subjects to hold the grip with all fingers. Prior to the start of each block of steptrack movements subjects were requested to hold their wrist in a neutral position, i.e., in the center of the manipulandum, and the center of the screen was adjusted to the position of the cursor corresponding to this neutral position (center point on screen). This was done to make sure that anatomic variation of hands did not interfere with task execution. The range of wrist movement from this position was checked to ensure that subjects were able to move freely in each direction. To provide visual feedback on task performance, angular displacement was measured in both (X and Y) planes by potentiometers mounted in-line with the axes of the manipulandum rings and displayed as a cursor (a 5×5 mm closed square) following digitization using a Power 1401 analog-to-digital converter controlled using Spike 2 [Cambridge Electronic Design (CED), Cambridge, UK].

To investigate kinematic and EMG results for all movements, data of the behavioral experiment were used. Pilot experiments comparing performance of the step-tracking task in sitting and supine positions had shown that there were no differences in kinematic and EMG data between the two positions and the EMG data from the behavioral experiment are not distorted by fMRI-related artifacts. During scanning, subject performance was visually monitored by a second computer in the MR control room.

# Task

Subjects were asked to place their cursor in the "center box" (3 × 1.5 cm open rectangle). A warning cross preceding the appearance of the target was displayed in this center box for 1 s. After disappearance of the warning cross, a target stimulus (3 × 1.5 cm open rectangle) appeared at one of eight possible positions (**Figure 2C**). The time intervals between warning cross and target were randomized (jitter: 0.8 ± 0.4 s). All eight directional stimuli had the same distance relative to the center (20◦ ) of the screen and were equally spaced. Regarding the hand position in the manipulandum, movements in 0◦ and 180◦ directions corresponded with extension (right) and flexion (left), respectively, whereas movements in 90◦ and 270◦ directions corresponded with radial (up) and ulnar (down) deviation, respectively.

After moving toward the target, subjects were required to hold the cursor in the target box until the target box disappeared (3 s after appearance of the target stimulus) whereupon they return to the neutral (center) position. Each of such movement trials, which was coined a full step-track movement, lasted 5 s. After every 10 step-tracks, there was a short break of 4 s. One step-track block consisted of 40 stimuli, five for each of the each different directions presented in fixed randomized order (randomized but in the same order for every subject). The entire task consisted of four blocks.

# A Priori Division into Composite and Singular Movement

To discriminate between movements that require a higher level of muscle co-activity and movements requiring less muscle coactivity, the eight movement directions were a priori divided in two groups, based on the step-tracking study of Hoffman and Strick (1999). We used a profile of scaled EMG activity of the same four arm muscles we investigated as a template to determine the number of muscles actively contributing to movement for each direction (**Figures 2A,B**). Muscles were regarded to significantly contribute to distinct movement directions when activity was larger than 50% of the maximum scaled EMG activity over all directions for that muscle; meaning that at least half of the muscle's peak level of agonist burst activity was required for movement in that particular direction. Since Hoffman and Strick investigated 12 directions, the number of muscles contributing to the four movement directions in our study (45◦ , 135◦ , 225◦ ,

and 315◦ ) were determined by averaging the number of muscles contributing to movement in their directions 1 and 2, 10 and 11, 7 and 8, and 4 and 5 o'clock, respectively. Movement directions involving multiple (>1) lower-arm muscles were regarded as requiring highly tuned muscle coordination during the initial agonist burst and were labeled "composite movements." The second group of movement directions, dominated by activity in single agonist muscles, was labeled "singular movements." This resulted in a balanced decomposition into two groups of four movement directions each; a "composite" group with movements directed in 0◦ , 45◦ , 135◦ and 315◦ , and a "singular" group consisting of movement directions 90◦ , 180◦ , 225◦ , and 270◦ (**Figure 2C**). Note that for example movement to direction 0◦ (full wrist extension) is considered a composite movement, since accomplishing movement in this direction requires co-activity of multiple extensor muscles (Hoffman and Strick, 1999). Realizing that this a priori division in singular and composite movements is based on a single human dataset, we later validated this division using our own EMG data in healthy subjects (see Section EMG Data).

# EMG Data Recording

To record muscle activity, four bipolar Ag/AgCl surface electrode pairs were placed on the lower (right) arm muscles m. Extensor carpi radialis longus (m. ECRL), m. Extensor carpi radialis brevis (m. ECRB), m. Flexor carpi radialis (m. FCR), and m. Extensor carpi ulnaris (m. ECU). A reference electrode was placed on the dorsal side of the left hand. To improve skin conductance, the skin was pre-treated with a scrub gel and subsequently a conductive paste was applied. EMG electrodes were placed longitudinally with respect to the muscle fibers and attached approximately 1.5 cm apart. The muscles were identified by palpation, using maximum voluntary contractions (EMG) toward the specific pulling direction of each individual muscle. To diminish cross-talk, we verified that movement toward the pulling direction mainly elicited activity in the EMG channel belonging to that specific muscle. EMG data were recorded at 5000 Hz using Brain Vision Recorder software (Brain Products GmbH Munchen, Germany).

# MRI Characteristics

fMRI data acquisition was performed using a 3 Tesla Magnetic Resonance System (Philips, Best, Netherlands) with a standard six-channel head coil. T2<sup>∗</sup> -weighted, 3D functional images were obtained using multislice echo planar imaging (EPI) with an echo time (TE) of 30 ms and a repetition time (TR) of 2000 ms. Per TR 39 axial slices, with a field of view (FOV) of 224 mm, flip angle of 5◦ with a 64 × 64 matrix and isotropic voxel size of 3.5 × 3.5 × 3.5 mm were acquired. Functional scanning included 106 volumes per block. To provide anatomical information (isotropic voxel size 1 × 1 × 1 mm), additional T1-weighted 3D anatomical scans with an axial orientation and a matrix size of 256×256 mm were obtained.

# Analysis of Kinematic and Electromyography Data

## Kinematic Data

Kinematic parameters for comparison of task execution between groups were derived using the X and Y displacement measured by the two potentiometers integrated in the manipulandum. The kinematic data were further analyzed using Matlab (Matlab R2007b, Mathworks, Natrick, USA). A custom-made script was used to determine a set of kinematic variables. For each individual subject reaction time (RT) and peak velocity (PV) were determined for each movement. RT was determined as the time (in ms) between stimulus presentation and movement onset. Movement onset was identified visually by a sudden change in total displacement (<sup>√</sup> X<sup>2</sup> + Y<sup>2</sup> ) of the manipulandum. PV was determined by the maximum of the velocity, calculated as the numerical first-order derivative of the total displacement, in degrees per second. Means and standard deviations (as a measure of variability) of RT and PV per direction of movement were calculated per subject.

arrows, singular movements as gray, dashed arrows.

# EMG Data

EMG data were exported to Matlab, where they were down sampled to 100 Hz, high-pass filtered (Butterworth Zero Phase shift filter with a cut-off of 10 Hz) and full-wave rectified (Meyers et al., 2003) by using a custom made script. To enable comparison of relative EMG activity between subjects, EMG data were normalized by the maximum EMG over all experimental trials for each muscle and subject. Next, we calculated the number of muscles that significantly contributed to movement in each direction, for each of our subjects. This was primarily done to verify the a priori division into singular and composite movements (see Section Task) and, secondly, to determine differences in the number of involved muscles between groups. Muscles were regarded as significantly contributing to movement in a specific direction when reaching a cut-off value of 0.5 (scaled EMG activity). The number of active muscles, indicating the amount of muscle co-activation, was further referred to as the activity index (AI) which could range theoretically from 0 to all 4 muscles.

To quantify the extent of specialized muscle activity, we calculated the differentiation quotient (DQ) by dividing the mean scaled EMG activity of the direction in which a muscle was most active by the mean scaled EMG activity of that muscle for the seven remaining directions. This was accomplished for each muscle and individual subject separately. DQ, thus, provided insight in whether a muscle was specifically active in a distinct direction, or equally active in multiple directions, i.e., a higher DQ indicated a higher extent of specialized activity for a specific muscle, whereas a lower DQ corresponded with a less specialized activity pattern of that muscle.

# Statistics

Statistical analysis was performed using PASW 18 (SPSS, Inc., Chicago IL). First, we used the Shapiro–Wilk test of normality to check the distribution of the data. Kinematic variables that were not normally distributed and were right-skewed were transformed using a Log<sup>10</sup> transformation (in case data were not normally distributed for one group, data of both groups were transformed). Separate mixed design ANOVAs were employed to assess general significant differences for all kinematic variables (PV and RT), and EMG variables AI and DQ. Before performing mixed ANOVAs, the assumption of sphericity was tested on each variable using Mauchly's test. If the assumption was rejected the Greenhouse–Geisser correction was applied. The betweensubject variable for the mixed ANOVAs was "group," (two levels: PD patients and healthy subjects). For kinematic parameters and AI, the within-subject variable was "movement direction" (two levels: "composite" and "singular" movement). For DQ, the within-subject variable was "muscle" (four levels: "m. ECU," "m. ECRL," "m. ECRB," and "m. FCR"). Main effect of muscle was further investigated employing Bonferroni corrected pairwise comparisons. The significance level was set at α = 0.05.

# fMRI Data Analysis

Processing of images and statistical analyses were conducted using Statistical Parametric Mapping (SPM) version 5 (2005, Wellcome Department of Cognitive Neurology, London, UK; http://www.fil.ion.ucl.ac.uk/spm). Pre-processing included standard slice time correction, realignment and co-registration of functional and anatomical scans. Images were normalized to the template of the Montreal Neurological Institute (MNI). Next, images were smoothed using a Gaussian filter of 8 mm full width at half maximum (FWHM). Event related analysis was performed; events were defined as the appearance of peripheral target stimuli in the step-tracking task. Brain activations were computed according to the standard statistical procedures in SPM. Statistical parametric maps per subject (first level analysis) were derived using a linear multiple regression model with event-related regressors and movement parameters as regressors of no interest to account for head movement-related effects. Scans were checked for head motion; all scans had maximally 3 mm translational motion and maximally 0.08◦ rotational motion. Two comparisons (T-contrasts) between the two types of directions (Composite > Singular and Singular > Composite) were generated at first level. The activation maps of the two between-task comparisons at first level were entered in separate ANOVAs (flexible factorial design) for initial statistical analysis of differences within the group of healthy subjects. These first level results were further used for statistical analysis of differences between groups at second level. To enable comparison of taskrelated differences between patients and healthy subjects, we used exclusive masking with a threshold of p = 0.05. Note that exclusive masks remove all voxels reaching significance in one contrast that overlap with the significant voxels in the other contrast, thereby enabling direct comparison of differences in activation patterns between healthy subjects and patients.

We primarily looked for effects in the BG/thalamus, premotor cortex (PMC), supplementary motor area (SMA), parietal cortex and cerebellum. Previous studies indicated that these specific areas are subject to changes related to PD (Playford et al., 1992; Samuel et al., 1997; Sabatini et al., 2000; Yu et al., 2007; Ma et al., 2009). We therefore determined a restricted volume including the BG and thalamus for statistical analysis with a small volume correction. This small volume was obtained by using a spherical volume of interest (VOI) [radius of 30 mm (15 voxels)] with a center placed at coordinate [0, 0, 0]. For changes of activation in the areas of interest, we used a threshold for voxel response height of p = 0.01 (cluster uncorrected and extent threshold of k = 30 voxels). To identify effects in cortical areas as well as in the cerebellum, voxel values were thresholded at a voxel response height of a liberal p = 0.01 (uncorrected) with an extent threshold of k = 10 voxels.

Activations in other regions were reported only when p < 0.001 (uncorrected, extended voxel threshold of k = 10 voxels). Activated brain regions were identified by rendering group activation maps onto the Automated Anatomical Labeling (AAL) template and Brodmann template in MRICron (Rorden et al., 2007).

# Results

# Subjects

Twelve PD patients [mean age: 59 ± 9 (range 38–69)] and 18 healthy subjects [mean age: 59 ± 5 (range 51–69)] participated in this study. One healthy subject was excluded from the behavioral part of the study, due to a technical problem that occurred while recording the kinematic data. An independent samples T-test revealed there were no significant age differences between patients and healthy subjects (p = 1.000). The clinical characteristics of the youngest patient were similar to those of the older patients. Moreover, this patient was not known to have genetic mutations and was therefore included despite her young age. A Mann–Whitney U-test showed that the gender distribution was similar between groups [7/12 male (PD), 9/18 male (controls), p = 0.723]. Similar testing ascertained that MMSE scores were also comparable between groups; the median MMSE score was 29 for PD patients, and 29 for healthy subjects (p = 0.113). The symptomatic state of all patients was described by their UPDRS and Hoehn and Yahr scores (see **Table 1**). Regarding the laterality of rigidity in PD patients, in 8/12 patients the right arm was more affected (difference in UPDRS of 1 point), in 2/12 patients the left arm was more affected (difference in UPDRS of 1 point). In 2/12 patients severity of rigidity did not differ between arms.

# Kinematic Results Reaction Time

Regarding median reaction time, we found an ordinal interaction effect between group and movement direction [F(1, 33) = 5.189, p = 0.029], which indicated that the difference in RT between composite and singular movements was larger in PD patients than in healthy subjects. There was no interaction effect regarding RT variability. Overall, composite movements required longer RTs [main movement direction effect; F(1, 33) = 32.126, p < 0.001] and resulted in higher RT variability [main movement direction effect; F(1, 33) = 9.466, p = 0.004]. Moreover, PD

TABLE 1 | Clinical characteristics.


*MMSE, Mini Mental State Examination; UPDRS, Unified Parkinson's Disease Rating Scale; H&Y, Hoehn and Yahr disease stage; Lat. Rigidity, laterality of rigidity, e.g., left or right arm. LEDD, levodopa equivalent daily dosis* = *levodopa dose (mg)* + *(0.3* \* *levodopa dose if using entacapone with each dose)* + *(slow release levodopa* \* *0.7)* + *(bromocriptine* \* *10)* + *(ropinirole* \* *20)* + *(pergolide* \* *100)* + *(pramipexole* \* *100)* + *(apomorphine* \* *10) (Esselink et al., 2004).*

patients showed longer RTs [main group effect; F(1, 33) = 8.290, p = 0.007] and higher RT variability [main group effect; F(1, 33) = 10.467, p = 0.003].

#### Peak Velocity

An ordinal interaction effect between group and movement direction [F(1, 33) = 3.310, p = 0.026, see **Table 2**] indicated that the difference in peak velocity between composite and singular movements was larger in PD patients in comparison to healthy subjects. A main direction effect was found, which implied that composite movements were performed with higher PV [F(1, 33) = 3.498, p = 0.036]. No group effect was found.

# EMG Parameters

# Activity Index (AI)

An ordinal interaction effect between group and movement direction was found (see **Table 3**), which indicated that the difference between the number of muscles involved in composite vs. singular movement was smaller in PD patients than in healthy subjects [F(1, 27) = 10.397, p = 0.003]. A main movement direction effect was found: regardless of group, the (a priori defined) composite movements did indeed involve more muscles than singular movements, as indicated by AI [F(1, 27) = 59.257, p < 0.001]. Moreover, a main group effect was found, revealing that PD patients showed a higher overall AI than healthy subjects [F(1, 27) = 13.568, p = 0.001].

# Differentiation Quotient (DQ)

Visually, the EMG activity patterns between the two groups were clearly different (**Figure 3**). Healthy subjects showed more specialized EMG activity than PD patients, as reflected in each of the investigated muscles being more distinctly active in a specific direction. The muscle activity configurations representing healthy subjects were almost encapsulated in the PD patient's configurations. This observation was quantified by our measure for differentiation of muscle activity (DQ): a main group


*Statistic results for kinematic parameters. RT, reaction time (ms); PV, peak velocity (degrees/second); Var, variability; C, composite; S, singular. Means and standard deviations are shown. \*Median (interquartile range).*

#### TABLE 3 | EMG parameters.


*Statistic results for all EMG measures. AI, activation index; DQ, differentiation quotient. Means and standard deviations are shown. \*Median (interquartile range). AI, activation index; DQ, differentiation quotient; m. ECU, (musculus) extensor carpi ulnaris; m. ECRL, extensor carpi radialis longus muscle; m. ECRB, extensor carpi radialis brevis muscle; m. FCR, flexor carpi radialis muscle.*

effect showed that patients had lower DQ scores than healthy subjects [F(1, 25) = 5.394, p = 0.028]. Additionally, we found a main effect of muscle [F(3, 25) = 17.048, p < 0.001] implying that some muscles showed a more specialized activity pattern than others. Post-hoc analysis revealed that m. ECU had the highest DQ compared to the other muscles, while m. FCR had the lowest. For m. ECRL and m. ECRB, DQ scores were similar (details in **Table 3**). No interaction effect was found.

# fMRI Results

## Within-group Comparisons: Healthy Subjects

To gain optimal insight in changes in brain activation patterns in PD patients, as compared to healthy subjects, activations related to the composite and singular movement conditions were first identified in healthy subjects (**Figure 4** and **Table 4**). We found that movement requiring more synergistic modulation (Composite > Singular) evoked a significant cluster of left striatal activation. Additionally, Composite > Singular revealed increased cortical activations comprising the SMA (BA6) and dorsolateral PMC (BA6) of both hemispheres, while contralateral to the side of movement we found increased activation in the medial prefrontal cortex (BA9), primary motor cortex (M1, BA4), and primary visual cortex (V1, BA17/18). Furthermore, the ipsilateral superior parietal cortex (BA7) showed more activation during composite movements. Healthy subject movement requiring less muscle tuning (Singular > Composite) was related to increased activations in the left (contralateral) ventral lateral thalamus and ipsilateral anterior (lobule IV/V) and posterior (crus 1) cerebellum. In addition, the right (ipsilateral) hippocampus showed increased activation related to singular movements.

FIGURE 3 | EMG results: mean scaled EMG activity for all directions for each muscle. Mean scaled activity is shown per group. PD patients are depicted as black, dashed lines, healthy subjects as gray lines. ECRL, m. extensor carpi radialis longus; ECRB, m. extensor carpi radialis brevis; FCR, m. flexor carpi radialis; ECU, m. extensor carpi ulnaris.

FIGURE 4 | fMRI results: activations in healthy subjects. Red activations: composite > singular, blue activations: singular > composite. Basal ganglia: 1, left striatal cluster; 2, thalamic cluster. Cerebellum: 1, anterior (lobule IV/V) cerebellum; 2, posterior (crus 1) cerebellum. Cortex: 1, medial prefrontal (BA9); 2, supplementary motor area (BA6); 3, (dorsal) premotor cortex; 4, primary motor cortex (BA4); 5, primary sensory cortex (BA2); 6, superior parietal cortex (BA7). For visualization purposes, all activations are shown above a threshold of *Z* = 2.4 [corresponding to uncorrected voxel level *p* < 0.01, without a restriction of cluster size (k)]. The z-coordinates indicate the position of the shown transversal planes relative to the AC-PC plane. Activations were rendered on the standard anatomical (ch2) template of MRICron (Rorden et al., 2007). L, left hemisphere.

# Between-group Comparisons: PD Patients vs. Healthy Subjects

Comparing the patterns of brain activations between groups using exclusive masking, revealed that for composite movements (Composite > Singular) patients showed decreased activations in the left ventral striatum (**Figure 5** and **Table 4**). In patients compared to healthy subjects, decreased cortical activation was also found in the SMA and bilateral (pre-) motor areas, the contralateral medial prefrontal and ipsilateral superior parietal cortex, while increased activation was seen mid temporally. For the comparison Singular > Composite, patients had decreased activation in the contralateral ventro-lateral thalamus. They showed increased activations in a cluster of the right pulvinar, extending to the bilateral anterior thalamus and dorsal caudate. The comparison Singular > Composite showed increased activations in patients distributed over the ipsilateral dorsal PMC, superior parietal and contralateral posterior cerebellum (lobule VI).

# Discussion

The employed step-tracking task, requiring subjects to make similar movement excursions into various directions, enabled a balanced dissociation between composite and singular movements. By investigating this dissociation in directional movements at behavioral and cerebral level, both in healthy subjects and PD patients, we were able to demonstrate a relation between composite movements and putative cortico-striatum circuitry, whereas cortico-cerebellar circuitry was stronger implicated in singular movements.

The fMRI results showed that decreased striatal activation was related to impairment of composite movement in PD, while singular movement in patients was related with increased right parietal and left cerebellar activation when compared to healthy subjects. The association between these brain, behavioral and muscle activity findings suggests that PD-related changes in cortico-striato-cortical function result in an impaired ability to select synergistic patterns of motion that demand particularly highly tuned muscle activity.

We acknowledge that the fMRI results were only identified at lenient statistical thresholds. This limitation of the study might raise valid critique if the identified clusters would have been without logical functional anatomical coherence. The fact that the patterns of activation did represent such coherence, both between striatum and ipsilateral cortical effects and between cerebellum and contralateral cortex activations, made us confident that these results represented physiological effects.

# Healthy Subjects Characteristics Behavioral Level

The a priori dichotomous classification of movement directions was confirmed by the two patterns of muscle activities. While the different movement directions shared common features such as movement amplitude, timing and speed, we found that composite movements did indeed involve more muscles than singular movements, as indicated by the muscle activity index (AI). The distinction between direction-associated singular and composite movements suggests a distinction between efficient and less efficient movement directions, which may particularly be due to anatomical characteristics such as the possible movement excursions in the wrist and position of muscle insertions, while gravity may be an additional external factor. In the end, the brain accomplished to reach similar movement results, given the described similarities in movement amplitude and speed.

Regarding the effect of gravity, movements in the directions 45◦ , 90◦ , and 135◦ require higher agonist activity to overcome gravitational effects and lower antagonist activity to terminate the movement, vs. less agonist activity and more antagonist activity for movements in the opposite, more downwards directions 225◦ , 270◦ and 335◦ , when the movement is "assisted" by gravity. Such physical characteristics are invariant parameters the brain has to deal with when organizing purposeful movement. Apparently, finding an optimal way of coupling various muscles contributes to solving these constraints. Regardless the cause of the dichotomy between singular and composite movements, these differences in muscle co-activity tuning provided specific parameters to investigate the underlying cerebral organization.

# Cerebral Level

The present fMRI results indicate that composite movement in healthy subjects is characterized by **left striatal activity**, corroborating the important role of the BG in selection of appropriate movement (Mink, 1996, 2003; Grillner et al., 2005; Lehericy et al., 2006). Furthermore, the co-occurrence of SMA activation is consistent with its role in movement selection (Deiber et al., 1999; Neubert et al., 2010). On the other hand, while composite movement elicits activation of the BG as well as cortical sensorimotor and premotor areas, singular movement was characterized by activation of the (contralateral) ventrolateral thalamus and ipsilateral cerebellum. This combination of activations is in accordance with the well-described functional connection between the contralateral thalamus and ipsilateral cerebellum in monkeys (Asanuma et al., 1983; Sakai et al., 1996). Higher activation of particularly the anterior lobe of the cerebellum found in the present study may emphasize its "corrective" role in movement optimization (Glickstein, 1992; Wolpert et al., 1998; Spencer et al., 2005). Such "corrective" aspects may become particularly urgent when movements are controlled by only a few opposite muscles. The latter may easily result in oversized movement excursions. The effective result of this putative cerebellar contribution is supported by less variability in movement execution at the behavioral level. Thus, we found a dissociation between BG involvement in highly tuned muscle co-activity, requiring more extensive planning and preparation to obtain adequately tuned patterns of coactive muscles, and cerebellar activation during movements requiring less muscle co-activity but more direct agonistantagonist corrections. This may reflect the complementary roles of these areas in motor control in which the BG are involved in movement selection, whereas the cerebellum has a role in movement optimization (Stein and Glickstein, 1992; Jueptner and Weiller, 1998; van Donkelaar et al., 2000; Bostan et al., 2010).

FIGURE 5 | fMRI results: differences between healthy subjects and Parkinson patients. Increased activations (SPM T-maps) for the two comparisons between composite and singular step-track movements. Green activations: PD < HC, purple activations: PD > HC. Basal ganglia: 1, left striatal cluster; 2, thalamic/pulvinar cluster. Cerebellum: 1, anterior cerebellum; 2, posterior cerebellum; Cortex: 1, medial prefrontal (BA9); 2, supplementary motor area (BA6); 3, (dorsal) premotor cortex; 4, primary motor cortex (BA4); 5, primary sensory

# PD Patients in Comparison to Healthy Subjects Behavioral Level

As hypothesized, patients showed less specialized muscle activity patterns than healthy subjects. Although directions requiring maximal muscle activity were the same in patients and healthy subjects (**Figure 3**), patients showed more muscle co-activity in the remaining directions, resulting in a less differentiated pattern and lower DQ. Similarly, patients employed more muscles for movements than healthy subjects, as indicated by a higher AI, particularly in singular, but also in composite movements. These findings imply decreased capacity to select appropriate muscle synergies. In addition, patients showed higher RTs and higher RT variability regardless of direction, which is in accordance with other studies investigating movement performance in PD (Majsak et al., 2008; Dounskaia et al., 2009). Moreover, the kinematic parameters RT variability and mean PV indicated a decline in motor performance in PD patients particularly for composite movements. Thus, PD-related changes in motor performance were most evident for composite movement and indicate that PD patients are especially impaired on tasks requiring highly tuned muscle co-activity.

# Cerebral Level

At cerebral level, the PD-related decreases in activation within the contralateral striatum and interconnected circuitry during movements with highly tuned coordination of co-active muscles

are in accordance with the classic PD model, although we had expected to find more extensive decreases in activation in the BG. The PD model describes a striatal dysfunction that induces enhanced inhibitory BG outflow to the thalamus and subsequently to the cortex (Albin et al., 1989; DeLong, 1990; Boecker et al., 2008; Obeso et al., 2008). The association between our fMRI results and decline in motor performance further underlines the role of the BG in movement selection. As compared to healthy subjects, the cortical increases in activation in PD during singular movement were particularly evident in the ipsilateral cortex and included the PMC, sensorimotor and parietal cortex. This ipsilateral distribution suggests the involvement of higher-order aspects of motor control. One may, in this respect, consider a stronger reliance on visual information in PD, through the parietal-premotor network (Praamstra et al., 1998; de Jong et al., 1999). Moreover, the ipsilateral (right) parietal cortex was found to be prominently active. This is in accordance with its involvement in visual processing and control of spatial attention (Gottlieb and Snyder, 2010) which is considered to be right hemisphere dominant (Malhotra et al., 2009; Thakral and Slotnick, 2009). By controlling shifts of spatial attention, as required during a task with shifting visual cues such as step-tracking, the parietal cortex plays a role in action selection (Cisek, 2007). By modulating selection via the PMC, the parietal cortex influences motor processing; an effect that seems to be stronger in patients as compared to

#### TABLE 4 | fMRI results.


*Brain activations for comparisons between singular and composite movement and between healthy subjects (HS) and Parkinson's disease patients (PD). "*+*," sign indicates activity in the masked condition for healthy subjects; "*↑↓*," signs indicate increased or decreased activations in PD patients compared to HS. Co-ordinates refer to the voxels of maximum activation within significant clusters (voxel level, uncorrected). Positive x, y, z coordinates (in mm) indicate locations right, anterior and superior of the middle of the anterior commissure, respectively.*

healthy subjects during singular movement. Additionally, we found that patients had increased activation in the superior posterior lobe of the left cerebellum. A contribution of this cerebellar region to visuospatial processing has been previously described (Stoodley and Schmahmann, 2009) and is consistent with distinct impairments on spatial tasks after damage of the left cerebellum (Gottwald et al., 2004; Hokkanen et al., 2006). Furthermore, a functional interaction between the posterior cerebellum and the opposite parietal cortex is effectuated by (crossed) connections (Sasaki et al., 1975). This interaction was further supported by a study on perception of hand movement that found a functional relation between the left posterolateral cerebellum and the right parietal cortex (Hagura et al., 2009).

These findings suggest a compensational mechanism involving the parietal cortex and the cerebellum. Compensational activation in PD patients involving the cerebellum is supported by the findings in the fMRI study of Yu et al. (2007), who examined differences in activation patterns during a simple, paced thumb pressing task and found significantly higher activations in the cerebellum in PD patients. These findings lead to the hypothesis of a compensational mechanism involving the cerebellum. In contrast to their study, our study was designed to differentiate between different movement tasks (requiring highvs. low-tuned muscle activity, respectively). This allows us to extend the hypothesis of compensational cerebellar activation in PD patients to a hypothesis that this may indeed be task-specific. Therefore, we propose that the increased activation of the left posterior cerebellar lobe and right parietal cortex in PD patients is due to increased reliance on visuospatial processing, possibly as a compensational strategy in the context of impaired BG selection.

# Conclusion

In the present study, we demonstrated a dissociation between high- and low-tuned muscle activity patterns for various directions of center-out step track movements of the right hand. The latter could thus be characterized as singular and composite movements, which were each related with a specific patterns of brain activation. These two movementrelated activation patterns showed differential changes in PD patients when compared to healthy subjects. In healthy subjects, we found a striking dissociation between involvement of the striatum and cortical areas in composite movement, vs. cerebellar involvement in singular movement; findings that may reflect the complementary roles of these areas in motor control. In patients we found decreased activation of the striatum and interconnected cortical areas for composite movement together with a decline in motor performance. These changes at both cerebral and behavioral level indicate that, as a result of changed cortico–striato–cortical functionality, PD patients are particularly impaired on tasks requiring highly tuned muscle coactivity. In singular movement, PD patients performed better and showed a combination of increased activation in the ipsilateral parietal cortex and left cerebellum. We interpret this as increased visuospatial processing, possibly deployed as a compensational mechanism.

# Author Contributions

CT, BD, NM, BC, and KL conception and design of research; CT performed experiments; AV, CT, and PY analyzed the data; CT, AV, BD, and NM interpreted results; GV and BC contributed analysis tools; AV, CT, and PY drafted manuscript; AV, CT, BD, BC, NM, and KL edited and revised manuscript, CT, AV, BD, and NM approved final version of manuscript.

# References


# Funding

This study was sponsored with a grant from the International Parkinson Foundation (IPF), grant title: "Initiation and Inhibition of Movement in patients with Parkinson's disease: New Insights Using a Direct Coupling between Movement, Muscle and Brain Activity."


area to the cerebellar cortex. Exp. Brain Res. 23, 91–102. doi: 10.1007/BF002 38732


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 van der Stouwe, Toxopeus, de Jong, Yavuz, Valsan, Conway, Leenders and Maurits. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Bimanual non-congruent actions in motor neglect syndrome: a combined behavioral/fMRI study

F. Garbarini <sup>1</sup> \* † , L. Turella2,3† , M. Rabuffetti <sup>4</sup> , A. Cantagallo<sup>5</sup> , A. Piedimonte<sup>1</sup> , E. Fainardi <sup>6</sup> , A. Berti <sup>1</sup> and L. Fadiga2,7

<sup>1</sup> SAMBA (SpAtial, Motor and Bodily Awareness) Research Group, Department of Psychology, University of Turin, Turin, Italy, <sup>2</sup> Istituto Italiano di Tecnologia (IIT), Genova, Italy, <sup>3</sup> Center for Mind/Brain (CIMeC), University of Trento, Trento, Italy, <sup>4</sup> Biomedical Technology Department, IRCCS Don Carlo Gnocchi Foundation, Milano, Italy, <sup>5</sup> BrainCare, Padova, Italy, <sup>6</sup> Department of Neuroradiology Unit, Neuroscience and Rehabilitation, Azienda Ospedaliera Universitaria, Ferrara, Italy,

In Motor Neglect (MN) syndrome, a specific impairment in non-congruent bimanual

<sup>7</sup> Section of Human Physiology, University of Ferrara, Ferrara, Italy

#### Edited by:

Gavin Buckingham, Heriot-Watt University, UK

#### Reviewed by:

Jolien Gooijers, KU Leuven, Belgium Raffaella Migliaccio, Institut National de la Santé et de la Recherche Médicale, France

#### \*Correspondence:

F. Garbarini, SAMBA (SpAtial, Motor and Bodily Awareness) Research Group, Department of Psychology, University of Turin, Via Po 14, 10123 Turin, Italy fra.garbarini@gmail.com; francesca.garbarini@unito.it

†These authors have contributed equally to this work.

Received: 30 June 2015 Accepted: 15 September 2015 Published: 06 October 2015

#### Citation:

Garbarini F, Turella L, Rabuffetti M, Cantagallo A, Piedimonte A, Fainardi E, Berti A and Fadiga L (2015) Bimanual non-congruent actions in motor neglect syndrome: a combined behavioral/fMRI study. Front. Hum. Neurosci. 9:541. doi: 10.3389/fnhum.2015.00541 movements has been described. In the present case-control study, we investigated the neuro-functional correlates of this behavioral deficit. Two right-brain-damaged (RBD) patients, one with (MN+) and one without (MN−) MN, were evaluated by means of functional Magnetic Resonance Imaging (fMRI) in a bimanual Circles-Lines (CL) paradigm. Patients were requested to perform right-hand movements (lines-drawing) and, simultaneously, congruent (lines-drawing) or non-congruent (circles-drawing) left-hand movements. In the behavioral task, MN− patient showed a bimanual-coupling-effect, while MN+ patient did not. The fMRI study showed that in MN−, a fronto-parietal network, mainly involving the pre-supplementary motor area (pre-SMA) and the posterior parietal cortex (PPC), was significantly more active in noncongruent than in congruent conditions, as previously shown in healthy subjects. On the contrary, MN+ patient showed an opposite pattern of activation both in pre-SMA and in PPC. Within this fronto-parietal network, the pre-SMA is supposed to exert an inhibitory influence on the default coupling of homologous muscles, thus allowing the execution of non-congruent movements. In MN syndrome, the described abnormal pre-SMA activity supports the hypothesis that a failure to inhibit ipsilesional motor programs might determine a specific impairment of non-congruent movements.

Keywords: motor neglect, fMRI, bimanual actions, bimanual coupling effect, supplementary and presupplementary motor area (pre-SMA; SMA)

# Introduction

Motor Neglect (MN) is a neuropsychological syndrome, which occurs as a result of stroke and is characterized by the underutilization of the contralesional limbs, in presence of normal strength, reflexes and sensibility and thus preserved potential for actual movement on the affected side. MN has been described as a ''pseudo-hemiplegia'' and is often interpreted as the consequence of damage to intentional motor circuits (Laplane and Degos, 1983; Gold et al., 1994; Coulthard et al., 2008; Garbarini et al., 2012, 2013c; Migliaccio et al., 2014). MN, especially in its pure form (without motor deficits), is a rare disorder—Laplane and Degos (1983) collected 20 patients over more than 10 years—and its frequency depends on the phase of the illness. According to some studies, signs of MN occurred in 12–33% of acute stroke patients (e.g., Buxbaum et al., 2004; Siekierka-Kleiser et al., 2006), but the frequency decreased to 8% in chronic patients (e.g., Buxbaum et al., 2004). In one study (Classen et al., 1997), 10 out of 16 patients with MN improved during the first 2 weeks (for a review, see Saevarsson, 2013; see also Migliaccio et al., 2014). Crucial to the present study, when MN patients are asked to perform bimanual movements, they only perform ipsilesional hand movements, even though they are actually capable of moving the contralesional hand. Recently, a behavioral dissociation has been found in MN patients, showing that the underutilization of the affected hand is greater when non-congruent (e.g., to bend one arm while extending the other; to open a bottle. . .) with respect to congruent (e.g., to clap the hand; to lift up a tray with both hands. . .) bimanual movements are required (Garbarini et al., 2013c). In the present case-control study, we investigated the neuro-functional correlates of this behavioral dissociation.

We tested two right-brain-damaged (RBD) patients with preserved upper limbs functionality, one with a pure form of MN (MN+) and the other one without MN (MN−), by using a functional Magnetic Resonance Imaging (fMRI) bimanual paradigm (see Section ''Materials and Methods''). We took advantage from a Circles-Lines (CL) task (Franz et al., 1991) in which, when people simultaneously draw lines with one hand and circles with the other hand, both trajectories tend to assume an oval shape, showing that hands motor programs interfere (bimanual coupling effect). It has been proposed that such motor constraints are tightly linked to motor intention and planning, rather than to movement execution. In healthy subjects, it has been demonstrated that the interference effect can be modulated by manipulating not the afferent sources of information, but the efferent level of movement planning and organization (Swinnen et al., 2003; Ridderikhoff et al., 2005; Spencer et al., 2005; Dounskaia et al., 2010; de Boer et al., 2013; Garbarini et al., 2015). Accordingly, in pathological conditions, where motor execution is damaged but motor intention is spared, bimanual coupling effects can be observed even in the absence of actual movements of one hand. As suggested by Garbarini et al. (2012), spatial coupling effects are present in RBD patients affected by contralateral (left) hemiplegia and anosognosia for hemiplegia (for temporal coupling effects in anosognosic patients see Pia et al., 2013; see also Garbarini and Pia, 2013). These patients claimed to move both hands when asked to draw lines with their right (intact) hand and circles with their left (paralyzed) hand. Although no movement of the left hand occurred, lines drawn with the right hand showed significant ''ovalizations''. Using the same CL paradigm, similar results were also found in amputees with illusory movements of the phantom limb (Franz and Ramachandran, 1998). Using a modified version of the CL task, coupling effects were also found in hemiplegic patients affected by a monothematic delusion of body-ownership, who identified the examiner's hand drawing circles as belonging to themselves (Garbarini et al., 2013b). In all these pathological conditions, where motor execution is damaged but motor intention is spared, actual movement execution seems unnecessary for bimanual coupling to occur: motor intention and programming are sufficient to trigger the interference effects. On the contrary, when motor execution is spared but motor intention is damaged, as in patients affected by MN, no bimanual constraints were found (Garbarini et al., 2012). The MN cases provide an interesting contrast to the AHP cases. The former are nonplegic but apparently lacking intention/planning, whereas the latter are plegic but still maintain intentions/plans for the affected hand.

According to these behavioral data in brain-damaged patients, previous neuroimaging data in healthy subjects, performing the CL task within the magnetic resonance (MR) scanner, showed the activity of brain circuits related to the intentional and predictive operation generating bimanual coupling (Garbarini et al., 2013a). These results support the role of a prefrontal-parietal network, mainly involving the pre-supplementary motor area (pre-SMA) and the posterior parietal cortex (PPC), that was significantly more active in noncongruent (CL) than in Congruent (Lines-Lines, LL) bimanual conditions.

Based on the above mentioned studies, we expected that MN+ patient, with respect to MN− patient, should show: (a) a worse behavioral performance in non-congruent (CL) than in congruent (LL) conditions; (b) a reduced activity when performing non-congruent conditions in pre-SMA and PPC. Overall, the expected results can reveal the neuro-functional correlates of the behavioral dissociation between congruent and non-congruent movements. More in general, they can represent the first neuro-functional investigation of the MN, shedding light on key areas of the neural network involved in this syndrome.

# Materials and Methods

# Participants

We recruited two RBD patients: one MN+ (male; 68 years old) and one MN− (male; 70 years old). The lesion extension of these patients was mapped and measured on the anatomical T1 by using MricroN software<sup>1</sup> (see **Figure 1**).

For the present study, we only selected stable patients (in the chronic phase of the illness) able to successfully perform functional task within the MR scanner. Exclusion criteria were: (1) previous neurological or psychiatric history; (2) severe general cognitive impairment; and (3) upperlimb motor deficits. Patients were classified as having or not MN based on clinical considerations, according to the following criteria: (i) spontaneous underutilization of the contralesional upper limb and hand during daily activities; and (ii) contrast between spontaneous underutilization of the left arm and hand, vs. normal movement and strength when the examiner actively encouraged the patient to use the arm. Both MN+ and MN− patients were also assessed using the following tests: general cognitive test (Mini-Mental State Examination—MMSE, Measso et al., 1993; cut off ≥24/30); tests for extrapersonal neglect (Bells Test, Gauthier et al., 1989;

<sup>1</sup>http://www.mccauslandcenter.sc.edu/mricro/mricron/index.html

cut off omissions L–R < 3) and for personal neglect (Fluff Test, Cocchini et al., 2010; cut off omissions L ≤ 2); assessment of hemiplegia and hemianesthesia (Pia et al., 2014; scores 0–3, 0 = no deficit; 3 = severe deficit). No deficits were reported in both patients at the time of testing, 1 year after stroke. As reported in the case history, both MN+ and MN− patients showed personal and extrapersonal neglect in the sub-acute phase, within 3 months after stroke. Neurological/neuropsychological assessment is summarized in **Table 1**.

The patients' motor performance during both congruent (LL) and non-congruent (CL) bimanual movements within the MRI scanner was evaluated with a score ranging from 0 to 2 (Garbarini et al., 2013c). Each bimanual block (for a total of 12 blocks) was evaluated and the mean score was reported. At the end of the fMRI acquisition, we also asked the patients a selfevaluation, using the same score, of both congruent and noncongruent movements. The examiner's score and patients' score are reported in **Table 2**.

Both patients gave their informed consent and the protocol was approved by the Ethical Committee ''Comitato Etico Unico della Provincia di Ferrara'' (Italy).

## Experimental Procedure

Patients were required to perform the ''CL'' task (Garbarini et al., 2012, 2013c), within a MR scanner while data regarding their brain activity was collected. The CL task involved drawing on a dual panel fMRI compatible tablet (Tam et al., 2012; Garbarini



#### TABLE 2 | Clinical evaluation of the patients' motor performance during the task.


In both the examiner's evaluation and the patient's self-evaluation the scores were ranked from 0 to 2: 0 = left hand movements were not performed; 1 = left hand movements were performed but not at the same time of the right hand movements; 2 = left and right hand movements were simultaneously performed. Note that, MN+ patient showed a specific impairment in simultaneously executing non-congruent (CL) movements (i.e., during LL conditions, left and right hand movements were simultaneously performed, score = 2; during CL conditions, left hand movements were performed asynchronously with respect to right hand ones, score = 1). However, a discrepancy emerged between this examiner's evaluation and the patient's self-evaluation, where the patient gave the higher score (2) also for the impaired performance at the CL task.

et al., 2013a), using one or both hands, in response to visually administered commitments, A head coil-mounted display system (IFIS-SA, Invivo Corporation, Gainesville, FL) was used to present visual stimuli via E-Prime software (Psychology Software Tools, Inc., Pittsburgh, PA), which also ensured synchronization with the MR scanner and the behavioral data collection. In addition, two of the authors (FG and AP) verified the correct execution of the tasks in the control room.

#### Experimental Task and Paradigm

The ''CL'' task (Garbarini et al., 2012, 2013a), adopted in the present study, consisted in the execution of different unimanual and bimanual motor tasks. The adopted experimental conditions required the patients to perform the following movements:


This set of behavioral tasks was designed to explore modulations in motor performance of the right (dominant) hand. The behavioral analysis thus enabled quantitative analysis of the interference effect of the controlesional left hand circles drawing, on the lines executed with the ipsilesional right hand.

The timeline of the study comprised an initial rest of 30 s followed by an alternation of experimental blocks of 15 s duration followed by rest blocks of the same duration. A pseudo-random sequence of experimental blocks was presented to the patients, comprising a total of 24 experimental blocks (6 repetitions for each of the 4 experimental conditions). A final 30 s rest block was presented after all the experimental conditions. During the experimental blocks, the patients had to perform hand movements according to the information (either lines or circles) shown on the head-mounted display (see **Figure 2** for a graphical representation of the paradigm).

# Behavioral Data Collection and Analysis

Behavioral data were collected using a dual panel fMRIcompatible tablet, a modified version of the one used by Tam et al. (2012). This version incorporated two separate panels and two styli allowing the simultaneous collection of data from the two hands (see **Figure 3**). Behavioral motor performance was recorded from each panel separately by a distinct computer positioned outside the scanner room. Before starting the fMRI study, the patients extensively practiced the task in order to be able to accomplish it smoothly within the scanner.

An Ovalization Index (OI) was defined to quantify the occurrence of lateral deviation when continuously drawing a straight vertical line. The strength of any bimanual coupling/interference effect was signaled by an increased OI value in the Non-congruent condition compared to the Congruent condition.

OI value was defined as the standard deviation of the righthand trajectories in relation to an absolute vertical line (a detailed description of the algorithm involved in calculating the OI in Garbarini et al., 2012). Briefly, OI index ranges between a value of zero for straight trajectories without any sign of ovalization and a value of 100 for circular trajectories. As a consequence, the value of the OI allows quantifying the bimanual coupling effect for each performed movement by comparing the bimanual movement of each hand with its unimanual equivalent. The amount of interference of the left hand in executing circles on the right hand executing lines is shown as an increase of the OI (bimanual coupling/interference effect). Furthermore, the average drawing frequency was computed for each block as the number of drawing cycles per second, or, alternatively, the inverse of the average cycle duration (in Hz).

# Functional Data Acquisition and Analysis

MR images were acquired on a 1.5T MR scanner (Phillips Achieva). Functional images were collected, while patients were performing the ''CL'' task, with an EPI T2<sup>∗</sup> -weighted sequence throughout the whole brain (TR = 2500 ms, TE = 50 ms, field of view 230 × 230 mm, in-plane resolution 3.59 × 3.59 mm, slice thickness = 4 mm, 30 slices). A total of 312 images were collected during one functional run. A high quality T1-weighted image (1 mm isotropic voxels) was also acquired to define the lesion extent as shown in **Figure 1**.

Analysis of fMRI data was carried out by using Statistical Parametric Mapping software<sup>2</sup> . Functional data were realigned using a two-step procedure implemented in SPM5. Data were registered to the first functional volume of the series and then to the mean image. Normalization of the T1-weighted image was performed on the MNI template provided within SPM by using the unified segmentation approach (Ashburner and Friston, 2005) and by applying a masking procedure excluding the part of the brain affected by the lesion. This type of analysis has been demonstrated to strongly improve the normalization procedure in patients with brain lesion (Crinion et al., 2007; Andersen et al., 2010). The resulting normalization parameters were applied to the T1 and to the functional images (resampling the voxels at 2 × 2 × 2 mm). Functional data were spatially smoothed using 8 mm FWHM Gaussian kernel.

2 SPM5, www.fil.ion.ucl.ac.uk/spm

24 experimental blocks (6 repetitions for each of the 4 experimental conditions) was presented to each patient. The study started and ended with a longer rest period (30 s). During the experimental condition blocks, patients had to perform hand movements following the visual cues appearing on the screen within two white hands. Whereas during the rest blocks, patients had to attend the picture depicting two white hands. The possible combination of the observed stimuli matched the experimental conditions: unimanual Lines with the right hand (L); unimanual Circles with the left hand (C); bimanual congruent Lines-Lines (LL), simultaneously with both hands (LL); bimanual non-congruent Circles-Lines (CL), simultaneously with the right hand drawing Lines and the left hand drawing Circles (CL).

A high-pass temporal filter (cut-off 128 s) was also applied to the time series. Whole-brain analysis was performed by applying the General Linear Model (GLM) for analysis of fMRI time series. Regressors were defined based on the timing of presentation for each of the conditions and were modeled using a box-car function convolved with the hemodynamic response function (HRF) with duration equal to the experimental block. Predictors of no interest were modeled to account for residual effects of the movements measured during the realignment procedure.

Contrasts of interest were obtained by entering the corresponding contrast vector in the design matrix. The threshold for the presented data was set at a p < 0.001 uncorrected for multiple comparisons and reporting only clusters comprising at least 10 voxels. In order to test the role of pre-SMA and PPC in bimanual coupling, we performed a ROI analysis on pre-SMA, left and right PPC. The coordinates for these regions were obtained from a recent study on healthy participants performing the same task (Garbarini et al., 2013a) and transformed from TAL to MNI space adopting the tal2mni function<sup>3</sup> . Beta values were extracted from spherical ROIs (radius 9 mm) centered on the coordinates in MNI space.

# Single-Subject Analyses

In order to analyze behavioral (OI values and drawing frequency) and neuroimaging (beta extracted from the ROI) data, recording from MN+ and MN− patient during the fMRI sessions, we used two different approaches in single-subject analysis: (a) Crawford's test (Crawford and Garthwaite, 2005) designed to test whether the discrepancy between two tasks (LL; CL) observed for each patient (MN+; MN−) is significantly different from the discrepancies in a control sample; and (b) Crawford's test (Crawford et al., 2010) designed to test the difference between two single cases (MN+ vs. MN−) by referring to a control sample. For both these methods, which need to refer to normative data from healthy population, we used behavioral and neuroimaging data from healthy participants (n = 12) tested in Garbarini et al. (2013a).

# Results

#### Behavioral Results

In order to quantify the interference (coupling) effects between the two hands motor programming, we analyzed the OI of the different experimental conditions. The bimanual coupling effect should cause, for the right hand always performing lines, the OI value to increase in the non-congruent CL condition (where the left hand performs circles) with respect to the congruent LL condition (where the left hand performs lines). As shown in **Figure 4**, the MN− patient's right hand trajectories in CL condition revealed a clear ovalization, while the MN+ patient's trajectories did not.

Crawford's tests revealed that in MN+ patient the discrepancy between the OI values of the two tasks (LL and CL) was significantly smaller than the discrepancies in the healthy subjects (OI value in normative sample [mean ± sd]: LL = 5.5 ± 1.5; CL = 13.4 ± 8.4; corr. between LL and CL = 0.8; in MN+ patient [mean]: LL = 10.4; CL = 11.5; T = 4.5; p = 0.001, two tailed). No difference between MN− patient and healthy subjects was found (OI value in MN− patient [mean]: LL = 8.0; CL = 32.5; T = 0.6; p = 0.52, two tailed). Crucial to the present study, directly comparing MN+ and MN− patient, Crawford's test showed significant differences when considering the OI increase in CL condition with respect to LL condition (difference CL minus LL in MN+ patient [mean]: 1.1; in MN− patient [mean]: 24.4; in normative sample [sd]: 6.8; Z(PCC): −2.4; p = 0.03, two tailed). This means that, in CL condition, an OI increase, comparable to that found in healthy subjects, was present only in MN−patient and not in MN+ patient. See **Figure 5**. With respect to the drawing frequency, Crawford's test did not show significant difference between MN+ and MN− patients, suggesting that both of them were comparable to the normative sample (Hertz in MN+ patient [mean]: 0.9; in MN− patient [mean]: 1.9; in normative sample [mean ± sd]: 1.3 ± 0.4; Z(PCC): 1.46; p = 0.17, two tailed).

MN− but not for MN+.

<sup>3</sup>http://imaging.mrc-cbu.cam.ac.uk/imaging/MniTalairach

FIGURE 5 | Behavioral results. Results of behavioral analysis, with the ovalization index (OI) value for the right hand as dependent variable and CL and LL conditions as independent variables, are reported within the histograms. The 0 value represents straight trajectories; 100 represents perfect circular trajectories; intermediate values represent ovalized trajectories, with the vertical axis longer than the horizontal one. Note, in MN−, the increased OI value in CL respect to LL conditions; in MN+, no modulation of the OI value in the contrast between CL and LL condition. The statistical comparison between the case MN+ and the control MN− is shown; <sup>∗</sup>p < 0.05. Error bars represent standard error of the mean (SEM).

# fMRI Results

When contrasting bimanual (LL; CL) against unimanual (L; C) conditions, MN− patient recruited a fronto-parietal network (see **Figure 6** and **Table 3**). In details, in the dominant hemisphere, activation was present within a widespread cluster with peak activity within the left superior parietal lobule, comprising also the left postcentral gyrus and the precuneus. The second cluster was located within the inferior temporal gyrus within the right non-dominant hemisphere and its activation was spreading within the hippocampus. Within the same hemisphere there was also a cluster within the inferior frontal gyrus (pars orbitalis) extending medially within the putamen and the medial prefrontal cortex. Within subcortical structure, there was a bilateral recruitment of the thalamus. Furthermore, activation was also present within other smaller clusters: one within the left precentral gyrus (dorsal premotor cortex) and bilaterally within the inferior frontal gyrus. In details, activation was present in a cluster within the right inferior frontal gyrus, in its most anterior subdivision (pars orbitalis); there was a bilateral recruitment of the pars opercularis of the inferior frontal gyrus, with one cluster located within the left hemisphere and another in the right hemisphere. Within the right hemisphere there was also a cluster in the parietal operculum. Finally, there were two clusters within the temporal cortex: one within the left temporal pole and one within the right superior temporal gyrus.

By contrast, MN+ patient showed a limited activation pattern comprising mainly one cluster within the left angular gyrus and two smaller clusters one always within the inferior parietal lobule and the other within the middle temporal gyrus (see **Figure 6** and **Table 3**).

# ROI Analysis: MN− vs. Healthy Participants

Crawford's test revealed significant difference between MN− patient and healthy subjects in none of the considered ROI (pre-SMA beta value in MN− patient [mean]: LL = 1.64; CL = 1.73; T = 2; p = 0.08, two tailed; left PPC: LL = 1.07; CL = 1.65; T = 0.93; p = 0.37, two tailed; right PPC: LL = 0.43; CL = 0.97; T = 0.33; p = 0.74, two tailed).

# ROI Analysis: MN+ vs. Healthy Participants

In ROI analysis, Crawford's tests revealed that in MN+ patient the discrepancy between the beta values for the two tasks (LL and CL) was significantly different with respect to the same discrepancies in healthy subjects for the pre-SMA (beta value in normative sample [mean ± sd]: LL = 0.27 ± 0.28;

> L + C). The activation maps are mapped on the lateral and medial views of a MNI template brain at p < 0.001 uncorrected. The color bar refers to t-values. The position of the tested ROIs are highlighted with circles. MNI coordinates: pre-supplementary motor area (pre-SMA) (4, −2, 52), left posterior parietal cortex (PPC) (−37, −50, 33), right PPC (26, −44, 36).

#### TABLE 3 | Activation peaks for the contrast between Bimanual and Unimanual actions (CL + LL > L + C) in MN+ and MN− patients.


<sup>a</sup>For brevity, only clusters with at least 10 contiguous voxels are reported. In red, clusters surviving cluster correction (p < 0.05) are reported. <sup>b</sup>Stereotaxic coordinates in MNI space are reported in mm.

CL = 0.74 ± 0.53; corr. between LL and CL = 0.87; in MN+ patient [mean]: LL = 0.71; CL = −0.12; T = 3; p = 0.015, two tailed) and for the left PPC (beta value in normative sample [mean ± sd]: LL = 0.36 ± 0.4; CL = 0.91 ± 0.66; corr. between LL and CL = 0.82; in MN+ patient [mean]: LL = 1.59; CL = 0.91; T = 4.14; p = 0.002, two tailed); no significant difference for the right PPC was found (beta value in normative sample [mean ± sd]: LL = 0.59 ± 0.34; CL = 1.13 ± 0.54; corr. between LL and CL = 0.83; in MN+ patient [mean]: LL = 0.57; CL = 0.55; T = 1.4; p = 0.019, two tailed).

#### ROI Analysis: MN+ vs. MN−

Crucial to the present study, directly comparing MN+ and MN− patient, Crawford's test showed significant differences when considering the beta value increase in CL condition with respect to LL condition, for pre-SMA (difference CL minus LL in MN+ patient [mean]: −0.84; in MN− patient [mean]: 0.09; in normative sample [sd]: 0.32; Z(PCC): 2.35; p = 0.04, two tailed) and left PPC (difference CL minus LL in MN+ patient [mean]: −0.67; in MN− patient [mean]: 0.57; in normative sample [sd]: 0.39; Z(PCC): 2.24; p = 0.05, two tailed); no significant difference was found for right PPC (difference CL minus LL in MN+ patient [mean]: −0.01; in MN− patient [mean]: 0.53; in normative sample [sd]: 0.32; Z(PCC): 1.19; p = 0.26, two tailed). Results for ROI analyses are reported in **Figure 7**.

# Discussion

In the present case-control study, we investigated the neurofunctional correlates of a behavioral dissociation between congruent and non-congruent bimanual movements in MN syndrome.

The behavioral study showed that, while patients without MN show normal coupling effect in a CL task, MN+ patient did not show any coupling. It is worth noting that the same MN+ patient was tested in a previous behavioral study, employing a similar CL task. At the time of the first test, he was not able to draw left hand circles during the bimanual CL condition and only drew right hand lines. One year later, although in the everyday life the patient spontaneously underused the left hand, he was able to perform bimanual movements, when explicitly required. However, in MN+ patient, a specific impairment in non-congruent bimanual movements was still evident both in ecological action and in the experimental task. Indeed, in LL condition, the MN+ patient could move both hands at the same time, while during CL condition his hands moved asynchronously. Interestingly, according to previous findings on motor awareness in MN syndrome, the patient, when asked to evaluate his performance during the task, was not aware of this specific impairment in CL condition (see **Table 2**).

In the neuroimaging study, contrasting bimanual (LL; CL) with unimanual (L; C) conditions, MN− patient recruited a

LL conditions, for all brain regions; in MN+, the significant decreased beta value in CL respect to LL conditions, for pre-SMA and left PPC, and no modulation between CL and LL condition or right PPC. The statistical comparison between the case MN+ and the control MN− is shown; <sup>∗</sup>p < 0.05. Error bars represent SEM.

fronto-parietal network known to be involved in the execution of bimanual movements (e.g., Nair et al., 2003; Debaere et al., 2004; Wenderoth et al., 2005). Activation was stronger and more widespread within the dominant hemisphere encompassing fronto-parietal networks involved in the planning, execution and online control of hand actions (Filimon, 2010; Turella and Lingnau, 2014; Gallivan and Culham, 2015). On the contrary, MN+ patient showed an abnormal pattern of activity, involving mainly the left angular gyrus even at a rather liberal statistical threshold (p < 0.001uncorr). This suggests, in MN+ patient, a similar cortical recruitment in both bimanual and unimanual conditions, as if bimanual movements were only a simple sum of unimanual actions. By directly comparing CL and LL conditions, in MN− patient we found that, as previously described in healthy subjects (Garbarini et al., 2013a), a fronto-parietal network, mainly involving pre-SMA and PPC, was significantly more active in non-congruent (CL) than in congruent (LL) conditions. On the contrary, MN+ patient showed an opposite pattern of activation; i.e., in pre-SMA and in left PPC a lower activity in non-congruent (CL) with respect to congruent (LL) conditions.

These behavioral and neuroimaging results are in accordance with a previous demonstration that MN patients fail to inhibit ipsilesional limb motor plans (Coulthard et al., 2008). Using a masked prime task, the authors investigated, in MN patients, the presence of the negative compatibility effect: i.e., the paradoxical reaction time, occurring when the interval between mask and target is 100–200 ms, slower when the prime and target are congruent and faster when they are non-congruent. This study showed that MN patients fail to inhibit the right hand motor plans (evoked by the non-congruent prime), which then intrude abnormally on left hand action planning, slowing down initiation of movement with the left hand. If motor planning for the controlesional arm is intruded by motor plans for the ipsilesional arm, it is likely to expect that congruent bimanual movements will be facilitated and non-congruent bimanual movements will be impaired.

Converging neuroimaging data showed that, during congruent bimanual movements, the (left) non-dominant motor system ''entrusts'' a part of the control of the nondominant hand to the (right) dominant motor system via the uncrossed efferent pathway (Aramaki et al., 2006). This normal physiological mechanism, can explain the facilitation in LL condition shown by the MN+ patient, wherein the dominant (intact) motor system implemented the same motor program on both hands. On the contrary, the (right) non-dominant hemisphere has a key role during the execution of bimanual non-congruent movements (Sadato et al., 1997; Wenderoth et al., 2004; Garbarini et al., 2013a). Within this hemispheric balance, the (bilateral) pre-SMA activity is supposed to exert an inhibitory function on the default coupling of homologous muscles, promoted by neural crosstalk, thus allowing the execution of non-congruent bimanual movements (Sadato et al., 1997). The abnormal pre-SMA activity (as well as the related abnormal PPC activity, Wenderoth et al., 2004; Garbarini et al., 2013a) we found in MN+ patient, supports the hypothesis that a failure to inhibit ipsilesional (dominant) motor programs (Coulthard et al., 2008) determines the MN+ patient's specific impairment in non-congruent CL condition.

From an anatomical point of view, the MN+ patient's lesion pattern (see **Figure 1**) was compatible to that described in a recent study (Migliaccio et al., 2014), stressing the role of the cingulum bundle in the MN syndrome. The cingulum is a major pathway of the medial motor system, also connecting this system with limbic structures (e.g., Catani et al., 2013), which underlie motivational aspects of actions (Devinsky et al., 1995). According to Migliaccio et al. (2014), damage to the cingulum is likely to disrupt the integrated functioning of the medial motor system, with subsequent impaired SMA and pre-SMA activity, thus causing the spontaneous underutilization of the contralesional limb. We can speculate that, in the MN+ patient tested here, a partial restoring of this connection between the cingulum and the limbic system, can be the reason of the patient's behavioral improvement from the first behavioral evaluation (when the patient did not perform bimanual movements in both ecological context or stimulus-driven tasks; see Garbarini et al., 2012) to the present fMRI experiment (when the patient did not spontaneously perform bimanual movements in ecological context, but was able to perform them in stimulus-driven tasks, as the one employed here). Crucially, damage to the cingulum can also lead to an imbalance between left and right medial motor systems, resulting in the specific impaired motor inhibition during non-congruent bimanual movements. Together with the cingulum, it is likely that another fiber bundle can be involved in this lack of inhibition: the SFL I, located just dorsal to the cingulum and, as recently demonstrated in human (Thiebaut de Schotten et al., 2011), connecting the medial parietal and frontal regions, known to play a crucial role in non-congruent bimanual movements. The fiber connections between these areas involved in the task, as well as their possible damage in MN patients, would be a specific matter of interest for future studies.

We acknowledge, as a limitation of the present study, that, being based on only two patients, these results need replication in further studies involving more cases. However, the choice to perform a case-control study was due to the rarity of a pure form of the MN syndrome (without motor deficit), especially in stable patients able to successfully perform a functional task

# References


within the MR scanner. Thus, although limited by the samplesize, the present study represent the first neuro-functional investigation of the MN syndrome, showing that an abnormal pre-SMA and parietal activity can lead to a failure to inhibit ipsilesional motor programs, causing both the underutilization of the contralesional limb, characterizing the MN syndrome, and the specific impairment in non-congruent bimanual movements, shown in the present study.

# Funding

This work has been funded by MIUR-PRIN 2012 grant to LF, by Compagnia di San Paolo 2013 grant to AB, by ''Futuro in Ricerca'' 2013 grant (FIRB 2013, project RBFR132BKP) to LT and by MIUR-SIR 2014 grant (project RBSI146V1D) to FG.

# Acknowledgments

The authors would like to thank the patients involved in the study and the team of Department of Neuroscience and Rehabilitation of the Azienda Ospedaliera of Ferrara for their precious assistance and help provided throughout the entire process of data collection.


case of anosognosia for hemiplegia and motor neglect. J. Neurol. Neurosurg. Psychiatry 84, 416–419. doi: 10.1136/jnnp-2012-302838


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Garbarini, Turella, Rabuffetti, Cantagallo, Piedimonte, Fainardi, Berti and Fadiga. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Direct comparisons of hand and mouth kinematics during grasping, feeding and fork-feeding actions

D. J. Quinlan1,2,3\* and J. C. Culham1,3,4

<sup>1</sup> Brain and Mind Institute, University of Western Ontario, London, ON, Canada, <sup>2</sup> Department of Psychology, Huron University College, London, ON, Canada, <sup>3</sup> Graduate Program in Neuroscience, University of Western Ontario, London, ON, Canada, <sup>4</sup> Department of Psychology, University of Western Ontario, London, ON, Canada

While a plethora of studies have examined the kinematics of human reach-to-grasp actions, few have investigated feeding, another ethologically important real-world action. Two seminal studies concluded that the kinematics of the mouth during feeding are comparable to those of the hand during grasping (Castiello, 1997; Churchill et al., 1999); however, feeding was done with a fork or spoon, not with the hand itself. Here, we directly compared grasping and feeding kinematics under equivalent conditions. Participants were presented with differently sized cubes of cheese (10-, 20- or 30-mm on each side) and asked to use the hand to grasp them or to use a fork to spear them and then bring them to the mouth to bite. We measured the apertures of the hand during grasping and the teeth during feeding, as well as reaching kinematics of the arm in both tasks. As in many past studies, we found that the hand oversized considerably larger (∼11–27 mm) than the food item during grasping; moreover, the amount of oversizing scaled with food size. Surprisingly, regardless of whether the hand or fork was used to transport the food, the mouth oversized only slightly larger (∼4–11 mm) than the food item during biting and the oversizing did not increase with food size. Total movement times were longer when using the fork compared to the hand, particularly when using the fork to bring food to the mouth. While reach velocity always peaked approximately halfway through the movement, relative to the reach the mouth opened more slowly than the hand, perhaps because less time was required for the smaller oversizing. Taken together, our results show that while many aspects of kinematics share some similarity between grasping and feeding, oversizing may reflect strategies unique to the hand vs. mouth (such as the need to have the digits approach the target surface perpendicularly for grip stability during lifting) and differences in the neural substrates of grasping and feeding.

Keywords: feeding, grasping, hand, mouth, fork, tool use, grip, transport

Citation: Quinlan DJ and Culham JC (2015) Direct comparisons of hand and mouth kinematics during grasping, feeding and fork-feeding actions. Front. Hum. Neurosci. 9:580. doi: 10.3389/fnhum.2015.00580

Edited by: Gavin Buckingham, Heriot-Watt University, UK

Reviewed by:

\*Correspondence: D. J. Quinlan dquinla4@uwo.ca

Received: 30 July 2015 Accepted: 05 October 2015 Published: 21 October 2015

Singapore Dylan Francis Cooke,

Charmayne Mary Lee Hughes, Nanyang Technological University,

University of California, Davis, USA

**Abbreviations:** H2F, Hand-to-Food movement; H2M, Hand-to-Mouth movement; F2F, Fork-to-Food movement; F2M, Fork-to-Mouth movement.

# INTRODUCTION

Grasping and self-feeding actions are two of the most frequent everyday functions of the hand, particularly in humans and other primates (Graziano and Aflalo, 2007). In fact, such actions may be so fundamental in daily life that they shape the organization of the cerebral cortex (Graziano and Aflalo, 2007; Graziano, 2008). If regions of motor and premotor cortex are stimulated for a duration comparable to a natural action (e.g., a half-second), complex natural actions such as reach-to-grasp, self-feeding actions, or defensive actions can be evoked (Graziano et al., 2002). Moreover, different actions are evoked by stimulation to different foci and these foci are arranged topographically (Graziano et al., 2002). This topography has been observed across three primate species, suggesting it is common across the primate lineage (Kaas et al., 2013). Here, we compare the behavioral properties of two of these fundamental actions—grasping and feeding. Specifically, given that these two actions appear to have different neural substrates, we investigated whether their kinematic properties differ as well.

Although behavioral studies of feeding have been surprisingly few, a rich literature on the kinematics of reach-to-grasp actions has revealed the strategies employed in using the hand to acquire a target. The seminal studies of Jeannerod (1981, 1984, 1986) led to the proposal that reach-to-grasp actions are comprised of two distinct components: a transport component that uses visual information about object location to move the arm/hand to the target object and a grip component that uses visual information about intrinsic object properties such as shape and size to preshape the hand appropriately. Other evidence has suggested the transport and grip components may rely on different substreams of the dorsal visual pathway (Rizzolatti and Matelli, 2003; Cavina-Pratesi et al., 2010; Fattori et al., 2010; Vesia and Crawford, 2012; Turella and Lingnau, 2014). Hundreds of kinematic studies of reach-to-grasp movements have examined the factors that affect measures associated with transport and grip components (e.g., reach velocity and hand grip aperture, respectively; e.g., Jones and Lederman, 2006). However, feeding movements also involve arm transport (to the mouth) and aperture preshaping (by the mouth), but these components have been seldom investigated.

Two studies that have investigated self-feeding actions concluded that the transport and grip components of these reachto-bite actions are similar to those of reach-to-grasp actions. One study measured kinematics while participants fed themselves cubes of cheese with a fork (Castiello, 1997). According to their description, participants ''were required to reach for the cheese with the fork and bring it to the mouth'' (p. 553) and ''close[d] their mouths around the fork.'' (p. 555). Results showed that as the cheese cube approached, the mouth opened to a size considerably larger than the cheese. This pattern is very similar to how the hand aperture oversizes and then closes down as the hand approaches the target object during a reach-tograsp movement, as shown in previous data (Jeannerod, 1984). Similarities were also seen in the transport component, whereby the final approach took longer when the target object was small vs. large. Due to such similarities between reach-to-grasp and reach-to-bite kinematics, Castiello (1997) suggested that these actions might be directed by a common motor plan that is controlled by shared neural circuitry. In another self-feeding study (Churchill et al., 1999), participants fed themselves yoghurt using a spoon; similar results and conclusions were obtained as in Castiello (1997). Perhaps it is due to these proposed similarities between grasping and feeding actions that little subsequent research has been conducted on this topic.

While these studies of feeding actions were impressive initial forays into a new area of kinematic research, several aspects of the experiments may have artificially exaggerated the similarities between the kinematics of reach-to-bite and reach-to-grasp actions. First, the grip component was not comparable between the hand and mouth. That is, in grasping an object with a precision grip, the finger and thumb contact the sides of the object (**Figure 1Aiii**); whereas, in both of the feeding studies, the mouth was used not to grip the food (cheese cube or dollop of yoghurt) but to reach around the food and then pull it into the mouth. As such, in these feeding paradigms, the food served as an obstacle such that the mouth necessarily had to open larger than the food to avoid striking the teeth. Likewise, perhaps the presence of a ''food obstacle'' within the mouth grip aperture serves to inflate maximum aperture, much like the presence of obstacles outside the hand grip aperture cause peak aperture to become smaller (Jackson et al., 1995; Tresilian, 1998). Second, in Castiello's (1997) study, the mouth aperture was determined by markers placed on the upper and lower lips. Alternatively, the markers could have been placed in a manner that would estimate the aperture between the teeth (or jaw). Although the aperture between the two lips would be correlated with the aperture between the teeth, the two are not always in perfect agreement because the lips are more elastic and can be moved somewhat independently of the teeth. In contrast, Churchill et al. (1999) placed markers on the forehead and chin, which would more directly reflect the aperture between the teeth because of skull and jaw anatomy. Notably, they found subtle kinematic differences between hand and mouth aperture. Lastly, both experiments had participants use a tool during the feeding actions, but compared the kinematics of the mouth to those exhibited when grasping with the hand alone. It remains a matter of debate how bodily actions are modified by tool use (e.g., Iriki et al., 1996; Cardinali et al., 2009; Gallivan et al., 2013). In fact, the introduction of a tool into reach-to-grasp actions alters some kinematic measures, such as lengthening the outward reach deceleration phase (Gentilucci et al., 2004). Taken together, these methodological differences between feeding and grasping may have affected the data and thus the conclusions; as such, it is worth re-examining how the two actions compare under conditions that are as similar as possible.

There are reasons to expect that reach-to-grasp and reachto-bite actions may be performed differently when the two tasks are directly comparable. First, as mentioned there is evidence for different neural substrates (e.g., Buccino et al., 2001; Graziano et al., 2002). Second, reach-to-grasp and feeding actions may rely on different sensory information. In reach-to-grasp actions, the actor has clear vision of the object, the transport effector (arm), and aperture (hand) throughout the movement. In contrast, in

feeding actions, the actor has clear vision of the object and transport effector (arm) initially but it degrades as the hand approaches the mouth [due to gaze (de Bruin et al., 2008) and limitations of vergence and accommodation]. Moreover, in feeding actions, the actor has no vision of the aperture (mouth).

Although visual information is much more limited for feeding than grasping actions, information from the somatic senses is richer and perhaps more highly weighted. During feeding actions, somatosensation provides additional information about the intrinsic object properties relevant for preshaping the mouth grip. Specifically, haptics and hand posture provide information about the size of an object as well as material properties such as its density and texture. Although proprioceptive and kinesthetic information about the arm's location and trajectory is available for both grasping and feeding actions, some evidence suggests that inward arm movements may rely on proprioceptive information to a greater degree (de Bruin et al., 2008).

In sum, although two kinematic studies suggested strong similarities between grasping and feeding actions, other evidence suggests possible differences; as such we wanted to revisit the comparison of grasping and feeding kinematics under directly comparable conditions. In Experiment 1, participants reached out to grasp cheese cubes of three different sizes using a precision grip with the finger and thumb [i.e., Hand-to-Food (H2F) movement] and then brought the food to the mouth to perform a ''precision bite'' by gripping the cheese cube between the teeth [i.e., Hand-to-Mouth (H2M) movement]. We measured: (1) the grip component based on hand aperture during the H2F movement or mouth aperture during the H2M movement and (2) the transport component based the velocity of the arm during both H2F and H2M movements. In Experiment 2, we examined whether kinematics would be affected when participants used a fork, instead of their fingers, to acquire the food item [i.e., Fork-to-Food (F2F) movement] and bring the food item to the mouth [i.e., Fork-to-Mouth (F2M) movement]. We expected that our paradigm, with more directly comparable actions, may reveal differences between the kinematics of grasping vs. feeding movements.

# MATERIALS AND METHODS

# Participants

# Experiment 1

Ten right-handed participants (four males, six females; mean age = 31.1 years) with normal or corrected-to-normal vision participated in this experiment. Prior to testing, participants were required to undergo two prescreening tests: (1) handedness was assessed using a modified Edinburgh handedness inventory and (2) stereoscopic vision was tested with a 3-D Vectographs stereoacuity test (Stereo Optical Co., Inc., Chicago, IL, USA). Only those participants who were strongly right-handed and had normal depth perception were tested in the experimental paradigm. We also ensured that participants did not have allergies to dairy products (because the experiment involved cheese cubes) or adhesives (because the experiment involved mounting markers with spirit gum and medical tape). At the time Experiments 1 and 2 were conducted, all procedures were approved by the Department of Psychology Research Ethics Board (PREB) at the University of Western Ontario. The PREB was a sub-REB of The University of Western Ontario's Research Ethics Board for Non-Medical Research Involving Human Subjects (NMREB) which was organized and operated according to the Tri-Council Policy Statement and the applicable laws and regulations of Ontario (Canada). Participants provided informed consent and were aware that they could terminate testing at any time.

# Experiment 2

Ten right-handed participants (seven males, three females; mean age = 29.6 years) with normal or corrected-to-normal vision took part in this experiment. All participants met the same inclusion criteria as those used in Experiment 1 and had used cutlery (i.e., a fork) from an early age.

# Kinematic Data Collection

Participants' movements were recorded by two three-camera opto-electronic recording systems (Optotrak, Northern DigitalTM, Waterloo, Canada). These systems performed motion capture of the three-dimensional (3-D) positions of Infrared-Emitting Diodes (IREDs) attached to key locations on participants' bodies. Using custom in-house software (OTCollect, programmed by Haitao Yang), the 3-D positions of each IRED were recorded at 100 Hz and used to calculate kinematic measures of transport and grip (e.g., reach velocity and aperture size). Each movement trial was recorded for a period of 3 s, enough time for the participant to reach out to grasp a food item and bring it to the mouth to bite it.

# IRED Positioning

# **Experiment 1**

Zoomed views in **Figure 1** illustrate the placement of the IREDs used to track apertures and transport kinematics. IREDs used for calculating hand grip aperture were placed on the side of the distal thumb and index finger, such that when the thumb and finger were brought together in a ''pinching'' action, these IREDs were immediately adjacent. To measure reach velocity, an IRED was also placed on the side of the index finger knuckle (metacarpophalangeal joint), where the finger meets the hand. IREDs positioned on the hand were secured using cloth medical tape that did not perceptibly alter normal hand movement. To measure and calculate mouth aperture, IREDs were positioned on the left side of the chin and left temple (left mental tubercle of the mandible and sphenoid bone, respectively). Facial IREDs were secured in position using a spirit gum adhesive.

# **Experiment 2**

As in Experiment 1, IREDs were positioned on the participant's chin and temple to calculate mouth aperture measures. However, unlike Experiment 1, the participant's reach velocity in Experiment 2 was determined by calculating the velocity of an IRED placed on the fork (See zoomed view in **Figure 1**). Previous studies investigating tool use in reach-to-grasp tasks have also employed similar IRED positioning (Churchill et al., 1999; Gentilucci et al., 2004).

# Procedure

# Experiment 1

The focus of Experiment 1 was to investigate and compare the kinematics of Hand-to-Food and Hand-to-Mouth movements as performed during a natural feeding action without the use of tools (See **Figure 1A**). The names of the four-conditions performed in Experiments 1 and 2: (1) Hand-to-Food (H2F); (2) Hand-to-Mouth (H2M); (3) Fork-to-Food (F2F); and (4) Forkto-Mouth (F2M)—are such that the first term of the condition name denotes what was used to capture and/or transport the food and the second term denotes the intended destination.

Prior to testing, participants were instructed to begin each trial in an ''initial position'' with the thumb and forefinger in the closed ''pinch'' position resting on the chin, with the mouth closed (upper and lower teeth touching) and with the eyes closed. A mozzarella cheese cube (10, 20, or 30 mm) was placed on the table at a comfortable reaching distance, approximately 30–40 cm away from the participant's torso and along the body midline. Participants were then instructed to open their eyes and wait for an auditory beep (∼ 200 ms) which signified the start of the trial (See **Figure 1Ai**). After this auditory cue, they were to simply reach out (**Figure 1Aii**), pick up the cheese cube with the thumb and forefinger using a precision grip (**Figure 1Aiii**) and bring the cheese cube to their mouth (**Figure 1Aiv**). Because we wanted participants to perform a mouth ''grip'' analogous to the hand grip during grasping, participants were instructed to bite the cube hard enough to hold the cube so that they could release their hand grip (**Figure 1Av**) but not to bite into the cheese. Upon completion of each trial, participants discarded the cheese cube, returned the hand to the initial position, closed their eyes, and waited for the next trial to begin. The absence of teeth marks on the cheese cubes verified that participants did not bite into the cheese. Participants were asked to perform this feeding action as naturally as possible and were given several practice trials prior to testing. The three cube sizes were presented in a randomized order until 15 repetitions of each size was completed. A fresh (unbitten) cheese cube was used on each trial.

# Experiment 2

In Experiment 2, participants followed the same paradigm as in Experiment 1, except that instead of grasping the cheese cube with the thumb and forefinger, the cheese cube was skewered and transported to the mouth using a fork (See **Figure 1Bi–v**).

# Data Parsing

Custom in-house software (OTDisplay, programmed by Haitao Yang) was used to parse the movement data into to-Food and to-Mouth movements. As is typical of many reach-to-grasp kinematic studies, a reach velocity threshold of 20 mm/s was used to demarcate onset and offset of the outward reaches toward the food. If reach velocity did not drop below the 20 mm/s threshold between the outward and inward actions, the local minimum of the velocity trace was used as the offset of the outward reach and the onset of the inward reach. Because the mouth typically continued to close after the hand velocity dropped below this threshold in inward actions, the offset of the inward actions was defined as the point at which reach velocity had dropped below 20 mm/s and the mouth aperture ceased closing (i.e., velocity = 0 mm/s).

# Data Processing

Using the methods outlined below, the following dependent variables were calculated: (1) Oversizing (i.e., difference between maximum grip aperture and final grip aperture); (2) Time of Maximum Grip Aperture; (3) Total Movement Time; (4) Time of Peak Reach Velocity; and (5) Peak Reach Velocity. These computed values were then used in statistical analysis.

Hand grip aperture (i.e., the distance between the thumb and forefinger) was calculated from the vector distance between the thumb and index finger IRED coordinates and is generally accepted as a means of reporting grip aperture. However, as IREDs can only abut one another, there is an offset between these IREDs even during a closed pinch grip. This ''offset'' constant was subtracted from aperture measures so that the calculated vector distance was an accurate measure of the true distance between the gripping surface of the index finger and thumb.

Mouth grip aperture (i.e., the distance between the upper and lower incisor teeth) however, cannot be calculated in the same fashion because IREDs can not be placed directly on the teeth. Thus we chose to place IREDs on the chin and temple, positions that are not prone to occlusion or exaggeration of aperture. To calculate accurate mouth grip aperture, at the beginning/end of the testing session, participants performed calibration trials in which they bit hard plastic blocks of known size (i.e., 10, 20, 30, 40 and 50 mm). By plotting these known aperture values (i.e., block sizes) and the chin/temple IRED vector distances on an XY scatter-plot, we fit a third order polynomial function to these data points. We then used the function to convert the vector distance between chin and temple IREDs displayed during to-Mouth movements into accurate mouth aperture values.

Lastly, the IREDs on the forefinger knuckle (Experiment 1) and on the fork (Experiment 2) were used to calculate reach velocity during the outward and inward movements. Reach velocity was calculated from positional information from all three dimensions.

# Data Analysis

# Extracting Data for Statistical Analysis

All data and movement profiles for both Experiments 1 and 2 were first analyzed with absolute time (in ms) on the x-axis (rather than relative (%) time, as in some studies; **Figures 2A, 3A**). As each movement took a different amount of time to complete (even within the same condition), movement profiles for reach velocity and hand/mouth aperture in each of the experimental conditions needed to be averaged for each participant. To achieve these averaged profiles, the average number of time points it took a given participant to complete the movement of a given condition (Example: Subject A, 10 mm cube, H2F) was calculated. All the movement profiles for that particular condition were then resampled to the average number of time points it took that participant to complete that movement condition. This resampling method ensured that the value of a peak measure and the time at which it occurred were preserved to a greater degree than when trials within a given condition are simply averaged without resampling. This process was repeated for each of the experimental conditions and for each participant. Once this was completed, peak reach velocity, the time of peak reach velocity, aperture oversizing, and the time of peak aperture were extracted from the resampled movement profiles of each participant. These measures, along with total movement time, were then analyzed using a repeated-measures analysis of variance (ANOVA), followed by post hoc, paired-sample t-tests where appropriate. For qualitative comparisons of coordination between transport and grip components for the three conditions in which both transport and grip variables were available (H2F, H2M, F2M), the data were also replotted with the x-axis rescaled to relative (%) movement time and the y-axis

FIGURE 2 | Grip component results. (A) Averaged aperture profiles (as a function of absolute time, in ms) illustrate that Hand-to-Food, Hand-to-Mouth and Fork-to-Mouth reaches display distinctly different grip patterns. (B) The hand oversizes considerably more than does the mouth, regardless of whether or not a fork was used to transport the food item. Moreover, oversizing scales with object size for the hand but not the mouth. (C) The mouth reaches peak aperture later than does the hand, particularly if a fork was used to transport the food.

FIGURE 3 | Transport component results. (A) Averaged reach velocity profiles (as a function of absolute time, in ms) illustrate that Hand-to-Food, Hand-to-Mouth, Fork-to-Food and Fork-to-Mouth reaches display distinctly different transport patterns. (B) Reaches toward the mouth (Hand-to-Mouth and Fork-to-Mouth) take longer than reaches toward food items (Hand-to-Food and Fork-to-Food). Likewise, reaches performed with a fork (Fork-to-Food and Fork-to-Mouth) take longer to execute than those reaches performed with the hand alone. (C) Reaches directed toward the mouth (Hand-to-Mouth and Fork-to-Mouth) attain lower peak reach velocities than do reaches directed toward food items (Hand-to-Food and Fork-to-Food), regardless of whether the reach is performed by the hand alone or a fork. Also, mouth-directed reaches also become slower as object size increases, a pattern not seen in food-directed reaches. (D) Fork reaches directed toward the mouth (Fork-to-Mouth) attain peak velocity far later than all other reach conditions. Similarly, reaches toward the mouth in general (Hand-to-Mouth and Fork-to-Mouth) attain peak velocity later than reaches directed toward food items (Hand-to-Food and Fork-to-Food). Also, Fork-to-Food, Fork-to-Mouth and Hand-to-Mouth reaches each attain peak reach velocity later as object size increases.

(reach velocity for transport component; grip aperture for grip component) rescaled to a percentage of the maximum value (**Figure 4**).

# Grip Component

Although a 2 (Effector: Hand vs. Fork) × 2 (Target: Food vs. Mouth) × 3 (Object Size: 10 vs. 20 vs. 30 mm) ANOVA is appropriate for transport-component measures, this is not possible for grip-component measures because during the F2F condition the hand already has a grip on the fork and therefore cannot provide hand aperture measures. Moreover, a 3 (Condition: Hand-to-Food vs. Hand-to-Mouth vs. Forkto-Mouth) × 3 (Object Size: 10 vs. 20 vs. 30 mm) ANOVA is not appropriate for grip-component measures because one of the three conditions (F2M) involved a different sample of participants than the other two conditions (H2F and H2F). As such, we conducted three 2 (Conditions) × 3 (Object Size: 10 vs. 20 vs. 30 mm) ANOVAs to compare the following conditions and investigate possible interactions with object size: (1) hand vs. mouth aperture in H2F vs. H2M conditions (respectively; within-subjects repeated-measures ANOVA); (2) mouth aperture during H2M vs. F2M conditions (mixed-model ANOVA); and (3) hand vs. mouth aperture in H2F and F2M conditions (respectively; mixed-model ANOVA). The latter ANOVA was included because it enables comparison with the same contrast employed in earlier studies that compared grasping to fork-feeding (Castiello, 1997; Churchill et al., 1999).

**Figure 2A** illustrates how grip aperture changes as a function of time. The two dependent variables related to the grip component that were analyzed statistically (Oversizing and Time of Peak Aperture) are illustrated in **Figures 2B,C**.

# Transport Component

All transport component measures (Total Movement Time, Peak Reach Velocity and Time of Peak Reach Velocity) were analyzed using a 2 (Effector: Hand vs. Fork, between-subjects) × 2 (Target: Food vs. Mouth, within-subjects) × 3 (Object Size: 10 vs. 20 vs. 30 mm, within-subjects) mixed ANOVA.

**Figure 3A** illustrates reach velocity as a function of time. The three dependent variables of the transport component are illustrated in **Figures 3B–D**.

# RESULTS

One of the more salient features of **Figures 2A, 3A** is that the average durations of the movements across conditions are quite different. In many kinematic studies, where the movements performed do not differ as widely as those of the current experiments, it is commonplace to resample the movements to quantify kinematic measures of interest in terms of the percentage of movement time (Examples; Jeannerod, 1984; Marteniuk et al., 1990; Herbort and Butz, 2010). Here, we find that the differences in movement time across conditions (e.g., 500 ms) are so large that resampling would give a misleading impression of the temporal unfolding of the movements. Therefore, our primary analyses on the data (**Figures 2, 3**) were computed on a real-time scale (in ms), which is later replotted on a relative-time scale (**Figure 4**) for comparisons between relative transport and grip timing.

# Aperture Oversizing

The most striking result for the aperture measures (See **Figure 2B**) was that the hand oversized much more during food grasping (H2F) than the mouth oversized during feeding with the hand (H2M) or a fork (F2M). In fact, the mouth typically only opened ∼4–11 mm larger than the food while the hand typically opened ∼11–27 mm larger. Furthermore, oversizing scaled strongly with object size for the hand, but not for the mouth. Mouth oversizing and its relationship to object size did not differ between feeding with the hand and feeding with a fork.

Statistical analyses supported these observations. There was a significant main effect of condition, main effect of object size, and interaction in the ANOVA comparing H2F vs. H2M × 3 sizes (all p < 0.001). Similarly, there was a significant main effect of condition, main effect of object size, and interaction in the ANOVA comparing H2F vs. F2M × 3 sizes (all p < 0.001). The ANOVA comparing mouth oversizing when feeding with the hand vs. fork (H2M vs. F2M) showed a main effect of object size (p < 0.001) but only a trend towards a main effect of condition (p = 0.06) and a trend toward an interaction (p = 0.06). Post hoc t-tests (Bonferroni-corrected p value of 0.05 for 18 comparisons = p < 0.0028) showed significantly greater oversizing for the hand than the mouth, regardless of whether the mouth was fed by the hand or a fork, at the two largest object sizes but not the smallest. In addition, t-tests showed that the hand oversized significantly more at larger sizes (20 vs. 10 mm, 30 vs. 20 mm, and 30 vs. 10 mm; p < 0.0028). In contrast, during H2M movements, the mouth showed less oversizing at the largest size (30 mm) than the middle size (20 mm), but no difference between the small size and the two larger sizes. Similarly, during F2M movements, the mouth showed less oversizing at the largest size (30 mm) than both of the smaller two sizes (10 and 20 mm; p < 0.001).

# Time of Peak Aperture

The results (See **Figure 2C**) showed that during grasping (H2F) the hand attains peak aperture significantly earlier than the mouth does when fed by hand or by fork (H2M and F2M). Also, mouth peak aperture occurs later when feeding with a fork (F2M) in comparison to feeding with the hand (H2M).

Statistical analyses supported these observations. There was a significant main effect of condition, main effect of object size, and interaction in the ANOVA comparing H2F vs. H2M × 3 object sizes (all p < 0.005). Similarly, there was a significant main effect of condition, main effect of object size, and interaction in the ANOVA comparing H2F vs. F2M × 3 object sizes (all p < 0.005). The ANOVA comparing time of peak mouth aperture when feeding with the hand vs. fork (H2M vs. F2M) showed a main effect of both object size and condition (p < 0.001), but no significant interaction. Post hoc t-tests (Bonferroni-corrected p value of 0.05 for 18 comparisons = p < 0.0028) showed that the hand reached peak aperture earlier than the mouth when feeding with the hand (H2F vs. H2M) for the two larger object sizes (20 and 30 mm; p < 0.0028) and approached our conservative significance level for the smallest object (10 mm; p = 0.006). Also, the mouth attained peak aperture later as object size increased when feeding by hand (H2M; 10 mm vs. 30 mm, p < 0.001), a pattern not present in hand aperture when reaching toward the food (H2F). In addition, the hand reached peak aperture (during hand feeding) earlier than the mouth when feeding with a fork (H2F vs. F2M) for all object sizes (p < 0.001). The main effects of Hand vs. Fork and Object Size during reaches to the mouth (H2M vs. F2M) demonstrate that the mouth reaches peak aperture later when a fork is used to feed oneself and that mouth reaches peak aperture later as object size increases.

# Total Movement Time

The two most notable patterns seen in total movement time (See **Figure 3B**) are that: (1) reaches made to the mouth (H2M and F2M) took longer to perform than those reaches directed toward the food (H2F and F2F) and (2) reaches with the fork (F2F and F2M) took longer than reaches performed with the hand (H2F and H2M).

Statistical analyses supported these observations. These effects were verified in the full (2 × 2 × 3) ANOVA, which showed significant main effects of Target (Food vs. Mouth; p < 0.005; within) and Effector (Hand vs. Fork reaches; p < 0.001; between) with no interaction between the two. Although there was no main effect of Object Size, there was a significant interaction between Target (Food vs. Mouth) and Object Size (p < 0.05) but no threeway interaction. To explore the Target × Object Size interaction, we collapsed across Effector and conducted post hoc t-tests (Bonferroni-corrected for nine comparisons, p < 0.0056) which only found a significant difference between outward movements toward the food vs. inward movements toward the mouth at 30 mm (p < 0.001).

# Peak Reach Velocity

One notable feature of **Figure 3C** is that reaches toward the mouth (H2M and F2M) display lower velocities than those reaches directed toward the food. Also, Peak Reach Velocity during Hand-to-Mouth and Fork-to-Mouth movements became slower as object size increased, whereas Hand-to-Food and Forkto-Food reach velocity was unaffected by object size.

Statistical analyses supported these observations. The ANOVA revealed a significant main effect of both Target and Object Size (p < 0.005 and p < 0.001, respectively), a significant interactions of Effector × Target (p < 0.05), a significant interaction between Target × Object Size (p < 0.05) but no three-way interaction. To examine the Effector × Target interaction, we collapsed the data across Object Size, and performed post hoc t-tests (Bonferroni-corrected p value of 0.05 for four comparisons = p < 0.0125). These revealed that the interaction was driven by the fact that fork reaches directed toward the mouth were performed slower than fork reaches directed toward the food (p < 0.005), whereas a similar comparison of reaches performed with the hand alone failed to reach significance (p = 0.462). To examine the Target × Object Size interaction, we collapsed the data across Effector, and performed post hoc t-tests (Bonferroni-corrected p value of 0.05 for nine comparisons = p < 0.0056). These revealed that reaches toward the mouth were slower than those reaches directed toward the food at all object sizes (10 mm, p = 0.001; 20 and 30 mm, p < 0.001) and that reaches toward the mouth became slower as object size increased (10 mm vs. 20 mm and 10 mm vs. 30 mm, p < 0.001; 20 mm vs. 30 mm, p = 0.002).

# Total Distance Travelled

Note, however, an important caveat in interpreting peak velocity data. As visual inspection of **Figure 3A** shows, the total distance travelled (i.e., area under the curve) differed between conditions even though the physical distance between food and mouth remained constant.

First, although the total distance travelled was similar across sizes within a condition, it was longer for actions with the fork (43.0 cm) than actions with the hand (36.8 cm). IREDs used to record reach trajectories for hand and mouth were placed at similar distances from the fingertips or fork tip, respectively, and actions with both effectors required a ∼180-degree rotation of the wrist. Nevertheless, the rotation arc was longer for the fork than the hand because the IRED was further from the wrist (pivot point).

Second and more interestingly, even within the same effector (and IRED), the total distance travelled differed for movements toward the food vs. toward the mouth. When using the hand, movements toward the food followed a longer path (H2F: 37.7 cm) than movements toward the mouth (H2M: 35.8 cm). We speculate that the hand may take more of an an arc trajectory en route to grasping the food (to ensure the index finger doesn't hit the far edge of the cheese cube placed on the table) but more of a straight trajectory when delivering the food to the mouth. In contrast, when using the fork, the difference was reversed, with participants following a longer trajectory when bringing the fork to the mouth (43.8 cm) vs. the food (42.2 cm). We speculate that the fork does not need to follow an arc when stabbing the food (because it is aimed at the centre of the cube and doesn't have to clear the edges) but may follow more of an arc when feeding such that the food approaches approximately perpendicular to the teeth. These speculations would be worth further investigation in future studies and would benefit from combining video recording of the actions in addition to kinematic tracking (e.g., Karl et al., 2012).

# Time of Peak Reach Velocity

One of the more notable features of **Figure 3D** is that fork reaches directed toward the mouth (F2M) attained peak velocity later than both fork reaches toward food (F2F) and hand reaches toward the mouth (H2M). It was also determined that reaches toward the mouth (H2M and F2M) attained peak velocity later than reaches directed toward the food (H2F and F2F). Also, there is evidence that both fork-reaches and reaches toward the mouth attain peak velocity later as object size increases.

Statistical analyses supported these observations. The ANOVA revealed significant main effects of Effector (p < 0.01), Target and Object Size (both p < 0.001). Moreover, there was a significant Effector × Target interaction (p < 0.001), a significant Target × Object Size interaction (p < 0.001), and a significant Effector × Object Size interaction (p < 0.05), but no three-way interaction. To investigate the Effector × Target interaction, we collapsed across Object Size and performed post hoc t-tests (Bonferroni-corrected p value of 0.05 for four comparisons = p < 0.0125). These revealed that reaches toward the mouth with a fork (F2M) attained peak reach velocity later than reaches toward the food with a fork (F2F, p < 0.001) and reaches toward the mouth with the hand (H2M, p < 0.001). There was no difference between reaches to the food vs. mouth when the hand alone was used (H2F and H2M). To investigate the Target × Object Size interaction (Bonferroni-corrected p value of 0.05 for nine comparisons = p < 0.0056), we collapsed across Effector and conducted post hoc t-tests. These revealed that reaches toward the mouth attained peak velocity later than reaches directed toward the food at all object sizes (p < 0.001). Also, there was some evidence to suggest that reaches toward the mouth attain peak velocity later as object size increases (10 mm vs. 30 mm, p < 0.005). Lastly, to investigate the Effector × Object Size interaction, we collapsed across Target and conducted post hoc t-tests (Bonferroni-corrected p value of 0.05 for nine comparisons = p < 0.0056). These revealed that fork-reaches attain peak velocity later as object size increases (10 mm vs. 30 mm, p < 0.005).

# Coordination of Transport and Grip Components

Although viewing the data in real time (that is, with ms on x-axis) gives the most accurate portrayal of how grasping and feeding actions unfold, it can also be valuable to examine the relative timing, which affords an easier comparison of how the transport and grip components of the actions are coordinated (cf. Churchill et al., 1999). **Figure 4** shows the transport data (reach velocity) and grip data (aperture) replotted as a percentage of maximum over relative (%) time. These plots reveal that the transport component (reach velocity) unfolds quite consistently, with peak velocity achieved approximately 40% of the way through the movement. In contrast, the grip component (hand or mouth aperture) has a very different profile during grasping than feeding actions (regardless of whether feeding occurs using the hand or a fork). First, the hand aperture during grasping begins to open considerably earlier (with the hand aperture achieving 50% of maximum approximately 1/3 of the way through the movement) than the mouth does (with the mouth aperture achieving 50% of maximum over halfway through the movement). Second, the hand aperture peaks somewhat earlier (∼70% of total movement time) than the mouth aperture (∼80% of total movement time), though the timing differences appear less pronounced than in the earlier phases of aperture opening.

# DISCUSSION

Our results demonstrate that when grasping and feeding movements are directly compared under highly similar conditions, the two actions clearly differ in the degree to which the hand and mouth oversize. Consistent with a large body of research on hand kinematics (beginning with Jeannerod, 1981, 1984, 1986), we found that the hand opens larger than the target during approach; moreover, maximum grip aperture scales with the size of the target. Surprisingly, however, we found that when the mouth directly bites the food items (cheese cubes in our case), oversizing is relatively small and nearly constant across object size. Although actions with a fork led to slower movements, particularly when the fork was brought to the mouth for feeding, the use of a fork had surprisingly little effect on mouth aperture.

The differences we observed between oversizing of the mouth and hand differ from the results of past investigations (Castiello, 1997; Churchill et al., 1999), which reported that the mouth aperture during feeding showed a similar degree of oversizing as the hand aperture shows during grasping. In these earlier studies, participants fed themselves with a fork (Castiello, 1997) or spoon (Churchill et al., 1999); however, the present results suggest the key difference between their results and ours was not the use of cutlery. That is, in our study, we found that the mouth showed little oversizing regardless of whether the hand or a fork was used to deliver the food. Rather, recall that in our study, participants bit the cheese cube directly with the teeth rather than pulling it off the implement into the mouth. We proposed that this approach makes the action of feeding have similar demands as grasping with the hand because both actions involve the closing of a bodily aperture upon—rather than around—the food item. In addition, we argue that our method of inferring the bite aperture (i.e., the aperture between the teeth) is more accurate than the previously used methods. Under these circumstances, we find far less oversizing of the mouth than the hand, particularly at large target sizes.

# Why does the Hand Oversize More than the Mouth?

The first and most obvious explanation for the greater oversizing of the hand than the mouth is simply that biomechanically the hand has a larger range of movement than the jaw; however, closer examination suggests a strong version of this explanation does not suffice. To investigate this hypothesis, we compared aperture oversizing displayed by the mouth, taking into consideration the maximum aperture the mouth is capable of producing. For the participants tested, the maximum aperture for the mouth was slightly larger than 50 mm (based on the largest block, 50-mm, used in calibration trials) so theoretically, participants could have reached a considerably larger maximum aperture for all sizes of the cheese cubes. Moreover, comparisons of apertures across the sizes argue against a hard limit. For example, in **Figure 2A**, the mouth reaches a maximum aperture of 36 mm when biting the largest (30-mm) object, yielding 6 mm of oversizing. Thus when biting the smallest (10-mm) object, theoretically the mouth could still open to 36 mm—but it doesn't. Rather, the mouth opens only to a maximum of 19 mm, 9 mm larger than the food. Similarly, when biting the medium-sized object (20-mm), the mouth could open to a larger maximum aperture (36 mm) than it does (30 mm).

Note, however, that these arguments do not preclude weaker versions of a ceiling effect argument, including the possibility that participants minimized mouth oversizing because opening the mouth wider than strictly necessary may be relatively more uncomfortable than opening the hand wider than necessary. It also does not preclude the possibility that opening the mouth wider than strictly necessary may be considered impolite by one's dinner companions (at least if they are adults). Interestingly, the fact that oversizing is less with the large (30-mm) cheese cube than the medium (20-mm) and small (10-mm) cheese cubes, may reflect ''padded ceiling effect'' (as suggested by a reviewer) in which the closer one gets to the limit the harder it pushes back. If indeed, this argument holds, it may partially account for why we see less oversizing (particularly for our 30-mm object) than Castiello did as he used smaller food items (5- and 20-mm cheese cubes).

A second possible explanation for the oversizing differences seen here is that Hand-to-Food and Hand-to-Mouth movements likely rely upon different sensory information for planning and adjustment. In particular, during grasping with the hand, visual information is available throughout the movement (including the visual preview to guide hand preshaping and visual feedback of the hand to enable online corrections) but haptic information is only provided following contact. In contrast, during feeding with the hand, an initial visual preview is available but visual feedback is limited (as the mouth is unseen, the view of the hand and food degrades during approach, and participants' gaze does not follow the food; de Bruin et al., 2008) whereas haptic feedback about object size (from the hand) is available throughout the movement. Given that factors that increase uncertainty (e.g., removal of feedback) often lead to increased oversizing (Wing et al., 1986; Athènes and Wing, 1989), one possible explanation for the oversizing differences we observed is that participants had less uncertainty about object size during Hand-to-Mouth actions than Hand-to-Food actions. However, our fork-feeding results call this suggestion into question. That is, during fork-feeding, participants have less precise information about the food size (as it is no longer conveyed by hand aperture, though weight may still provide a partial cue), yet the maximum mouth aperture remained similar.

A third possible explanation could be that the mouth oversizes less due to a difference in the speed-accuracy trade-off between the goals. Put another way, when feeding, accuracy may be emphasized over speed to a greater degree than when grasping. Due to the slowed movement and increased accuracy, less oversizing may be needed.

A fourth possible explanation is that aperture closure strategies may differ between hand and mouth. Gripping an object is rarely the end goal of a hand grasping action; rather, typically it is the means to acquire an object for further manipulation such as lifting, moving, manually exploring—or even feeding. Indeed, participants show little or no oversizing when no such manipulation is possible, such as in flat pictures (Holmes and Heath, 2013) or grasps performed toward remembered objects that are no longer present (Goodale et al., 1994, Experiment 2). Manipulative actions require a firm grip to prevent slippage. One of the strategies proposed for achieving a stable precision grip is to have the index finger and thumb approach the target perpendicular to the respective surfaces at locations that transect the target's centre of mass (Smeets and Brenner, 1999). This perpendicular approach strategy necessitates at least some grip oversizing which may unfold with a particular curvature to ensure smooth movements. In contrast, feeding actions such as our cheese biting task here serve different functions, typically to chew the food and/or bring the food to the tongue to initiate swallowing. Moreover, the effects of gravity may be more relevant for hand grasping, where slippage could lead to dropping the food, than mouth biting, where slippage is less consequential. As such, it may be that the benefits of oversizing are stronger for hand grasping to enable a smooth perpendicular approach of the digits than for biting. Consistent with this argument, others have found a similar absence of oversizing when participants grasped body parts on the face (Edwards et al., 2005), which are in no danger of slipping.

Of course, it may be that multiple factors, including all of the above, make some contribution to the kinematic differences between our conditions.

The relationships between aperture oversizing and object size are harder to interpret. Here, we found that hand oversizing increases with object size while mouth oversizing decreases (between the medium-sized and large objects). Note, however, that past studies of hand grip aperture as a function of object size have found mixed effects, with many finding a slope <1 (indicating that oversizing decreases with object size) with others finding a slope >1 (indicating that oversizing increases with object size (see meta-analysis in Smeets and Brenner, 1999, Figure 6A). Many studies do not use a careful calibration to examine oversizing per se but rather display the raw measurement of the distance between markers on the finger and thumb, which can include an offset. In addition, the effects could well depend on the range of sizes employed, with larger sizes being more likely to reveal hard or soft ceiling effects.

# Would the Mouth Always Show Less Oversizing than the Hand?

We have no doubt that many other potential variables could affect feeding strategies. These include the nature of the food and the typical means of feeding. Here, we used cubes of firm cheese (mozzarella) and instructed participants to bite the food without swallowing it. Our rationale for this was to make the biting action as similar as possible to the grip used in conventional grasping studies. However, we may have seen different outcomes for example if the subjects had been eating cubes of a softer cheese (e.g., brie) and simply using the teeth to move the food toward the tongue and throat or if we had used a harder cheese (e.g., parmesan) and instructed to take a bite. And of course there are many other foods (e.g., apples, popcorn) that have distinctive eating strategies. Our results also do not speak to the development of eating strategies. Anecdotally, infants open their mouths widely when being fed, although this may be due in part to the uncertainty of being fed by another person (Ferri et al., 2010). Nevertheless, our findings provide an interesting starting point for examining the kinematic strategies of feeding.

Interestingly feeding actions may affect not just the kinematics of the mouth but also the kinematics of the hand as it acquires the food during grasping. Specifically, when participants grasp a piece of food with the intention of placing it into the mouth, the maximum grip aperture of the hand does not open as wide as when they grasp the food with the intention of placing it in a bib below the mouth (Flindall and Gonzalez, 2013). This effect is only found with the right hand but not the left, which has led to the suggestion that the right hand is specialized for grasp-to-eat actions (Flindall and Gonzalez, 2013). Moreover, it occurs even if participants only bring the food to the mouth but do not actually eat it (Flindall and Gonzalez, 2014). Because our movement sequence involved grasping the food before bringing it to the mouth, these results suggest that the difference we observed between hand and mouth apertures would be even stronger if the grasping phase had involved a different goal (such as moving the cheese to a different location on the table).

# Neuroanatomy and Development of Grasping and Feeding Actions

In addition to behavioral differences, hand and mouth actions may rely on at least partially different neural substrates. For example, neurons in different divisions of premotor cortex respond to grasping vs. feeding actions (Rizzolatti et al., 1987, 1988). As detailed in the introduction, neurostimulation studies in other primate species (Graziano et al., 2002; Kaas et al., 2013) have revealed that hand grasping actions and feeding actions are evoked in different cortical sites within motor, premotor, and parietal cortex. These results suggest that both grasping and feeding (along with other actions like defensive movements and locomotion) may be ethologically relevant, fundamental actions within the motor repertoire that are associated with different properties (such as the region of space in which the actions occur or the reliance on different types of sensory information; Graziano and Aflalo, 2007).

While human neuroimaging has clearly identified the neural substrates for grasping and reaching actions (Binkofski et al., 1998; Culham et al., 2003; Castiello, 2005; Cavina-Pratesi et al., 2010; Turella and Lingnau, 2014), surprisingly little research has been done to investigate the neural substrates of feeding actions. In large part, this is due to technical limitations, especially with the predominant neuroimaging technique, functional MRI (fMRI). For example, our own attempts to study real feeding actions have been hampered by severe artifacts related to the movement of the mass of the arm (Barry et al., 2010), a larger problem for feeding actions (which recruit more proximal musculature: shoulders and biceps) than grasping (which can be performed predominantly using distal musculature: wrist and hand; Culham et al., 2003).

One early neuroimaging study used positron emission tomography (PET), which is not susceptible to mass motion artifacts, to examine human brain activation while participants grasped or bit a piece of candy off a fork moved toward the participant by the experimenter (Castiello et al., 2000). They reported similar activation for grasping and biting; however, the sample size was small (n = 5), the data were heavily smoothed (12-mm kernel), and the two actions were not directly contrasted. Thus this result suggests coarse similarity; however, it is possible that further investigation could reveal differences.

Indeed several fMRI studies have found that observation of actions with the hand and mouth (and in some cases other body parts) evoked activation in different, somatotopically organized foci within parietal, premotor, and lateral occipito-temporal cortex (Buccino et al., 2001; Wheaton et al., 2004; Pelphrey et al., 2005; Orlov et al., 2010). Specifically, while observation of hand grasping actions evoked activation in the anterior intraparietal sulcus and ventral premotor cortex, observation of mouth actions (such as biting an apple or chewing) yielded activation below these sites, in the anterior part of the inferior parietal lobule and inferior frontal gyrus, in or near Broca's area (Buccino et al., 2001). In addition, while hand images activate the lateral occipitotemporal cortex (Bracci et al., 2010, 2012), mouth images activate a more anterior focus in the superior temporal sulcus (Wheaton et al., 2004; Pelphrey et al., 2005) and a more posterior/inferior focus (Orlov et al., 2010). The different neural substrates for hand actions and mouth actions raise the question of which foci would be activated by hand-to-mouth actions. One possibility is that such actions would evoke somatotopic activation for both effectors (hand and mouth); however, based on the neurostimulation studies in non-human primates (Graziano et al., 2002; Kaas et al., 2013), we expect that hand-to-mouth actions likely recruit different zones of sensorimotor cortex than hand-to-object actions like grasping.

Feeding and grasping movements may also differ in their developmental trajectories. Neonates are not only capable of making deliberate reaches to their mouth, they also make anticipatory mouth opening movements (Rochat et al., 1988; Blass et al., 1989; Takaya et al., 2003), suggesting that functional hand-mouth coordinated movements have developed prior to birth. Ultrasound movies of human fetuses have demonstrated that more than half of the arm movements produced (19–35 weeks gestation) resulted in hand contact with the mouth accompanied by anticipatory mouth opening, which suggests that these were intentional hand-mouth movements (Myowa-Yamakoshi and Takeshita, 2006). As little light is present in utero, it has been suggested that these movements are learned and performed using proprioception alone (Butterworth and Hopkins, 1988). In comparison, reaches to external targets are thought to develop rapidly over the first year of life and become fine-tuned throughout much of childhood (for review and longitudinal study, see Schneiberg et al., 2002), predominately guided by visual input; however, some propose that objectdirected actions may initially rely more on proprioceptive/haptic guidance than visual guidance (e.g., Clifton et al., 1993; Karl and Whishaw, 2014).

# Timing and Coordination of Movements

In addition to differences in the magnitude of oversizing, clear differences were also observed in the timing of the movements. Most notably, feeding actions took longer than grasping actions, particularly when a fork was used and the relative coordination of aperture opening and reaching differed between grasping and feeding.

Taken together, these results suggest that the well-established temporal coordination between the transport and grip component differs for the hand and mouth. Perhaps because the mouth requires less oversizing, it can begin opening and reach peak aperture relatively later than the hand does because less time is required for closure.

One other notable difference between grasping and feeding actions is the combination of effectors involved. Grasping actions predominantly utilize arm, wrist and hand movements (as in most laboratory studies of grasping, objects are placed easily within reach and little torso movement is required). However, feeding actions require coordination of the arm, wrist and hand with the mouth, head and torso. During feeding, the actor may use trunk and head movements to a greater degree, especially when greater accuracy is required (e.g., taking a liquid vs. solid from a spoon; van der Kamp and Steenbergen, 1999).

# CONCLUSION

In conclusion, unlike previous studies of feeding actions which showed that grip and transport kinematics of grasping and feeding movements are similar, we show here that when equivalent movements of the hand and mouth are compared,

# REFERENCES


numerous kinematic differences become apparent. In particular, when using their fingers to feed themselves, participants oversize the mouth considerably less than they oversize the hand when grasping. Although a number of explanations are possible, the one we favor is that grasping and biting may utilize different strategies. Moreover, they may rely on partially different neural substrates. The use of a fork to feed slowed the movement but had negligible impact on the grip component, including oversizing, suggesting that the key determinant of oversizing is the effector employed. Although these studies do not definitively explain the reasons for different strategies, they suggest that kinematically, and perhaps also neurally, feeding is not merely ''grasping with the mouth'' but rather has it own strategies worthy of further investigation.

# AUTHOR CONTRIBUTIONS

DQ and JC conceived of the research and wrote the paper. DQ collected and analyzed the data.

# ACKNOWLEDGMENTS

This research was funded by Discovery Grants from the Natural Sciences and Engineering Research Council of Canada (249877- 2006-RGPIN) to JCC. We thank Haitao Yang for technical assistance.


in the medial parieto-occipital cortex of the macaque monkey. J. Neurosci. 30, 342–349. doi: 10.1523/JNEUROSCI.3800-09.2010


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Quinlan and Culham. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Are All Spatial Reference Frames Egocentric? Reinterpreting Evidence for Allocentric, Object-Centered, or World-Centered Reference Frames

#### Flavia Filimon1, 2 \*

<sup>1</sup> Adaptive Behavior and Cognition, Max Planck Institute for Human Development, Berlin, Germany, <sup>2</sup> Berlin School of Mind and Brain, Humboldt Universität zu Berlin, Berlin, Germany

The use and neural representation of egocentric spatial reference frames is well-documented. In contrast, whether the brain represents spatial relationships between objects in allocentric, object-centered, or world-centered coordinates is debated. Here, I review behavioral, neuropsychological, neurophysiological (neuronal recording), and neuroimaging evidence for and against allocentric, object-centered, or world-centered spatial reference frames. Based on theoretical considerations, simulations, and empirical findings from spatial navigation, spatial judgments, and goal-directed movements, I suggest that all spatial representations may in fact be dependent on egocentric reference frames.

Keywords: allocentric, object-centered, egocentric, spatial reference frames, parietal sensorimotor transformations, place cells, cognitive map, perception and action

## Edited by:

J. Douglas Crawford, York University, Canada

#### Reviewed by:

Katja Fiehler, Giessen University, Germany Matthias Niemeier, University of Toronto Scarborough, Canada

#### \*Correspondence:

Flavia Filimon flavia.filimon@gmail.com

Received: 08 July 2015 Accepted: 16 November 2015 Published: 09 December 2015

#### Citation:

Filimon F (2015) Are All Spatial Reference Frames Egocentric? Reinterpreting Evidence for Allocentric, Object-Centered, or World-Centered Reference Frames. Front. Hum. Neurosci. 9:648. doi: 10.3389/fnhum.2015.00648

# INTRODUCTION

Do animals use spatial reference frames that are independent of an egocentric viewpoint? In other words, does the brain represent map-like spatial layouts, or spatial locations of objects and landmarks, in an allocentric, or "other-centered" spatial reference frame, independent of the ego's perspective or location? Does the choice of spatial reference frame depend on (passive) perception vs. sensorimotor interactions with the environment, such as target-directed movements or navigation?

It is well-established that neurons in many brain regions, especially parieto-frontal cortex, represent the spatial location of objects in egocentric spatial reference frames, centered on various body parts such as the eye (retina), the head, or the hand (Colby, 1998; Hagler et al., 2007; Sereno and Huang, 2014). However, whether the brain also represents spatial locations of external objects relative to other objects in an allocentric or object-centered spatial reference frame, or constructs an abstract map of such relationships that is independent of the egocentric perspective, is debated (Bennett, 1996; Driver and Pouget, 2000; Wang and Spelke, 2002; Burgess, 2006; Wehner et al., 2006; Rorden et al., 2012; Li et al., 2014).

Here, I review empirical (behavioral, neuropsychological, neurophysiological, and neuroimaging) evidence for and against allocentric vs. egocentric spatial representations. In addition, I discuss theoretical considerations and computational models addressing this distinction.

Based on theoretical considerations and empirical evidence, I suggest that object-centered, allocentric, or world-centered spatial representations may be explained via egocentric spatial reference frames. I shall argue that allocentric task effects could alternatively be explained via the following processes:


The latter point suggests that landmark or scene recognition via viewpoint-matching is more akin to object recognition than a spatial representation of object coordinates and locations relative to an external, environment-based reference frame. As such, the brain might not rely on allocentric spatial reference frames either for spatial judgments in spatial perception, or during navigation, or in sensorimotor transformations for goal-directed movements (e.g., grasping, pointing, or eye movements) toward external objects. Thus, I will argue that neither the way we encode space, nor the way we interact with space, need make use of allocentric spatial reference frames independent of egocentric representations. Object-based representations do exist, especially in the ventral visual stream, but are not spatial in the sense of referring locations external to the viewer to another external object. Ventral object-centered representations are essentially akin to object recognition, with spatial decisions remaining anchored to a fundamentally egocentric spatial reference frame.

I will commence with some theoretical examples for why it is difficult if not impossible to relate spatial locations (whether left or right, up or down, or simply "the center of ") to external, non-egocentric coordinates. I will then review empirical evidence for different spatial reference frames in navigation, spatial judgments, and goal-directed movements (interactions with spatial targets), as well as computational (simulation) explanations for the effects observed. By attempting to unify a wide range of findings from multiple research areas, this review will necessarily not be fully comprehensive within each domain, but will instead highlight representative studies. Finally, I will conclude with a new suggested categorization of networks contributing to spatial processing, as well as with several predictions made by the egocentric account.

# THEORETICAL CONSIDERATIONS: CAN SPATIAL REPRESENTATIONS BE INDEPENDENT OF THE EGOCENTRIC PERSPECTIVE OR POSITION?

Different definitions have been used to define the term "allocentric." Klatzky (1998), for instance, distinguishes between three "functional modules": egocentric locational representation, allocentric locational representation, and allocentric heading. Whereas egocentric locational representations reference locations of objects to the observer (ego), allocentric representations reference object locations to space external to the perceiver. For instance, positions could be represented in Cartesian or Polar coordinates with the origin centered on an external reference object (Klatzky, 1998). Allocentric heading, on the other hand, defines the angle between an object's axis of orientation and an external reference direction. Other authors have proposed distinctions between "allocentric" and "object-centered" representations (e.g., Humphreys et al., 2013).

Although different authors have used the terms "allocentric," "object-centered," or "world-centered" in many different ways, the majority of the spatial cognition literature has used these terms to refer to representations of spatial relationships between objects or landmarks that do not reference objects' locations to the viewer's body, but to other, external objects (Foley et al., 2015). Here, I shall refer to "allocentric," "object-centered," "objectbased," "object-relative," "world-centered," or "cognitive maplike" interchangeably, to refer to the representation of the spatial location of an object relative to that of another external object, independent of the ego's position or orientation, whether present, imagined, or remembered. This is equivalent to Klatzky's (1998) allocentric locational representation.

In contrast, I shall referto "egocentric" or "ego-relative" spatial reference frames whenever the observer invokes the position or orientation of the present, remembered, or imagined (e.g., mentally rotated or translated) self, as opposed to an external landmark, to represent the location of external objects.

A spatial reference frame means the receptive field (RF) of a neuron, or the response of the neural population as a whole, is anchored to a particular reference point. For instance, an eyecentered reference frame moves with the eyes (Colby, 1998). A cell preferring stimulation in the left visual field only signals objects when they fall in that cell's RF, which is anchored to the retina. As the eyes move across the visual field, objects' spatial locations change constantly relative to the retina (e.g., an object "left of the eyes" can suddenly be "right of the eyes"). Objects' spatial locations are thus constantly updated such that different eye-centered cells, with spatial RFs tiling the visual space, signal the new eye-centered location. An external, "abstract" reference frame, on the other hand, would represent object locations relative to an external reference point, independent of where the observer is (e.g., the location of the microwave relative to the fridge).

For the purpose of this paper it is irrelevant whether neurons with similar reference frames are arranged in a map of space, such as a retinotopic map of space where cells with similar preferences (e.g., "left half of space") are clustered together. Cells can be eyecentered and yet be part of either an orderly retinotopic map or a scrambled map of space, with neighboring eye-centered cells having retinal response fields in different locations (Filimon, 2010). I also do not distinguish between reference frames or maps of space represented at the single cell or population level—e.g., the entire population may signal "left of me," but individual cells' responses may be less clear-cut. The important point addressed here is whether any neural representations, at the single-cell or population level, explicitly signal spatial relationships between objects independent of their spatial location relative to the ego, i.e., whether an explicit object-centered or allocentric spatial representation is formed at whichever computational stage of processing. As defined by Deneve and Pouget (2003), an explicit representation would involve neurons with invariant responses in object-centered coordinates—e.g., the cell should only respond to "left of object," regardless of where the object is relative to the ego.

Klatzky (1998) also made the distinction between primitive parameters conveyed by a spatial representation and derived parameters which can be computed from primitives in one or more computational steps. Thus, allocentric location is a primitive parameter in an allocentric locational representation, just like egocentric location is a primitive in the egocentric locational representation (Klatzky, 1998). However, I will review evidence that suggests that allocentric location representations are unlikely to be primitives, but are instead derived from egocentric representations at higher levels of the computational hierarchy, and may not be represented explicitly.

**Figure 1** shows several examples of spatial arrangements that would at first instance appear to be object-based, allocentric spatial relationships. For instance, one could refer to left/right terminology to describe the spatial location of a window relative to a door.

In **Figure 1A** (left) one could argue that the window is "left of the door" and the door is "right of the window," regardless of whether the observer is located left of the house (where both the window and the door are on the egocentric right) or to the right of the house (where both objects are on the egocentric left). The fact that the window is "left" of another object, even though it is egocentrically on the right, could be interpreted as an objectcentered, ego-independent spatial representation. However, as can be seen in **Figure 1A** (right), this arrangement is nevertheless dependent on the egocentric viewpoint. Once the observer has walked inside the house, viewing the door and window from the inside, the left–right relationship is reversed: now the window is to the right of the door and the door is to the left of the window. This example demonstrates the ego-dependence of

"left" and "right" spatial judgments. The observer merely has to imagine the house aligned with the egocentric center point, such that the house's left (right) and the egocentric left (right) are congruent. Such imagined rotation or imagined translation that transforms the ego's orientation or position relative to an object, or conversely the position of an object relative to the ego, has been called imaginal updating (Klatzky, 1998). Since the definition of left and right depends on the egocentric perspective, this definition of left/right relative to the object (the house or any landmark on it) is not an example of true allocentric or object-centered (ego-independent) spatial representations.

**Figure 1B** demonstrates another possible way of conceptualizing object-centered spatial representations. Instead of using spatial judgment terms such as "left" and "right," which appear tied to egocentric perspectives, one could use "center of an object." Clearly something that is in the center of an object should remain in the center of the object regardless of whether the observer is in front or behind that object. However, as **Figure 1B** demonstrates, establishing the center point of, e.g., a wall, remains dependent of the egocentric perspective: as soon as the observer is positioned at one end of the object (e.g., at the left end of the wall), the distorted retinal perspective obtained from that (egocentric) location makes it much harder to determine where the center point of the wall is. This may not apply to small objects that can be foveated. However, for small objects (which can be mentally shifted to line up with the fovea), the egocentric left/right and the object's left/right are congruent, and the center point can be estimated based on retinal extent. Alternatively, small objects may be treated as a point in space. As explained above, the critical test for an allocentric representation is independence of object locations from any egocentric perspective, thus relying on abstract spatial relationships between objects independent of the observer.

Avoiding "left/right" and "center of " terminology, one might devise a stimulus (**Figure 1C**) where the relative spatial alignment of two objects is what matters (e.g., the alignment on a clock between the minutes hand and the minute mark corresponding to 25 min). Does the clock indicate 25 min past the hour? As **Figure 1C** (right) demonstrates, this depends on the egocentric perspective: viewed from the side, the alignment between the minute hand and the twenty-fifth minute mark appears shifted such that one is unsure if the time is 12:25 or 12:26. Thus, even relative spatial alignment between two objects does not appear ego-independent.

Finally, ignoring examples that rely on absolute spatial location (either left or right of center, estimating the center based on distance from the edge, or detecting alignment based on distance between two objects), what about spatial proximity? **Figure 1D** shows two square keys that appear identical. One of the square keys is located next to a little round key. One can argue that no matter what egocentric perspective one assumes (no matter how the keys are rotated on the key chain), the square key in question will always be closer to the little round key than the other square key. Therefore, this should constitute an allocentric, ego-independent spatial representation. However, rather than involving spatial cognition, this example may rely on object recognition followed by rule-based reasoning: identify the little key first, then take the square key next to it (regardless of spatial distances or locations). Whereas egocentric spatial selectivity (however malleable) is already present before training, rules need learned. Alternatively, the square key and little key could be encoded holistically as a unit, with one feature activating the entire object configuration in object memory. For instance, in face perception, the spatial location of the nose could be represented relative to the spatial location of the eyes, or the face could be perceived holistically. Holistic object recognition relies on matching entire configurations of features to a stored template. This differs from representing individual features' spatial location relative to other features' spatial location in an allocentric spatial frame, because the spatial relationship between feature A and C should remain unchanged if other parts of the object (features B, D, E, for example) are removed. Logothetis (2000) has argued that not only faces, but even arbitrary objects are processed holistically, as a unit, with neurons responding to particular feature configurations rather than processing individual features.

The examples in **Figure 1** primarily pertain to reference frames for spatial judgments. However, it could be argued that the main purpose of allocentric spatial frames is navigation and orienting in the environment. Perhaps identifying locations as "north of " or "west of " another object would reveal true allocentric spatial cognition. After all, north remains north regardless of an animal's orientation or location.

However, even seemingly external, allocentric, coordinates such as north, south, west, and east may be re-centered on the ego's up, down, left and right coordinates. **Figure 2** (left) shows a right-side up map of Germany, with north pointing up. In this orientation, it is easy to figure out, for instance, that Moscow (Russia), located east/north-east relative to Germany, is somewhere slightly up and to the right of the image. However, when the map is rotated downward (**Figure 2**, right), it is much harder to guess where Moscow is, despite the fact that the cardinal directions are still indicated. Why are upside-down maps hard to read? Subjectively, it seems that we perform better when "north" is lined up with the egocentric "up," and when west and east correspond to the egocentric left and right, because we are then able to rely on our egocentric spatial reference frame to point relative to us. It is likely that most people mentally rotate the map upright to match their egocentric coordinates when making such spatial decisions, rather than relying on an abstract, allocentric map independent of our egocentric coordinates.

Multiple animal species may rely on magnetoreception to orient relative to cardinal directions (Eder et al., 2012; Wu and Dickman, 2012). Note that comparing the ego's heading to an external reference direction is not the same as allocentric heading in Klatzky's (1998) terminology, which would involve comparing the axis of orientation of an external object and the external reference direction (e.g., "north"). The magnetic field axis appears to be used as an external reference direction to which the egocentric axis is compared during navigation. In other words, the deviation of the ego's axis from an external axis, not the relationship between one object's axis and another external axis, is signaled. Thus, the question remains: does this magnetic sense allow animals to compute the location of one object relative to another object (e.g., object A is "north" of object B), independent of the animal's orientation, or does it signal "I'm still too far south" or "if I head this way, the destination is ahead?" The

FIGURE 2 | (Left) A "right side up" map of Germany, with the four cardinal directions (North, South, West, East) indicated. (Right) An upside-down (rotated) map of Germany, with correspondingly rotated cardinal directions. Pointing to Moscow (Russia) is easy with the left map, but harder with the rotated map on the right. Despite the cardinal directions being indicated, it is much harder to orient oneself in the map on the right. This is presumably due to the fact that we tend to mentally line up north, south, west and east with our egocentric coordinates: north is up, south is down, west is left, east is right. As soon as the familiar, egocentric arrangement is disturbed, it takes us longer to mentally rotate what is supposedly an abstract, viewer-independent, hence

allocentric map, back up to match our egocentric coordinates.

latter still entails referencing places in the environment relative

The process of aligning oneself with an external axis so that, e.g., north-selective cells receive the strongest stimulation,

can be represented relative to one's own body. This would be equivalent to assuming that "representation of any external point must be allocentric, because that point is, after all, external to the perceiving ego." Any external point can be represented relative to the ego in egocentric coordinates, thus an external object does not by default imply allocentric processing.

Rather, the question is: are external objects represented relative to other external object locations, independent of the egocentric perspective (whether actual or imagined/remembered)? Evidence for the latter would constitute a true allocentric representation. This is precisely the role hippocampal place cells have been proposed to play in navigation, discussed next.

# NAVIGATION: DO PLACE CELLS, GRID CELLS, AND HEAD DIRECTION CELLS FORM AN ALLOCENTRIC MAP OF SPACE?

Upon the discovery of place cells in the rat hippocampus (O'Keefe and Dostrovsky, 1971; O'Keefe and Nadel, 1978), it was suggested that place cells, together with head direction cells and grid cells, form an internal 'cognitive map' (Tolman, 1948) of the environment, representing allocentric space (for reviews, see McNaughton et al., 2006; Moser et al., 2008).

Hippocampal place cells fire at a particular location in the environment (the cell's place field), independent of the rat's orientation inside that place field (**Figure 3A**, top). "Grid cells," located in medial entorhinal cortex, display similar spatial tuning, except that each cell has multiple firing fields, effectively forming a periodic array or grid that tiles the environment (**Figure 3B**; Moser et al., 2008). Similarly, head direction cells (**Figure 3C**), present in multiple regions including the presubiculum and thalamus, indicate the direction the animal's head is facing, independent of the position or orientation of the animal in the environment (McNaughton et al., 2006). All these cells are anchored to (visual or other sensory) environmental cues (landmarks), and rotate or move their place fields or preferred head direction relative to such external distal cues, if the cues are rotated (Muller and Kubie, 1987; Moser et al., 2008). In other words, these cells appear to signal where the animal thinks it is located (or the direction it thinks it is facing). Place fields, grid fields, and head direction signals also persist in the dark, suggesting a reliance on self-motion (path integration) information for maintaining and updating such representations (Moser et al., 2008; e.g., by keeping track of how many steps the animal has taken, or vestibular, head turning signals).

Due to the independence of place and grid fields of the direction from which the animal enters a place or grid field, and hence of the animal's egocentric orientation, it has been suggested that these cells contribute to an allocentric map of the environment (Moser et al., 2008).

However, several pieces of evidence suggest alternative interpretations to an allocentric observer-independent map of space. Although a rat's orientation appears to have no influence on place cells in simple laboratory environments such as highwalled cylinders or open circular platforms, place fields are in fact spatially and directionally selective in environments that require the animal to plan a route between points of special significance, such as in radial mazes where food has been placed (Markus et al., 1995). In such cases, place cells respond at a particular location in the environment only if the animal traverses that location in a particular direction, but not in the other direction (**Figure 3A**). This contradicts an abstract map-like representation of the environment, since a place on the map should remain the same regardless of how it is traversed. By "abstract map" I mean a "cartographer-like

to the ego.

map" independent of the animal's orientation, goals, motivations, memory, or other factors unrelated to the spatial relationship between objects.

In addition, the size of a place field depends on the amount of incoming sensory information. In big brown bats, hippocampal place fields are small immediately after an echolocating call, but rapidly start to diffuse as time passes and echo information decreases (Ulanovsky and Moss, 2011). Moreover, the size of the place field depends on the exploratory mode of the animal: when the bat is scanning the environment from a fixed location using echolocation (akin to a primate saccading around from a fixed position), place fields are more diffuse, and place cells exhibit lower firing rates, than during locomotion through the environment (Ulanovsky and Moss, 2011). The fact that place cells respond differently to the same locations in the environment depending on the animal's behavior, and amount of sensory information received, seems to contradict an abstract map signaling fixed, allocentric, ego-independent relationships between places. After all, the relationship between a door and a window should not change depending on whether the ego is observing this relationship remotely or is passing by. Note that this is unlikely due to a difference in recall: the animal is scanning the landmark in question in both cases, i.e., the landmark has been activated in memory (recalled). What appears to differ is the egocentric relationship of the animal relative to the landmark.

Moreover, place fields are over-represented at motivationally salient locations, such as around a hidden platform in a water maze toward which rats are trained to swim (Hollup et al., 2001). This suggests a dependence of the spatial representation on the ego's behavioral goals, rather than a cartographer-like map of the environment.

Place cells are also re-activated during sleep, when the animal dreams about, imagines, or remembers being in a certain place (Pavlides and Winson, 1989). However, this is consistent with the idea that place cells signal the animal's current, remembered, or imagined position in the environment relative to some landmark.

Thus, although place cells might appear to encode a cognitive, map-like representation of an environment, place cells might not signal abstract spatial relationships between two places or two landmarks, independent of where the animal is located. Place cells may instead signal place recognition, e.g., "I'm by the door," regardless of whether I have my back to the door or am facing the door. If the door moves (without the animal noticing), a place cell's place field shifts to continue signaling "I am by the door," even though this is a new geocentric location. Such cells may not indicate "The door is by the window." In this sense, place cells might act more like object recognition cells than cells that represent spatial relationships between landmarks independent of the observer.

Similarly, while grid cells may map out a regular grid across an environment, with cells responding at fixed, regular intervals as the animal traverses it, the rigid grid-like structure would seem to preclude a flexible spatial representation of one object relative to another object, since no specific object-based relationship is signaled by such an arrangement. Both grid cells and place cells are driven by self-motion cues as the animal keeps track of its changing position (Moser et al., 2008).

Head direction cells signal the animal's heading relative to an external landmark. As described above, however, this signal may compare an egocentric (head) orientation with an external landmark, not the orientation of an external object to a reference landmark.

A recently discovered type of cells, entorhinal border cells, respond along the boundaries of an environment and may form a reference frame for place representations (Solstad et al., 2008). However, such cells do not fire at a distance from a wall or other boundary, but only along the boundary. This may suggest that rather than forming an abstract allocentric reference frame, they signal to the animal "I am near the wall." Thus, rather than signaling an abstract, allocentric environmental geometry, border cells may similarly represent the ego relative to some landmark, or conversely the landmark relative to the ego, not one landmark relative to another landmark.

Further support for the idea that hippocampal place cells are involved in place recognition in a process more akin to object recognition than spatial cognition comes from recent evidence that human place cells are reactivated during retrieval of objects associated with specific episodic memories (Miller et al., 2013). Participants navigated in a virtual environment, where they were presented with different objects at different locations. At the end of each trial, participants were asked to recall as many of the items as possible, in any order. The authors found that place cells' firing patterns during spontaneous recall of an item were similar to those during exploration of the environment where they had encountered the item. This suggests that recall of objects reactivates their spatial context, but also that place cells encode episodic memories more generally (Miller et al., 2013). Similar to rat place cells (Markus et al., 1995), the majority of human place cells were direction-dependent, only exhibiting place fields when traversed in a particular direction (Miller et al., 2013). This is consistent with an egocentric-dependent viewpoint in scene encoding and recognition, rather than an abstract, allocentric map implemented by place cells.

Finally, it is unknown whether place cells, grid cells, and other types of cells that have been studied in small-scale laboratory environments contribute to navigation in much larger, natural, environments, because it has been impossible to record from such cells in kilometer-sized environments (Geva-Sagiv et al., 2015). In most laboratory experiments, the entire spatial environment can be perceived with little or no movement, meaning that all information needed to calculate the spatial location of different landmarks is available from the animal's current location (Wolbers and Wiener, 2014). This means that in practice, the use of allocentric as opposed to egocentric information may be poorly controlled.

While the functional interpretation of place, grid, head direction, and boundary cells and their contribution to an allocentric map of the environment remains unclear, behavioral studies on animal navigation have also questioned whether animals make use of an allocentric, cognitive map during navigation.

Bennett (1996) has argued that a critical test of a "cognitive map" of space is the ability to take novel shortcuts, instead of following previously experienced routes. According to Bennett, previous evidence for shortcut-taking and putative cognitive maps in insects, birds, rodents, as well as human and nonhuman primates can be explained more simply either as path integration or recognition of familiar landmarks from a different angle, followed by movement toward them. The animal would thus only need to memorize routes and recognize landmarks to navigate toward them, rather than store a detailed cognitive map of spatial relationships between landmarks. The lack of shortcut-taking ability and hence absence of evidence for a cognitive map is supported by more recent research in a variety of species (Wehner et al., 2006; Grieves and Dudchenko, 2013). Instead, many species appear to rely on view-dependent place recognition, and to match learned viewpoints when approaching landmarks (Wang and Spelke, 2002).

However, when path integration and view-dependent place recognition fail, subjects do appear to be reorienting based on the geometry of a room or based on the "shape of the surface layout" (Wang and Spelke, 2002). Disoriented subjects search for target objects both at the correct corner and geometrically opposite corner of a room—but do not appear to be relying on the spatial configuration between objects (Wang and Spelke, 2002). In other words, not all allocentric information is represented; instead, simpler, geometric layout information is used, which perhaps functions more like object recognition.

In summary, it is unclear if place cells, grid cells, border cells, and head direction cells form the building blocks of an abstract, allocentric map of the environment for navigation, and to what extent these cells are involved in representing the spatial location of an external object relative to another object. Behavioral studies have questioned whether animals actually use an allocentric map for navigation, and whether whichever internal representation is used has the same characteristics as an abstract, cartographic map of the environment (Ekstrom et al., 2014).

# BEHAVIORAL STUDIES: SPATIAL REFERENCE FRAMES FOR GOAL-DIRECTED MOVEMENT

Several behavioral studies have investigated which spatial reference frames are used in goal-directed actions, such as (delayed or immediate) pointing, reaching, grasping, or saccades to (visual or remembered) targets (for reviews, see Battaglia-Mayer et al., 2003; Crawford et al., 2011). Many such studies have investigated spatial reference frames in the context of spatial updating (Colby, 1998; Crawford et al., 2011), where a spatial target is briefly presented, followed by a change in gaze direction before the reach (or saccade) to the remembered location of the target. Saccade or reach endpoint errors and other metrics can then be investigated in the context of landmarks being present vs. absent at the moment of target presentation (**Figure 4**).

Substantial evidence exists for gaze-centered (egocentric) updating of reach targets following an intervening saccade, for both immediate and delayed movements (Henriques et al., 1998; Medendorp and Crawford, 2002; Thompson and Henriques, 2008; Rogers et al., 2009; Selen and Medendorp, 2011). These studies suggest that the spatial location of a visual target is maintained in an eye-centered reference frame (i.e., as the retinal distance between the current gaze direction or fixation point, and the remembered target location), and is updated across eye movements. While some evidence suggests that gaze-centered updating persists even after long delays (Fiehler et al., 2011), others have suggested that allocentric spatial representations are used when movements are delayed (Westwood and Goodale, 2003).

Several studies have demonstrated more accurate reaching in the presence of landmarks following gaze shifts (e.g., Byrne et al., 2010), and that integration of egocentric and allocentric or landmark information may depend on the stability of

visual cues; i.e., the weight assigned to landmarks depends on whether the landmark is moving around (Byrne and Crawford, 2010). Note that the presence of a landmark should not automatically be assumed to involve allocentric (object-centered) reference frames. Both the landmark and the target could be represented relative to the ego. However, can behavioral differences between memory-guided reaches with and without landmarks be explained without relying on the assumption that an allocentric spatial reference frame is used? What accounts for the observed behavioral effects? I will describe two representative experiments in detail to illustrate how egocentrically-encoded landmarks could contribute to such differences.

not know which of the two knobs (left and right squares) will be removed. Adapted from Chafee et al. (2005) (see text), with permission.

In a study by Schütz et al. (2013), subjects reached to remembered target locations after intervening saccades, either in the presence or absence of visual landmarks. Subjects foveated a briefly displayed target, and continued fixating its location after its disappearance. After a delay of 0, 8, or 12 s subjects then saccaded to a new fixation cross which appeared at various visual eccentricities. Following the gaze shift, the fixation cross also disappeared and subjects reached to the remembered target location in complete darkness. In the allocentric condition, two light tubes were present left and right of the screen, respectively. Pointing errors varied systematically with gaze shift, e.g., when fixating to the left, subjects overshot the remembered target location in the opposite direction, in both the visual landmark and the no-landmark condition. This is consistent with previous evidence that reaching is carried out in eye-centered (hence, egocentric) coordinates (Henriques et al., 1998). Moreover, the different delays led to similar reach endpoint errors, i.e., the effect of the (egocentric) gaze shift remained the same regardless of a delay or not. This suggests that both immediate and delayed reaches rely on gaze-dependent (egocentric) spatial representations.

In addition to varying with gaze shift (an egocentric influence), however, endpoint errors were reduced in the landmark condition. One possible interpretation of this landmark influence is that egocentric and allocentric spatial representations are combined (Schütz et al., 2013). While it is possible that reach targets are represented relative to both landmarks and gaze position, an entirely egocentric explanation cannot be ruled out. For instance, both the initial target and the landmarks could be represented in gaze-centered coordinates. In the no-landmark condition, the target disappears before the fixation cross reappears at a novel location, with the subject sitting in complete darkness during the variable delay. When the novel fixation cross appears, the egocentric estimate of how far the eyes have moved relative to the remembered target (the retinal distance) is less precise. Even in the 0 s delay condition, the target still disappears before the new fixation cross appears, i.e., the new fixation location and the target are never simultaneously displayed, which may lead to a less precise calculation of the saccade vector from (former) target location to (novel) fixation cross location. Previous research (Chen et al., 2011) has shown egocentric information decays gradually, with decay commencing as soon as the target disappears (Westwood and Goodale, 2003). In the absence of external visual landmarks, these factors could thus contribute to a less accurate estimation of how far the eyes have moved away from the initial target location, or greater uncertainty regarding gaze position relative to the former target location (in retinal coordinates), when the reach is initiated. In contrast, in the allocentric landmark condition the landmarks are present throughout the trial, which can lead to a more accurate retinal (egocentric) estimate of how far the eyes have moved. Subjects can represent both the target and the landmarks relative to their gaze when initially viewing the target, and update this eye-centered representation after the saccade. For instance, the left landmark may be at −10◦ of visual angle relative to the target in the beginning, and at −5 ◦ after the saccade to the new fixation cross, when the reach target has disappeared. The gaze shift vector (in eye-centered coordinates) will thus be estimated more accurately, and can be subtracted from the previous eye-centered position of the hand, to more accurately lead the hand to the remembered target position (in eye-centered coordinates, e.g., Medendorp and Crawford, 2002).

Thus, although the combination of allocentric and egocentric cues remains a possibility, the reduced endpoint reach error in the landmark condition could be explained in terms of less accurate egocentric updating. This explanation is more parsimonious, as it involves a single (egocentric) spatial reference frame. To tease apart these competing accounts, the egocentric account makes a testable prediction: if the new fixation cross were to appear before the target is extinguished, there should be reduced uncertainty regarding how far the eyes have moved, even in the absence of landmarks, and hence reduced endpoint errors, similar to the landmark condition. Future experiments could address this prediction. A second prediction could be tested to tease apart allocentric vs. egocentric influences: the two light tubes (landmarks) could be briefly turned off at the same time as the target, during the saccade to a new fixation cross. The landmarks could reappear just before or at the time of the reach. The prediction is that a disruption in egocentric updating of how far the eyes have moved will lead to greater reach error, even when the landmarks reappear later. This would support an egocentric explanation of the landmark effect.

In another study, Chen et al. (2011) compared the rate of memory decay for egocentric and allocentric reach targets, using delayed reaching to remembered target locations following intervening saccades.

In the egocentric condition, a target appeared in the periphery relative to the fixation cross. After the target disappeared, subjects shifted their gaze to a new fixation location. Following a variable (short, medium, or long) delay, the fixation cross disappeared, and subjects reached to the remembered (and egocentrically remapped) location of the reach target.

In the allocentric condition (**Figure 4A**), the target was surrounded by four landmarks. These landmarks reappeared at a different location following the short, medium or long delay after the gaze shift, and subjects reached to the remembered (and remapped) target location, relative to the landmarks.

In a similar third condition, the allo-to-ego conversion condition, the four landmarks reappeared at the new location both before and after the variable delay.

The authors found that in the egocentric and allo-toego conversion condition, reaching variance (endpoint error, reduced precision) increased from short to medium delays, whereas reaching variance remained constant across delays in the allocentric condition. Similarly, reaction times in the egocentric and allo-to-ego conditions were longer at short delays compared to longer delays, whereas reaction times did not vary according to delay in the allocentric condition. The authors concluded that egocentric representations of target locations decay faster than allocentric representations. It was also suggested that allocentric information is converted to an egocentric representation at the first possible opportunity (Chen et al., 2011). Thus, the allocentric landmarks appearing both before and after the delay in the allo-to-ego condition could be used to infer the location of the target in egocentric coordinates before the delay (an alloto-ego conversion at the first opportunity), and this egocentric information decays with increasing delays. This interpretation could explain the increase in endpoint errors across delays in the egocentric and the allo-to-ego conditions, and the absence of a modulation by delay in the allocentric condition (when landmarks only appear after delays).

Can these behavioral differences between egocentric and allocentric conditions be explained using a purely egocentric reference frame? It is possible that both the target and the surrounding landmarks were represented in egocentric coordinates, and were mentally shifted to center on the fovea (i.e., the center of mass of the square in **Figure 4A** would line up with the fixation point). As such, a target closer to e.g., the bottom left landmark would also be in the egocentric lower left relative to the fovea. When the landmarks reappeared at a new egocentric location, the new target location could be remapped in egocentric coordinates based on shifting the entire structure (landmarks plus retinocentrically remapped/remembered target) to the new retinal location. Alternatively, even without mentally shifting the landmarks to imagine them around the fixation point, retinal distance vectors can be computed from the fixation point to both the landmark nearest the target ("vector x") and to the target ("vector y"). The difference between vectors x and y can be stored as a retinal vector ("z"). When the landmark reappears at a different location in the visual field, the retinal vector to its (egocentric) coordinates is calculated, and the difference vector z can be added to infer the new target location in egocentric, rather than allocentric, coordinates. Egocentric remapping of targets has been demonstrated in multiple brain regions (Colby, 1998).

Why then were there differences between egocentric and allocentric reach accuracies and reaction times? Unlike in the egocentric condition, the allocentric landmarks reappear after the delay, just before movement onset, thereby facilitating remapping of the remembered target in egocentric coordinates just before movement onset. Since the landmarks are displayed just before movement onset in each of the three delay conditions, with the delays preceding, not following, the reappearance of the landmarks at the new location, the (egocentrically) remapped location does not get a chance to decay before movement onset. This could explain the shorter and constant reaction times in the landmark condition compared to the egocentric condition. In contrast, in the egocentric condition no new cues are presented after the intervening saccade and variable delay. The longer the delay, the greater the egocentric information decay, consistent with the authors' interpretation (Chen et al., 2011).

What about the allo-to-ego condition, which resembled the egocentric condition in terms of an increase in reach errors across delays? In the allo-to-ego condition, the amount of time the landmarks are displayed at the new location is halved: instead of reappearing for 1.5 s after the delay, they appear for only 0.75 s before and 0.75 s after the variable delay. This shorter presentation time may have led subjects to rely on the first reappearance of the landmarks to update both landmarks and the target in egocentric coordinates, as suggested by the authors. Since the variable delay follows the first reappearance of landmarks, egocentric information decays just like in the no-landmark, egocentric, condition.

In summary, although it is possible that a fundamental difference exists between egocentric spatial representations, thought to decay rapidly across delays, and allocentric spatial representations, which are thought to be more stable and decay less rapidly, these results are equally compatible with an egocentric remapping of all targets, whether surrounded by landmarks or not, accompanied by an egocentric decay in all cases where the remapped information precedes a variable delay. This and similar studies therefore do not necessarily demonstrate the existence of allocentric spatial representations.

Behavioral studies have also investigated visual illusions such as the Müller–Lyer illusion, in which a line segment is flanked by either pointed arrow heads or arrow tails. Subjects perceive identical-length segments with arrow tails as longer than those with arrow heads, which could be interpreted as evidence of allocentric encoding of object features relative to each other. However, Howe and Purves (2005) have shown that this illusion can be explained by natural image statistics where the physical sources giving rise to a 2D retinal image of a line segment with arrow heads tend to belong to the same plane (object, or surface area), whereas physical sources for arrow tails are less likely to come from the same plane. The illusion could thus arise from a probabilistic interpretation of 2D retinal projections of the real world—and would not require allocentric spatial encoding of individual features. A review of 33 studies of pointing to Müller– Lyer stimuli showed that visually-guided pointing (rather than from memory) is typically not subject to the Müller–Lyer illusion, suggesting that this illusion is mediated by the ventral rather than dorsal visual stream (Bruno et al., 2008).

Other studies have investigated pointing accuracy to surrounding objects after subjects were disoriented through self-rotation, with objects hidden from view (Wang and Spelke, 2000, 2002). In such experiments subjects show increased configuration pointing errors, i.e., a deterioration in the internal representation of the angular relationship between targets (e.g., where the TV is relative to the table). This has been interpreted as a disruption to dynamic egocentric updating of target locations (relative to the current ego location), even after controlling for vestibular stimulation, re-orientation via an external light, and other factors, contradicting an enduring cognitive map of allocentric spatial relations between objects independent of the observer (Wang and Spelke, 2000).

Conversely, other studies have shown that disorientation leads to much lower error in "judgments of relative direction" (JRD tasks), where, rather than pointing from the current ego location to objects' locations, subjects imagine themselves by an object and point to another object from that imagined location (e.g., imagining the ego by the door and pointing toward the TV from that location; Burgess, 2006; Waller and Hodgson, 2006; Ekstrom et al., 2014).

However, it is unclear whether higher performance in the JRD task necessarily means subjects rely on stored allocentric representations of object locations relative to each other. The JRD task may simply involve accessing stored egocentric viewpoints, mentally rotating (shifting) the ego to one of the objects, and making an egocentric decision as to where objects are—relative to the ego. In Waller and Hodgson's study (Waller and Hodgson, 2006), for example, participants walked past each of the objects to be encoded, thereby presumably obtaining multiple egocentric viewpoints on the scene layout. Disorientation does not affect JRDs compared to pointing from the current ego orientation, because JRDs rely on stored egocentric viewpoints, whereas orientation-dependent pointing requires re-establishing egorelative object locations anew. Behavioral differences or effects between two experimental conditions thus do not necessarily demonstrate that allocentric vs. egocentric spatial reference frames are used. The two tasks can be viewed as different egocentric tasks, with differences due different egocentric mechanisms being activated (mental rotation of the ego and recall of egocentric viewpoints vs. remapping current target locations relative to the ego following disorientation). Such mental rotations are supported by evidence that recognition times of arrays of objects displayed on a circular table, when rotated to various degrees, increase linearly with the angle of rotation away from the original display (Wang and Spelke, 2002).

If allocentric tasks can be solved by mentally rotating or shifting either the ego or a display of landmarks back to an egocentric (perhaps retinal) center, what if only subsets of objects are shifted in a scene—could reach errors reveal whether subjects encode targets relative to objects rather than the ego? Fiehler et al. (2014) found that the greater the number of objects shifted, the greater the deviation of reach endpoints in the direction of object shifts. While this suggests a plausible allocentric mechanism whereby target locations are encoded relative to other objects, rather than relative to the ego, this could depend on whether an egocentric reference point is provided during encoding of object (target) locations. If a retinal reference point is missing (no fixation cross provided during encoding), subjects may not notice shifts in clusters of objects and still rely on view-dependent (partial) scene recognition, with reaching performed relative to a presumed egocentric reference point that could not be accurately established during encoding. Shifting single large or single smaller local objects had no effect on reach endpoint errors (Fiehler et al., 2014). Similar view-dependent local scene encoding or retinal visual distance calculations can account for other studies in which combined egocentric and allocentric influences were examined (Byrne and Henriques, 2012; Camors et al., 2015).

# NEUROPSYCHOLOGY: OBJECT-CENTERED SPATIAL NEGLECT?

A number of neuropsychological studies of hemineglect patients have identified seemingly dissociable egocentric vs. objectcentered (or allocentric) neglect symptoms, as well as dissociable brain damage sites (for reviews and critiques, see Olson, 2003; Rorden et al., 2012; Yue et al., 2012; Humphreys et al., 2013; Li et al., 2014).

Following damage to (predominantly) the right hemisphere, patients exhibit unawareness of the contralateral (egocentric left) side of space (Humphreys et al., 2013). In addition to egocentrically-defined hemineglect, some patients ignore the left half of an object or of objects, even if presented in their intact (egocentrically right) hemifield, or even if rotated such that the left half of the object falls on the (intact) right visual field (e.g., Caramazza and Hillis, 1990; Driver and Halligan, 1991; Behrmann and Moscovitch, 1994; Behrmann and Tipper, 1994; for review, see Humphreys et al., 2013). The fact that the left half of an object is neglected even when rotated and presented in the egocentric right half of space has been interpreted as evidence for object-centered spatial representations.

However, alternative explanations have been proposed for this pattern of object-based hemineglect. For instance, rotated objects presented in non-canonical orientations may be mentally rotated back upright to match an egocentric, canonical (mental) representation of the object, the left half of which is then ignored (Buxbaum et al., 1996; Humphreys et al., 2013).

Similarly, computational models suggest that a decreasing attentional gradient from (the egocentric) right to left could lead to the left half of any item anywhere in the visual field being less salient and therefore more likely to be ignored (Driver and Pouget, 2000; Pouget and Sejnowski, 2001). Models relying on such "relative egocentric neglect" (Driver and Pouget, 2000; Pouget and Sejnowski, 2001) have successfully modeled what appears to be object-centered neglect (Pouget and Sejnowski, 1997, 2001; Mozer, 1999, 2002).

Such a lesion-induced (egocentric) gradient of salience, which could affect either the stored representation of an object or the allocation of attention to this representation, is supported by evidence that the severity of allocentric neglect is modulated by egocentric position, with milder allocentric deficits at more ipsilesional egocentric positions (Niemeier and Karnath, 2002; Karnath et al., 2011). The field of view across which such a gradient in salience is exhibited may be flexibly adjusted (similar to a zoom lens; Niemeier and Karnath, 2002; Karnath et al., 2011; Rorden et al., 2012). For instance, exploratory eye movement patterns in neglect patients did not differ between egocentric and allocentric neglect, but rather differed according to the task goal and strategies, with the same item either detected or neglected depending on the task (Karnath and Niemeier, 2002).

However, double dissociations between egocentric and allocentric neglect have been reported, together with apparent double-dissociations in lesion sites (Humphreys and Heinke, 1998; Humphreys et al., 2013). Egocentric neglect tends to be associated with more anterior sites in supramarginal gyrus and superior temporal cortex, whereas allocentric neglect tends to correlate with more posterior injuries such as to the angular gyrus (Medina et al., 2009; Chechlacz et al., 2010; Verdon et al., 2010).

In contrast, several recent studies have reported that allocentric neglect co-occurs with egocentric neglect, and that the lesion sites overlap (Rorden et al., 2012; Yue et al., 2012; Li et al., 2014). Rorden et al. argue that previous studies have used vague or categorical criteria in classifying patients with allocentric vs. egocentric neglect, leading to an apparent double dissociation between deficits. (For instance, a patient with both egocentric and allocentric deficits would be categorized as allocentric-only, leading to an apparent double-dissociation). To identify whether egocentric and object-centered neglect are dissociable, Rorden et al. used a "defect detection" task in which right-hemisphere stroke patients had to separately circle intact circles and triangles as well as circles and triangles with a "defect" (e.g., a gap in the left half of a circle). Unlike previous studies, which had coded allocentric and egocentric neglect in a categorical, dichotomous manner, thereby ignoring the varying severity of deficits, Rorden et al. used a continuous measure. Allocentric neglect scores were calculated based on the number of correctly detected items with defects as well as intact items correctly marked, on both the contralesional and ipsilesional side. In addition, they also used a center of cancelation task to calculate egocentric neglect scores based on how many targets (e.g., the letter A) were identified in a cluttered field of letters, weighted according to their position from left to right.

Confirming previous findings by Yue et al. (2012), Rorden et al. (2012) found that allocentric deficits were always observed in conjunction with egocentric deficits, with no pure cases of allocentric neglect. In contrast, egocentric neglect did occur on its own. The allocentric neglect score was strongly correlated with patients' egocentric neglect score, and substantial allocentric neglect was only present with substantial egocentric neglect, suggesting that allocentric neglect is a function of severe egocentric neglect.

Moreover, the regions of brain damage associated with egocentric and allocentric neglect strongly overlapped. Rorden et al. (2012) suggest that previous findings of an association between posterior temporo-parietal lesions with allocentric neglect, and superior and middle temporal lesions with egocentric neglect, may in fact result from the same mechanism, namely the extent to which the middle cerebral artery territory is affected by stroke. According to this account, allocentric deficits may be subclinical in milder forms of neglect, which are associated with damage restricted to the central aspect of the middle cerebral artery territory, thus producing what appears to be purely egocentric neglect. In contrast, more severe forms of neglect, comprising both egocentric and allocentric deficits, are due to damage to a larger extent of middle cerebral artery territory, including more posterior regions typically associated with allocentric neglect.

The fact that patterns of object-centered neglect can be explained in terms of an egocentric gradient in salience, as well as recent evidence of a lack of double-dissociation between egocentric and allocentric neglect symptoms and lesion sites, argue against independent egocentric and allocentric spatial representations, and support a single (egocentric) mechanism.

Other neuropsychological investigations have focused on lesions to the ventral visual stream. For instance, patient D.F. shows impairment in (conscious) visual shape perception, but accurate visuomotor performance (such as correct grip aperture) in actions directed to different object shapes (Goodale and Milner, 1992; Goodale and Humphrey, 1998). This has been interpreted as evidence for separate vision for perception (ventral) and vision for action (dorsal) streams (Goodale and Milner, 1992). Schenk (2006) has questioned whether D.F.'s impairment is perceptual, rather than allocentric. In Schenk (2006), D.F. was impaired on a visuomotor task that involved proprioceptively-guided pointing to the right or left of the current hand position by a similar amount as displayed visually between a visual cross and visual target. As suggested by Milner and Goodale (2008), however, the impairment could have been due to the task requiring D.F. to make a perceptual judgment (visual estimate) of the distance between the visual stimuli, before being able to translate that visual distance into a visuomotor plan to a different location. Moreover, this estimate could happen via a "perspectival" (egocentric viewpoint-dependent) mechanism (Foley et al., 2015), rather than an allocentric mechanism. The latter interpretation would thus suggest the ventral visual stream is involved in perceptual (e.g., visual size) estimates underlying shape perception, not necessarily allocentric spatial cognition.

Foley et al. (2015) have argued that whereas the dorsal visual stream uses effector-based egocentric spatial representations, the ventral visual stream may use a perspectival egocentric representation of scenes or objects. Note that this perspectival account is compatible with holistic configural scene or object processing (Logothetis, 2000). Moreover, according to Foley et al., the purpose of ventral visual stream computations is object recognition, attaching emotional or reward value to such a representation, or habitual learning (i.e., what to do with such an object, regardless of the current egocentric perspective on it).

These proposed processes are consistent with the findings presented in the present review, and are compatible with an egocentric account of spatial processing.

# NEUROPHYSIOLOGY: EVIDENCE FOR OBJECT-BASED SPATIAL REPRESENTATIONS, OR RULE-BASED DECISIONS?

Neurophysiological studies have shown that multiple egocentric (e.g., hand-centered and eye-centered) representations of the same target can co-exist in parallel or change fluidly during sensorimotor transformations (Battaglia-Mayer et al., 2003). In fact, many neurons exhibit hybrid (e.g., both eye and handcentered) reference frames (Avillac et al., 2005; Mullette-Gillman et al., 2009). Here I examine whethersingle-unit neurophysiology evidence supports the representation of an allocentric reference frame at the neuronal level at any point in the sensorimotor transformation. A number of single-unit recording studies have reported object-centered spatial representations in both prefrontal and posterior parietal cortex. In a series of studies, Olson and colleagues (Olson and Gettner, 1995, 1999; Olson and Tremblay, 2000; Tremblay et al., 2002; for review, see Olson, 2003) reported object-centered spatial selectivity in macaque supplementary eye field (SEF) neurons during saccade planning. A typical task (**Figure 4B**) involves first presenting a horizontal bar with a cue left or right on the bar, at various retinal locations, while the monkey is fixating centrally. Following a variableduration delay, the horizontal bar is presented at another location in the visual field. After a second variable-length delay, the fixation point disappears and the monkey executes a saccade to the remembered target location relative to the object, i.e., left or right on the bar, regardless of whether the bar is now in the left or right visual field. Interestingly, many SEF neurons show differential activity during the post-cue delay prior to object-left vs. object-right saccades, even though the monkey does not yet know the direction of the physical saccade. In other words, while the monkey is holding the object-centered location in working memory, after the bar and cue disappear, but before the new horizontal bar appears, SEF cells selectively signal object-right vs. object-left locations, suggesting object-centered spatial selectivity. This effect is also obtained if color cues or discontinuous objects/cues (e.g., left vs. right of two dots) are used to instruct left vs. right saccades relative to the object (Olson, 2003).

While these results are consistent with object-centered spatial representations in SEF, several additional findings allow for an alternative interpretation. For instance, the neurons that prefer the bar-right condition are predominantly in the left hemisphere, while bar-left neurons predominate in the right hemisphere (Olson, 2003), consistent with an egocentric contralateral representation of each half of space. The fact that neurons selective for object coordinates are arranged according to egocentric space in the brain could suggest a recentering of the mental representation of the object during the delay, such that the left half of the object falls in the (egocentrically) left visual field and the right half of the object in the (egocentrically) right visual field.

The idea that a (re-)centered mental representation is driving these responses is also supported by other characteristics of SEF neurons' responses: the object-centered spatial selectivity emerges during the post-cue delay, even when the new target bar isn't visible yet, i.e., before a new object-relative target position can be calculated (e.g., **Figures 1**, **4**, in Olson, 2003).

Interestingly, color cues take longer (200 ms) than spatial configuration cues to evoke object-centered activity, suggesting a top-down, rule-based decision process, perhaps coming from other prefrontal regions such as dorsolateral prefrontal cortex (DLPFC). SEF neurons can also learn to respond to color instructions even if the color cue that signals an object-left rule appears at the right of the object (dot array; Olson, 2003). In such cases, the neuron indicates both the object-relative location of the cue (i.e., if the neuron prefers left on the object, yet the cue signaling a future left-object saccade appears on the right, the neuron responds weakly to the cue) and the objectrelative location of the target (i.e., if the target then appears on the left in a left-object preferring neuron, a strong response is obtained; Olson, 2003). This pattern has been interpreted as object-centered spatial selectivity, and that the target could not be selected by an object-centered rule (since the cue appeared on the right of the object, and yet instructed a left response; Olson, 2003). However, this response pattern occurred in SEF neurons previously trained to select targets using precisely an objectcentered rule. Importantly, as discussed by Olson (2003), SEF neurons only show weak object-centered signals before training.

This training-dependence suggests that rather than responding to object-based spatial locations in a bottom-up manner (via object-centered spatial selectivity), such putatively object-centered neurons require extensive training, i.e., respond most likely to top-down signals. This could suggest rule-based decision making signals from other (perhaps dorsolateral prefrontal) regions, rather than spatial perception in an objectcentered spatial reference frame. A testable prediction is that DLPFC activity should precede SEF activity on such tasks. In humans, a testable fMRI prediction would be that the effective connectivity between e.g., DLPFC and SEF should increase when the rule needs applied.

This interpretation of a superimposition of a rule onto SEF neuronal activity is also consistent with the fact that SEF neurons showed a modulation by egocentric saccade directions, i.e., a right-object selective SEF neuron still showed some preference for physically (egocentric) rightward saccades even if they fell on the (non-preferred) left end of the object (Olson, 2003).

Other studies have investigated object-centered representations in posterior parietal areas (for review, see Chafee and Crowe, 2012). Crowe et al. (2004) recorded from inferior parietal area 7a while monkeys were shown visual stimuli depicting octogonal mazes (viewed from the top), with a straight main path extending from the center box out (**Figure 4C**). In exit mazes, the main path exited to the perimeter, whereas in no-exit mazes, the main path ended in a dead end inside the maze. Monkeys mentally solved mazes to determine whether each maze had an exit path or not, without moving their eyes from the fixation point located at the center of each maze. While mentally solving the maze task, one quarter of neurons in parietal area 7a exhibited spatial tuning for maze path directions.

Interestingly, and consistent with the top-down hypothesis of object-centered processing, neuronal tuning for maze path direction only emerged after training (Crowe et al., 2004). In other words, naive animals that viewed the same maze stimuli without solving them did not show tuning to path direction. This argues against an existing, object-centered spatial representation, i.e., an "allocentric lens" through which spatial relationships in the world are viewed. If object-based spatial relationships did exist, these neurons should have represented them in a "bottom-up" manner just like retinocentric or egocentric spatial relationships are represented, which do not require task training. A neuron that has a preference for a certain object-centered spatial relationship (e.g., maze path exiting to the right of the maze) should exhibit such an object-centered preference whenever the monkey is looking at such a stimulus. It is possible that allocentric spatial tuning takes longer to develop with more complex visual stimuli, where multiple object-centered spatial relationships could be represented. Such training dependence, however, is also observed for simple bar stimuli, as reported by Olson (2003).

As in SEF, object-centered parietal area 7a neurons had a preference for contralateral path directions. In other words, neurons located in the left hemisphere preferred maze exits to the egocentric right. However, preferred maze path directions (e.g., up and to the right) were largely independent of receptive field (RF) locations as mapped with spot stimuli (Crowe et al., 2004). Spatial tuning for path direction in the maze task was also not systematically related to saccade direction tuning as mapped in an oculomotor control task. While this dissociation between the RFs mapped using control tasks and maze path direction would seem to suggest an independence of egocentric variables, it is also possible that individual neurons' RFs obtained with the visually more complex maze object shift dynamically with more complex tasks. The fact that the maze task needs solved mentally (without moving the eyes) would suggest that some mental remapping of information across receptive fields is necessary. I.e., neurons might dynamically and predictively represent the information expected to fall in their RFs if the eyes were moved. Thus, the classically defined RF location as mapped by spot light stimuli would seem less relevant than finding out what kind of remapping might be happening during mental solving of the maze task. Remapping of information even prior to saccades has been demonstrated in neighboring area LIP (Colby, 1998).

In fact, a subsequent study of the maze task (Crowe et al., 2005) studied the neuronal population dynamics during maze task solving. Crowe et al. (2005) found that following presentation of the maze, the population vector (the direction signaled by the majority of cells) in parietal area 7a began to grow in the direction of the exit path. In trials in which maze paths had a right-angle turn, the population vector rotated in the direction of the turn, however, not 90◦ , but 45◦ .

In other words, imagine a triangle corner centered at the fovea, with one triangle side extending vertically up from the fixation point; from the top of the vertical side, another side extends to the right, forming a right angle with the vertical line. If you were to move your eyes up one side of the triangle and then turn 90◦ right, the hypotenuse is 45◦ relative to the vertical meridian from your initial fixation point. In object coordinates, the configuration of the path toward the exit is first up, then 90◦ to the right. However, the populations of cells that became active were first cells preferring up, then cells preferring 45◦ to the right.

This is the vector angle one would expect if the vector origin were anchored to the fovea (initial fixation point), with the tip of the vector signaling the maze exit from the foveal origin to 45◦ up and to the right, as suggested by the authors (Crowe et al., 2005; Chafee and Crowe, 2012). This suggests the maze problem was solved from an egocentric, specifically retinocentric, perspective, and is less consistent with an object-centered representation, at least at the population level.

Another approach to studying object-centered spatial representations is to use a visual "object construction task" (**Figures 4D,E**), in which presentation of a model object consisting of a configuration of elements is followed by a test object in which one element is missing (Chafee et al., 2005, 2007). For instance, an inverted T-like structure consisting of Tetris-like blocks arranged vertically and horizontally was followed by a test structure where one block was missing left or right of the vertical object axis. Monkeys were trained to then "complete" the test object by choosing between two elements, one of which was on the correct side of the missing element location. Once the element was chosen, it was attached to the test object at the appropriate location.

By presenting either the test object or the model object at different retinal locations, Chafee et al. (2005, 2007) could investigate whether neurons in area 7a are sensitive to the object-referenced location of the missing element (e.g., top right of the object) regardless of the egocentric (retinal) location of the element. Chafee and colleagues found two populations of neurons in area 7a. One population coded the missing element in viewer-referenced (egocentric) coordinates, whereas a partially overlapping population encoded the missing element in object-referenced coordinates, signaling the missing piece both when the test object appeared left and right of the fixation cross. Object-centered neurons showed object-centered responses both when the whole shape (model) was presented, and when the test object (with a missing piece) was shown.

Several neurons indicated a joint viewer- and objectreferenced influence, responding more strongly when both the element and the object were on the preferred side (for instance, both on the egocentric left and object-referenced left).

As in Olson (2003), this task (Chafee et al., 2005, 2007) allows either the model or the test object to be mentally translated to the ego-center (the fovea), where left or right on the object becomes a simple egocentric decision. The putative object-referenced population could thus be remapping locations in an ego-relative way. Consistent with the ego-relative interpretation, and similar to Olson (2003) and Crowe et al. (2004, 2005), object-referenced neurons preferred contralateral "missing elements" (relative to the object).

Moreover, the data suggest that this process is rule-dependent. In some trials, model objects contained two elements, one left and one right on the object, either of which could be removed in the test stage (**Figure 4E**). During the delay between model and test object presentation, the monkey could thus not know which of the elements would be removed for such ambiguous model objects. Interestingly, in contrast to trials where the element that would be removed was obvious during the model stage, there were no object-centered responses during presentation of the indeterminate model object, with object-centered responses only emerging after the test object revealed which element was missing.

Why would an object-centered neuron not signal "left on the object" regardless of which element (left or right) would end up being removed? It could be argued that if these neurons were encoding spatial locations relative to objects, then neurons selective for "left of the object" should have signaled the objectrelative location of the element during the indeterminate model presentation as well as during test object (missing-piece-object) presentation. The fact that neurons "waited" until the missing element was revealed during the test object phase suggests that such neurons might encode rules, not spatial relations: at the moment of the ambiguous model, neurons could not yet apply any rule, since either of the two elements could be removed; the rule to be applied only emerged in the test stage. This suggests that these neurons do not have a true object-centered spatial preference such as "left on the object." Rather, they encode the rule "detect if a certain ego-relative element is missing." Thus, a coding of relative retinocentric position, rather than objectcentered spatial reference frames, cannot be ruled out.

If neurons in inferior parietal area 7a are involved in mentally re-centering a peripherally-displayed visual stimulus such that it lines up with the fovea or ego center, one would expect objectcentered responses that signal "object left" or "object right" regardless of retinal position to be somewhat delayed compared to simple egocentric responses. In fact, this is exactly what was found by Crowe et al. (2008). Information in retina-centered coordinates emerged first, and was followed by neural signals coding object-relative positions. The strength of egocentric and object-centered signals was correlated, and object-centered responses could be predicted from retina-centered responses, but not vice versa (Crowe et al., 2008). Thus, each location on an object is presumably first represented retinocentrically, e.g., for an object in the left visual field, the left edge of the object is represented as "further left" than the right edge of the object, which is represented as "left but closer to the midline." These retinocentric coordinates are subsequently transformed into "object-left" and "object-right." This is consistent with a mental shifting of the object to the ego-center, at which point the remapped (mentally shifted) "left" and "right" in object-based coordinates match the egocentric left and right.

Note that this suggests that allocentric reference frames are derived from egocentric reference frames, and are thus not at the same level in the computational hierarchy, i.e., objectbased locations are not a primitive parameter in allocentric spatial processing in the same way egocentric locations constitute a primitive in egocentric spatial processing (Klatzky, 1998). This suggests that regardless of what level (which layers, or projections between layers in a multi-layer network) egocentric and allocentric computations take place at, the egocentric coordinates need computed first before being fed into a network that can construct object-based representations. The feasibility of transforming egocentric representations into objectdirected responses using a basis function network that lacks explicit object-centered representations and whose neurons have retinotopic response fields, has been demonstrated by Deneve and Pouget (2003). Object-referenced actions emerge as mappings between the relative and absolute retinal locations of an object and particular motor commands—at no point in the network do such cells, or does the network, create an explicit object-centered spatial representation.

Even if a bottom-up transformation of egocentric to objectcentered coordinates is possible, the rule-like behavior of some of these parietal neurons, and extensive training required to exhibit object-centered responses, however, also suggest a top-down modulatory signal. As with area SEF, this leads to the prediction of an earlier prefrontal than posterior parietal response. This prediction has in fact been tested. A recent study using simultaneous recordings in macaque prefrontal and posterior parietal cortex showed that rule-based spatial categorization signals are stronger and emerge earlier in dorsolateral prefrontal cortex than in area 7a (Goodwin et al., 2012). Monkeys were trained to categorize dots as either "left" or "right," or "above," or "below" a boundary in response to a rule cue. Thus, the same dot location could be classified as left or right, or as above or below, depending on the rule cue. Both parietal and prefrontal neurons represented spatial categories according to the rule, but with earlier and stronger rule-dependent modulation of category signals in prefrontal cortex, suggesting executive control over spatial processing.

This suggests the possibility that a number of object-relative responses found in area 7a and SEF are likely rule-dependent spatial responses, rather than spatial perception or representations of spatial relations between objects in a bottom-up manner, and explains why these object-relative responses (which likely depend on mental transformations of ego-centered responses) require a lot of training.

Finally, it should also be noted that, in contrast to objectbased modulations of eye movement planning in SEF (Olson, 2003), posterior parietal area LIP, which is also involved in eye movement planning, did not show object-centered coding of saccade targets (Sabes et al., 2002). It is possible that this is due to the fact that the stimuli used by Sabes et al. involved the rotation of an irregular, asymmetric shape, in contrast to stimuli that can be mentally translated left or right to match the ego center. Another prediction therefore is that object-centered effects might disappear if rotations of more complex, asymmetric objects were employed, which make the mental transformation back to egocentric coordinates more difficult.

Outside parietal and frontal areas, medial superior temporal (MST) neurons have been reported to signal target motion independent of eye or head movements, possibly in a worldcentered reference frame (Ilg et al., 2004). However, Sereno and Sereno (1991) have shown that position-independent, MST-like motion selectivity responses can develop in third-layer units of a feedforward network despite position-dependent direction selectivity within their receptive fields.

Chafee and Crowe (2012) distinguish between first-order (e.g., sensorimotor signals tightly coupled to stimuli or movements, in an egocentric frame of reference), second-order (signals are still dependent on e.g., egocentric position and movement parameters, but can be modulated by cognitive factors, such as attention, working memory, delayed planning), and third-order (complete sensorimotor independence both temporally and spatially) signals. The neurophysiology evidence on object-based spatial representations reviewed here is consistent with a highly abstract, cognitive signal. While it is debatable to what extent this abstract signal is independent of ego-relative parameters, it seems clear that these are high-level, cognitive signals that are likely "trained into the brain" (Chafee and Crowe, 2012).

# FMRI STUDIES: BRAIN ACTIVATIONS FOR EGOCENTRIC VS. ALLOCENTRIC TASKS

While single-unit recordings are restricted to small numbers of brain regions, can neuroimaging reveal additional brain networks subserving allocentric spatial representations? Numerous fMRI studies have attempted to identify the neural substrates of allocentric and egocentric spatial processing (for reviews, see Galati et al., 2010; Boccia et al., 2014).

Despite the wide variety of tasks (and definitions) employed to probe allocentric spatial cognition, most studies fall into three broad categories: (1) spatial judgment tasks, e.g., tasks requiring subjects to report left/right locations relative to egocentric or object-centered coordinates (e.g., Galati et al., 2000; Neggers et al., 2006, similar to **Figure 4B**); or requiring spatial proximity or alignment judgments between two objects or objects and the ego (e.g., Saj et al., 2014); (2) spatial navigation tasks (virtual, imagined, or remembered; e.g., Committeri et al., 2004; Zhang and Ekstrom, 2013); and (3) allocentrically-guided movements, e.g., pointing or reaching to spatial targets relative to another object vs. relative to the ego (e.g., Thaler and Goodale, 2011; Chen et al., 2014).

In general, both egocentric and allocentric tasks have been reported to activate overlapping parieto-frontal networks, with generally greater egocentric than allocentric activations in superior parietal and superior frontal cortex, especially in the right hemisphere (Galati et al., 2000, 2010; Committeri et al., 2004; Neggers et al., 2006; Zhang and Ekstrom, 2013; Chen et al., 2014; Saj et al., 2014). Additional foci of greater egocentric than allocentric activation have been reported in superior or middle temporal gyrus (Neggers et al., 2006).

Note that despite evidence of overlapping parieto-frontal activations for both allocentric and egocentric tasks, no objectcentered topographic maps have been found in parieto-frontal areas, across multiple attempts (Sereno et al., 2009), in contrast to well-established retinotopic or face-centered maps in parietal and prefrontal cortex (Hagler et al., 2007; Filimon, 2010; Sereno and Huang, 2014).

Allocentric tasks induce greater fMRI activations than egocentric tasks in temporal lobe structures and occipital regions, including the lingual gyrus (Galati et al., 2000; Committeri et al., 2004; Neggers et al., 2006; Chen et al., 2014); inferior temporal gyrus (Committeri et al., 2004; Zaehle et al., 2007; Saj et al., 2014); and hippocampus (Galati et al., 2000; Zaehle et al., 2007). Other fMRI studies have reported increased functional connectivity between the hippocampus, the superior parietal cortex, and precuneus in allocentric tasks (Zhang and Ekstrom, 2013). Thus, despite the overlap between egocentric and allocentric task activations, allocentric tasks rely more on ventral occipitotemporal networks, whereas egocentric tasks activate parietofrontal networks more strongly (for an exception to the latter pattern, see Thaler and Goodale (2011) as well as Zaehle et al., 2007).

While this pattern is consistent with the idea of functionally and partially anatomically separate neural processes underlying allocentric and egocentric spatial cognition, here I examine whether different activation patterns (and ventral visual stream activations in particular) provide evidence for a separate allocentric spatial reference frame. I will argue that the different patterns of activation are task- and strategy-dependent, where the egocentric spatial frame is relied upon to varying degrees in combination with non-spatial object-recognition processes.

Regarding spatial judgment tasks, at first glance, the greater parieto-frontal activation for egocentric tasks reported by most studies appears puzzling. If allocentric processing involves mentally shifting or rotating objects to the egocentric midline, such that an object's left and right are concordant with the egocentric left and right, wouldn't this imply greater activation for allocentric than egocentric tasks, at least in posterior parietal cortex, due to allocentric tasks in fact relying on additional ego-relative processing?

In fact, such a pattern of greater parietal activation for allocentric tasks has been reported, and appears to depend on the nature of the task. Zaehle et al. (2007) for instance, used verbal descriptions of spatial relations instead of actual visual images. In the allocentric condition, subjects listened to descriptions of the location of geometric shapes (triangles, circles, squares) relative to each other (e.g., shape A was to the left of shape B, B was above C, shape C was to the right of D). Subjects were then asked to infer the spatial relationship between two shapes whose spatial relationship to each other had not been described, but could be inferred from the other objects (e.g., where D was relative to A). In the egocentric condition, spatial locations of objects were described relative to the body as well as relative to other objects (e.g., shape A is to your right; shape B is to the right of shape A), but subjects had to infer the spatial relationship of a target object relative to themselves (e.g., whether shape B was to their right).

Zaehle et al. (2007) found that, although both egocentric and allocentric conditions activated parieto-frontal regions, inferior temporal gyrus, and occipital areas, the allocentric condition led to greater activation than the egocentric condition in the right superior and inferior parietal lobule, the right superior and inferior frontal gyrus, the ventrolateral occipito-temporal cortex (inferior temporal gyrus), and the hippocampus.

These results are consistent with mental imagery of the different visual shapes. However, notice that here, both the egocentric and the allocentric conditions invoke mental imagery. The greater right parietal activations for the allocentric condition could be due to the additional effort of translating object-relative spatial locations of each object into egocentric coordinates, whereas in the egocentric condition, this relationship is already described. The allocentric task can be solved equally by keeping track of each object's location relative to oneself, and comparing the egocentric location of shape A and that of shape D. As argued in previous sections, it is possible to solve this type of problem in purely egocentric terms.

Why then have other studies reported the reverse pattern of greater parieto-frontal activations for egocentric than allocentric conditions? One possible reason is that in contrast to Zaehle et al.'s study, where both the egocentric and the allocentric conditions required mental imagery, most other fMRI studies use visual stimuli (e.g., Galati et al., 2000; Neggers et al., 2006). Spatial judgments based on actual egocentric visual stimulation may lead to stronger activations than spatial judgments based on imagined object translations back to the ego center in allocentric conditions.

For instance, Galati et al. (2000) and Neggers et al. (2006) both used a horizontal bar intersected by vertical lines at various positions relative to the bar midpoint. The horizontal bar was also displayed at various horizontal positions relative to the ego-center. In the allocentric condition, subjects had to report whether the vertical line was left or right of the horizontal bar midpoint, regardless of its egocentric position. In the egocentric condition, subjects reported whether the vertical line was to the left or right of their body midline. Both studies found stronger right posterior parietal activations for the egocentric condition compared to the allocentric task. Moreover, allocentric activations were much weaker overall, with neither study reporting significantly greater allocentric than egocentric activations. In Galati et al. (2000), there was a trend for greater medial occipital and hippocampal activation in the allocentric compared to egocentric condition, which however did not reach statistical significance. Both of these studies are consistent with more robust activation of posterior parietal cortex when the egocentric spatial location is presented visually rather than mentally imagined. Alternatively, it is possible that establishing the egocentric "body midline" may require greater effort due to less precise proprioceptive mechanisms, compared to estimating simple retinal distances in the allocentric condition, thus leading to greater activation in the egocentric condition. Similar results were also obtained by Saj et al. (2014), who used vertical alignment judgments between two shapes vs. between one shape and the egocentric midline, as allocentric and egocentric tasks, respectively. This task can also be solved in purely egocentric terms, by calculating the retinocentric vector from the fovea to each shape. If one vector is longer than the other, clearly the two shapes are not aligned with each other. Similar to other studies, Saj et al. also obtained stronger right posterior parietal activations for the egocentric compared to the allocentric task, and greater allocentric than egocentric activation in left inferior temporal cortex.

Other fMRI studies have compared egocentric vs. allocentric tasks in spatial navigation or more complex virtual environments. Committeri et al. (2004) used snapshots of a virtual environment taken from different points of view, representing a central square with a fountain and a three-winged palace surrounding it. Inside the courtyard, two target objects and a reference object were displayed at different spatial distances to each other, to the central wing of the palace, and to the subject. Subjects had to decide which of the two target objects was closer to them (viewer-centered condition), which was closer to the reference object (object-centered condition), and which was closest to the central wing of the palace (landmark-centered condition). Note that each of these conditions is equally solvable in egocentric terms: during training, subjects learn view-specific layouts of the environment, together with where the central wing is relative to them, in each scene. Hence deciding which target object is closer to the central wing of the palace (landmark condition) could be solved by first establishing whether the central wing is on the egocentric right or left, and which of the two target objects is more right or left, in egocentric terms. Deciding which of two target objects is closer to a reference object (object-centered condition) likewise involves estimating which retinal distance between two points (target 1 and reference object, or target 2 and reference object) is shorter, together with depth and size cues of objects that are nearer or farther. In contrast to the other conditions, the landmark condition additionally requires retrieval of different (viewpoint-dependent) scene views from memory. All three conditions activated posterior parietal cortex. The main differences consisted of a bilateral ventro-lateral occipito-temporal activation (inferior temporal gyrus) present only in the object-centered condition, and medial occipitotemporal (fusiform, lingual gyrus and parahippocampal cortex) activations in the landmark condition (which relied on scene recognition).

Thus, the ventral visual stream activations are consistent with representations of visual distances (similar to Saj et al., 2014, where the retinal distance between two objects had to be estimated in the allocentric task). This is also consistent with patient D.F.'s deficits discussed in the Neuropsychology Section above. The medial occipito-temporal activations are consistent with view-dependent scene recognition. While allocentric spatial frames independent of the viewer's perspective could be postulated, the alternative egocentric explanations are at least as likely.

Also supporting the interpretation that directly perceived egocentric coordinates activate spatial networks more than egorelative mental transformations, Zhang and Ekstrom (2013) found that a simple control condition of just navigating to a visible target led to as much, if not more, activation as various imagined mental transformations necessary for navigating from one landmark to the next, in retrosplenial cortex, precuneus, parahippocampus, and superior parietal cortex. This is consistent with Bennett's (1996) hypothesis that animals navigate most efficiently based on recognized landmarks by moving toward them, rather than by using cognitive maps of ego-independent allocentric spatial relationships between landmarks.

Moreover, Huang and Sereno (2013) recently showed that the mental navigation network, which includes retrosplenial cortex, posterior parietal, premotor, precuneus, parahippocampal, and occipital regions, largely overlaps with retinotopic, and hence egocentric, maps. In fact, they suggest that this bottomup retinotopic organization helps encode scene and location information in an eye-centered reference frame for use in topdown, mentally simulated navigation.

The greater reliance on ventral visual or temporal lobe activations in some allocentric tasks could thus be interpreted as tasks that place greater memory or mental navigation demands—but nevertheless from an egocentric perspective, e.g., remembering sequences of landmarks from an egocentric perspective.

Finally, other fMRI studies have investigated the use of allocentric and egocentric frames of reference in the context of planning and executing movements toward remembered or remapped targets (Thaler and Goodale, 2011; Chen et al., 2014). Although a shift from dorsal to ventral visual regions has been proposed for immediate vs. delayed movements, respectively, both dorsal and ventral visual stream areas are re-activated at the time of delayed movements, with greater reliance on ventral areas in the case of delayed grasping compared to pointing, presumably because detailed visual information about object size and shape needs re-activated (Singhal et al., 2013).

In a study by Chen et al. (2014) differently colored horizontal dots indicated the fixation point, target, and allocentric landmarks. The fixation dot appeared first, followed by a target together with a landmark cue, at various horizontal eccentricities. The target and landmark then disappeared, and the fixation point was shifted to the center. Following a 12 s delay, the landmark reappeared either at the same or different location and an auditory reach instruction was given. In the egocentric conditions, subjects either reached to the remembered egocentric location of the target (pro-reach), or to the opposite location of the egocentric target (anti-reach). In the allocentric condition, subjects reached to the remembered location of the target relative to the allocentric landmark (i.e., if the landmark had shifted, the implied reach target shifted with it). In control trials, subjects reported the color of the target.

During the delay, the exact location of the future reach target could not be predicted, since the allocentric landmark could reappear at novel locations relative to the fixation point. Similarly, the egocentric target location could be revealed as either the remembered location or the opposite location (although in principle subjects could be maintaining two simultaneous egocentric target locations in working memory). However, since targets and landmarks consisted of differently colored dots subjects could presumably rehearse the target-landmark configuration as a unit, akin to an object configuration, in the allocentric condition (e.g., a red and blue dot for target left, landmark right, respectively). As argued previously, this configuration remains dependent on how this arrangement appeared from the ego perspective (see **Figure 1** and Section Theoretical Considerations).

As expected, Chen et al. (2014) found that during the delay, both egocentric and allocentric target encoding activated parietal and premotor areas. However, egocentric encoding of target position activated the posterior parietal lobe and PMd (dorsal premotor cortex) more strongly than the allocentric target encoding condition. Conversely, during the delay, the allocentric condition led to greater activation in the lingual gyrus, cuneus, and calcarine, i.e., all visual areas. Note that this is consistent with a spatial encoding in the egocentric condition, but a more visual configuration, similar to object processing, in the allocentric condition.

Thus there obviously is an effect of "allocentric" cues however, it is debatable whether this should be interpreted as an allocentric spatial reference frame effect rather than a ventral visual stream, object configuration or object processing effect, where multiple visual stimuli are treated as a unit (c.f. Logothetis, 2000).

Effects of "target left of the allocentric landmark" vs. "target right of the allocentric landmark" during the delay were also constrained to the ventral visual pathway, namely the inferior temporal gyrus and inferior occipital gyrus (Chen et al., 2014). This is also consistent with a retinotopic representation of an object, with egocentrically more left vs. more right locations activating object processing areas that contain retinotopic visual maps (Huang and Sereno, 2013).

Although Thaler and Goodale (2011) found the opposite pattern (allo > ego in parieto-frontal circuits) for cursor movements to allocentrically-defined targets, the delay and movement planning phases were not separated, and allocentric targets could have been immediately converted to egocentric coordinates from the beginning.

In summary, fMRI studies have generally shown a pattern of overlapping activations in parieto-frontal regions for allocentric and egocentric tasks, which can presumably be explained by the common translation of both "egocentric" and "allocentric" targets into ego-relative coordinates. Additionally, regions specific to allocentric spatial judgment tasks overlap with ventral visual areas involved in object and object configuration processing. Allocentric tasks that involve mental navigation between different landmarks presumably involve additional mental transformations of the ego into different imagined orientations, hence activating hippocampal and related (e.g., retrosplenial) regions that encode or store multiple viewdependent scene representations during navigation. Although the activation patterns for egocentric and allocentric tasks are partly distinct, they do not unequivocally support the existence of allocentric spatial reference frames, and could thus be reinterpreted using egocentric reference frames alone.

# CONCLUSIONS AND PREDICTIONS

The evidence reviewed here, spanning behavioral, neuronal, neuropsychological, and neuroimaging studies, suggests that allocentric spatial representations may not be independent of egocentric coordinates, whether for navigation, spatial perception, or target-directed movements. Both empirical evidence and theoretical considerations suggest that spatial mechanisms relying only on egocentric reference frames cannot be ruled out. Egocentric explanations for allocentric effects have been proposed before (e.g., Bennett, 1996; Mozer, 1999; Driver and Pouget, 2000; Wang and Spelke, 2002; Deneve and Pouget, 2003; Rorden et al., 2012). This review has attempted to unify a wide variety of findings from multiple fields of investigation, and to show how egocentric mechanisms could account for allocentric task effects in multiple domains.

Not all spatial judgments rely on the immediate (present) egocentric perspective, but can involve mental transformations such as imagined rotations and translations of the ego or of objects (ego-relative coordinates). Moreover, abstract rules can be built upon ego-relative spatial representations, as discussed in the Neurophysiology Section. Similarly, for recognition of landmarks to be possible from multiple viewpoints, viewpoint-invariance needs to be established.

This suggests a hierarchy of representations, including parieto-frontal egocentric spatial representations and egorelative remapping, landmark and object recognition in the ventral visual stream, path integration in hippocampal and related structures, and categorical rule representation involving prefrontal networks. While these networks undoubtedly work together to solve the many complex spatial tasks that animals face, the key argument made here is that bottom-up spatial representations are fundamentally ego-dependent.

What criteria could be used to evaluate whether an allocentric spatial reference frame is used? Some possible criteria include:


tied to that object, such that a cell should respond to object A at the new, updated allocentric location, even if other objects in the scene have not moved.

Other specific testable predictions include:

	- the improved spatial localization accuracy when presenting spatial targets relative to landmarks should disappear if the allocentric landmark is an irregular shape that is rotated between initial and post-delay presentations. Conversely, if the target is encoded in an object-centered reference frame, rotation of the landmark should have no effect on accuracies (or on reaction times), since the allocentric relationship should be independent of the egocentric perspective.
	- hemineglect patients would be expected to show no objectbased neglect for novel objects that are radially symmetric or which lack an intrinsic longitudinal axis that could be mentally rotated upright to match an egocentric, viewpointdependent representation of such an object. Instead, the egocentrically-defined contralesional half of such unfamiliar objects would be expected to be ignored in any orientation. The lack of a canonical upright orientation for such objects predicts that mental rotations should not take place for these objects.
	- object-based neglect will vary as a function of encoding vs. retrieval, and familiarity with an object. In other words, object-centered neglect should appear for novel objects experienced in a particular orientation over and over again, as a view-dependent mental representation becomes established over time.
	- brain activity for object-based spatial decisions should be slower than for egocentric spatial decisions (note that behavioral reaction times may not be sensitive enough to detect such temporal delays). EEG or MEG, or event-related fMRI and effective connectivity, could establish the time courses of different brain networks during "allocentric" and "egocentric" tasks. Egocentric decisions should show an earlier temporal profile compared to allocentric decisions, at least in parieto-frontal networks associated with space perception. Rule-based spatial decision making should activate prefrontal decision making regions such as DLPFC (Filimon et al., 2013) earlier than parieto-frontal spatial networks.
	- the fMRI literature suggests that parietal (or parieto-frontal) activations should generally be stronger for seen, rather than imagined, spatial relations. Stronger activations for visual observation than imagery, or for visible compared to invisible reaching, have indeed been reported in the posterior intraparietal sulcus and high-level visual areas (Filimon et al., 2007, 2009, 2015).

However, the more difficult the (allo-to-ego) mental transformation required for an allocentric stimulus (e.g., mental rotations, etc.), the stronger the activation should be.

The current review demonstrates several difficulties and challenges in teasing apart allocentric spatial reference frames, non-spatial mechanisms, and egocentric representations. The examples given here illustrate that it is possible to explain a wide variety of allocentric task effects using egocentric spatial reference frames. The interpretation offered here is of course only one possible interpretation, and it is certainly possible to refer to object recognition as "allocentric" if what is meant by that is the ability to categorize multiple viewpoints as the same object. However, this is not necessarily an

# REFERENCES


Bennett, A. T. (1996). Do animals have cognitive maps? J. Exp. Biol. 199, 219–224.


agreed-upon definition. Future studies could test the specific predictions made by the egocentric account and control for alternative non-spatial explanations. A clear and consistent definition of the term allocentric will be a key step in this direction.

# ACKNOWLEDGMENTS

I thank Artemiza Woodgate (née Filimon) and Jonathan D. Nelson for critical feedback and support.


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Filimon. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Haptically Guided Grasping. fMRI Shows Right-Hemisphere Parietal Stimulus Encoding, and Bilateral Dorso-Ventral Parietal Gradients of Object- and Action-Related Processing during Grasp Execution

## Mattia Marangon, Agnieszka Kubiak and Gregory Króliczak \*

Action and Cognition Laboratory, Department of Social Sciences, Institute of Psychology, Adam Mickiewicz University in Poznan, Pozna ´ n, Poland ´

#### Edited by:

Simona Monaco, University of Trento, Italy

#### Reviewed by:

Anthony Singhal, University of Alberta, Canada Robert Leslie Whitwell, The University of Western Ontario, Canada

> \*Correspondence: Gregory Króliczak krolgreg@amu.edu.pl;

krol.greg@gmail.com

Received: 30 September 2015 Accepted: 07 December 2015

Published: 05 January 2016

#### Citation:

Marangon M, Kubiak A and Króliczak G (2016) Haptically Guided Grasping. fMRI Shows Right-Hemisphere Parietal Stimulus Encoding, and Bilateral Dorso-Ventral Parietal Gradients of Object- and Action-Related Processing during Grasp Execution. Front. Hum. Neurosci. 9:691. doi: 10.3389/fnhum.2015.00691 The neural bases of haptically-guided grasp planning and execution are largely unknown, especially for stimuli having no visual representations. Therefore, we used functional magnetic resonance imaging (fMRI) to monitor brain activity during haptic exploration of novel 3D complex objects, subsequent grasp planning, and the execution of the pre-planned grasps. Haptic object exploration, involving extraction of shape, orientation, and length of the to-be-grasped targets, was associated with the fronto-parietal, temporo-occipital, and insular cortex activity. Yet, only the anterior divisions of the posterior parietal cortex (PPC) of the right hemisphere were significantly more engaged in exploration of complex objects (vs. simple control disks). None of these regions were re-recruited during the planning phase. Even more surprisingly, the left-hemisphere intraparietal, temporal, and occipital areas that were significantly invoked for grasp planning did not show sensitivity to object features. Finally, grasp execution, involving the re-recruitment of the critical right-hemisphere PPC clusters, was also significantly associated with two kinds of bilateral parieto-frontal processes. The first represents transformations of grasp-relevant target features and is linked to the dorso-dorsal (lateral and medial) parieto-frontal networks. The second monitors grasp kinematics and belongs to the ventro-dorsal networks. Indeed, signal modulations associated with these distinct functions follow dorso-ventral gradients, with left aIPS showing significant sensitivity to both target features and the characteristics of the required grasp. Thus, our results from the haptic domain are consistent with the notion that the parietal processing for action guidance reflects primarily transformations from object-related to effector-related coding, and these mechanisms are rather independent of sensory input modality.

Keywords: haptic exploration, encoding bias, action planning, grasp execution, complex objects, dorsal stream

# INTRODUCTION

When searching a key in a deep pocket, or reaching for an electric torch in a drawer right after an evening power outage, our fingers are used to actively explore the encountered shapes to find the desired target. Yet, when there is only a single, and unobstructed goal object with a familiar size and/or structure, the hand—even though directed somewhat "blindly"—may already be suitably open and even rotated in anticipation for grasping the expected target. Such skilled actions are possible in the absence of direct vision because the control of manual tasks in the sighted person is under such conditions likely mediated by the visuallyencoded properties of objects processed in the ventral perceptual stream (Goodale and Milner, 1992; Milner and Goodale, 2008; see also Króliczak et al., 2008; Singhal et al., 2013; cf. Ungerleider and Mishkin, 1982). Of course, purely sensorimotor and/or kinesthetic information (e.g., Fiehler et al., 2008) must be also incorporated in the functioning of the dorsal action stream (Goodale and Milner, 1992) for the guidance of such motor skills (For a recent review on the contributions of visual and haptic information to reaching and grasping see Stone and Gonzalez, 2015; see also a review on somatosensory processes involved in perception and action by Dijkerman and de Haan, 2007).

It is not known, though, whether or not action guidance would rely on similar circuits if confronted with completely unfamiliar objects or their shapes that had never been encoded with the use of vision—a situation a person who loses sight later in life would be confronted with. On the one hand, there is compelling evidence that when object shape information is first acquired exclusively by active touch (haptic exploration) its encoding is associated not only with the dorsal, superior parietal lobule activity (Binkofski et al., 1999a) but can also invoke ventral stream regions, such as the ventro-lateral extents of the occipital lobe typically associated with visual tasks (James et al., 2002; see also Amedi et al., 2001, 2002). On the other hand, the haptic parallels to dissociated visual processing of objects for different tasks (e.g., Ungerleider and Mishkin, 1982; cf. Goodale and Milner, 1992; see also Rizzolatti and Matelli, 2003; Binkofski and Buxbaum, 2013) are limited. That is, despite evidence that haptic object recognition (what an object is) vs. its localization (where it is positioned) is also mediated by relatively independent mechanisms, both of these skills have been shown to invoke the dorsal-stream regions. In fact, it has been demonstrated that there is somewhat greater inferior parietal lobule contribution to haptic object recognition, and bilateral superior parietal lobule involvement in tactile object localization (Reed et al., 2005; see also Reed et al., 2004). Therefore, the pathways underlying haptically-driven action guidance (Dijkerman and de Haan, 2007) may differ markedly from those originally proposed for visually-guided actions. That is, the superior parietal cortex may underlie encoding of object properties for the control of actions directed toward these objects (cf. Jäncke et al., 2001; Fiehler et al., 2008), whereas the more ventral pathways, including secondary somatosensory cortex and terminating in the insula, may play a greater role in object recognition (see also James et al., 2007).

Here, we investigated the neural underpinning of hapticallyguided grasping directed at objects never seen before. To this end, functional magnetic resonance imaging (fMRI) was used to measure the blood oxygen level dependent (BOLD) signal changes associated with exploration of the shape and orientation of novel objects, the subsequent grasp planning, and the actual execution of grasping movements directed at these objects. As such, all the tasks performed in this experiment were based entirely on the haptically acquired information. Not only were we interested in testing for any analogies to visually guided performance of grasping but we were also interested to get to know the patterns of brain activity that would emerge during the preparatory phases, ultimately leading to the grasping of the target objects. Based on previous studies on delayed manual actions (Fiehler et al., 2011; Singhal et al., 2013), we hypothesized that the areas involved in object shape, size and orientation encoding—i.e., engaged during haptic exploration—would be later invoked for object grasping. We also assumed that grasping of the more complex objects (vs. much simpler circular disks) could reveal not only the involvement of the superior parietal lobule (Binkofski et al., 1999a) but also some ventral stream, and/or insular cortex contribution to the task (James et al., 2002; Dijkerman and de Haan, 2007). Finally, we hypothesized that if the encoded object shape information is stored over a brief delay period, its reactivation during the planning phase may invoke re-recruitment of regions anterior to the ones that would be engaged during the grasping task (cf. Valyear et al., 2007; Singhal et al., 2013). That is, anterior vs. posterior activity gradients were expected within regions contributing to planning vs. execution of haptically-guided actions.

# METHODS

# Participants

Ten University of Oregon students and postdoctoral fellows with no history of neurological or psychiatric disorders (four females; mean age = 28.1, SD = 5.2) took part in this study after giving written informed consent. All of them were righthanded as measured by the Edinburgh Handedness Inventory (Oldfield, 1971), had normal or corrected-to-normal visual acuity (important mainly for reading instructions and just one control task), and they were all compensated financially for their time. The local Ethics Committee, and the Bio-Ethics Committee at Poznan University of Medical Sciences, approved the experimental protocols, which conformed to the WMA Declaration of Helsinki.

# Familiarization Phase

There was a short practice that took place just before the study proper, with objects that were not a part of the experimental set. Participants were told that their task is to explore the novel objects in order to find their axes of elongation because they will be later asked to grasp these objects. Everybody was encouraged to explore the targets carefully to be absolutely sure how they should be grasped, and it typically took the whole exploration time to perform this task for the majority of complex objects. As to simple circular disks, participants were asked to explore them for the whole task interval (so that any difference in brain activity should not be due to lack of exploratory movements in this simpler task, but due to clear differences in object shape processing). There was no specific instruction about how the grasp planning should be performed. As to grasping the complex objects, participants were explicitly told to grasp them along their long axes. Circular shapes, conversely, were to be grasped in the most convenient way. Importantly, participants were asked not to correct for any grip imprecision, and were instructed not to lift the objects (off the surface to which they were attached).

# Stimuli and Procedure

The experimental stimuli consisted of 32 three-dimensional objects of different shapes and sizes, and most of them were merely larger versions of the stimuli used earlier by Króliczak et al. (2008). Made of white translucent plastic, these objects had a constant depth (of 0.6 cm), but varied in length (between 3.4 and 4.6 cm), and width (typically between 2.6 and 3.6 cm, although the narrower part of objects was close to 0.7 cm). The examples of stimuli used are shown in **Figure 1**, and all of them, their order, and orientations are shown in **Supplementary Figure 1**. Arranged pseudorandomly into four (4) sets of eight objects of different shapes, orientations (with an equal number rotated slightly leftward and rightward), and/or to some degree also sizes, with each set including two circular control disks (4 cm in diameter), they were attached to four Velcro strips. The separate strips were then placed centrally on, and presented with, a custom made MR-compatible device somewhat resembling the "Grasparatus" created and used in the Culham Lab (e.g., Króliczak et al., 2008).

Each consecutive trial consisted of a different object, but each object in a given set was eventually presented three times, thus resulting in 24 trials per run. The four sets of objects were changed pseudorandomly between each of the five consecutive runs for each participant. This means that one of the object sets (a random one, but most often the one that was used in the first run) was presented for the second time in the 5th and last run. Thus, each participant received a different, pseudorandom order of target objects (by manipulating the order of object sets). They were attached to the grasparatus's drum, which was located above the participants' hips. Notably, for the person lying in the MRI scanner, the stimuli were within the reach of the hand,

FIGURE 1 | Examples of three-dimensional stimuli used in the experiment. Only one object was presented at a given time. Participants neither saw these stimuli before, nor during the study proper (i.e., no visual feedback was ever provided during any of the tasks). Although the orientation of the objects varied, their location remained the same. Upon their haptic exploration, and subsequent planning, participants grasped these targets with a precision (pincer) grip, using the index finger and thumb. The complex objects were always grasped along their longer axes, whereas the circular disks were typically grasped with the most comfortable grip.

but could be neither seen directly nor via the mirror (which actually reflected instructions from the screen located behind the scanner). Similarly, the participants did not have any visual feedback of their hands.

All the manual tasks in the main study of this project were performed with the dominant right hand, whose initial position was indicated by a custom-made start key placed near the belly button. A participant was first asked to explore the presented object for 5 s (starting with an "EXPLORE" cue), and then to move the hand back to press the key within the subsequent 2 s (a period clearly marked with a "RETURN" cue). Next, during a variable interval of 3.5, 4.5, or 5.5 s, the task was to plan a grasping movement of the just explored object (with the beginning of this task indicated with a "PLAN" cue). Subsequently, 50% of the trials involved the execution of the pre-planned grasp (triggered by a "GRASP" cue) wherein a complex object was always grasped along its longer axis, and a simple circular disk was grasped with the most comfortable grip and hand orientation. In 25% of trials the task was simply to reach toward an object and touch it with the knuckles (the "REACH" cue), and in the remaining 25% of trials a participant was asked to withhold a planned response (upon hearing the "WAIT" cue, which given the cue that followed effectively meant "no-go"). Each task concluded with a "REST" cue, resulting in an inter-trial interval that varied between 7.5 and 9.5 s (starting from the beginning of the cue). Upon completion of a given task by the participant, the experimenter rotated the drum manually to present the next object. The drum rotation typically followed the "REST" cue, which was easy to time because the experimenter could also hear all the cues via headphones. Given the adopted duration and variability of events within trials, a single run typically lasted just over 9 min. Trial structure and timing is shown in **Figure 2**.

Initially, we intended to present all the task cues (e.g., explore, return, grasp, etc.) auditorily via the MR-compatible headphones, with the duration of each cue set to 750 ms. However, given that during pilot testing a volunteer complained about distortions in auditory signal, to make sure that all the cues can be easily understood, in addition to fixing sound quality we also decided to present visual cues for 1.5 s each. This was done with a white Tahoma Regular font on a black background, in capital letters, size 100, subtending ca. 10 × 2 ◦ when projected on a screen behind the scanner bore, and viewed via a mirror from a distance of ∼70 cm. The onsets of the cues were synchronized. This manipulation resembles naturally occurring situations wherein we may hear a request for action and see an accompanying visual signal (e.g., gesture) that strengthens its clarity, but we actually do not see the target of the to-be-performed response.

Testing was carried out in a darkened room. Although the "commands" were displayed visually, the grasping task was guided exclusively based on information obtained with haptic exploration a few seconds before, including the variable time interval for grasp planning. Because no visual feedback was ever provided during task performance, its execution (due to task novelty, i.e., little practice with the task and novel stimuli) seemed quite difficult at first. To make grasping actions a bit easier, despite changes in orientation, the objects were presented in the same central location.

# Additional Localizer Scans

All the 10 participants were also tested in at least two different functional localizer runs (Some of the participants agreed to perform a given localizer scan twice).

The first functional localizer served to identify the brain area known as aIPS, and the tactile-visual subdivision of the lateral occipital cortex, dubbed LOtv. As the first acronym implies, the aIPS is located anteriorly in the intraparietal sulcus (typically on its lateral bank, near or at the intersection with the postcentral gyrus), and it has been linked to the guidance of grasping movements (e.g., Binkofski et al., 1998; Culham et al., 2003; Króliczak et al., 2007; cf. Gallivan et al., 2009; Monaco et al., 2011). As the second acronym implies, the LOtv is a multimodal area, located anteriorly, and more inferiorly, to area MT+, at the junction of the ascending limb of the inferior temporal gyrus, and the lateral occipital sulcus (Amedi et al., 2002). Participants were asked to search for and explore small toy plastic objects, such as animals, tools, and other man made gadgets, placed among irregular pebble-like or more regular cube-like plastic shapes in the bags attached to the wrists of their hands. The task was to find a meaningful shape with the tips of the fingers, categorize it if possible, and continue searching for further toys resembling common objects. There were 12 blocks of the exploration task, each lasting 16 s, interleaved with 12 blocks of 16-s rest periods, during which the fingers were kept still, but touched the shapes, regardless of whether they were meaningful or not. The order of blocks was counterbalanced across participants.

The second functional localizer served to identify the brain area known as LO cortex. Located on the lateral extent of the occipital lobe this area is typically defined by viewing intact vs. scrambled objects (e.g., Kanwisher et al., 1997; see also Ferber et al., 2003; Large et al., 2005). Typically, this object-selective area—implicated in the bottom-up analyses of visual shapes is located right behind (but may also partly overlap with) the motion-selective area MT+ (cf. Kourtzi and Kanwisher, 2000; see also Dumoulin et al., 2000).

It was the only task in this study when participants actually looked at visual images of objects and their scrambled counterparts, and it was always run last. In addition to common household objects and tools, participants were also shown computer generated novel objects (used earlier by Harman et al., 1999; Króliczak et al., 2003), as well as the silhouettes of the previously explored shapes. Indeed, it was actually the first time when the haptically experienced shapes were also encoded visually. Ten (10) different objects, with one random repeated, each separated by a 150-ms mask composed of thin intersecting parallel (horizontal and vertical) lines, were shown in six blocks lasting 12 s, separated either by six blocks of 10 scrambled objects (one random repeated) which were also separated by the 150 ms mask, or by six blocks of rest periods with a fixation point. There were two blocks with visual images of common items, two blocks with novel objects, and two blocks with the silhouettes of the haptically experienced shapes. The same number of blocks was used for the presentation of their scrambled counterparts. Participants performed a one back task wherein they were to indicate with a button press the appearance of the repeated object, or the repeated scrambled pattern.

Because exploratory finger movements may not only be associated with the engagement of LOtv, but also motion sensitive regions (Kourtzi and Kanwisher, 2000; Amedi et al., 2002; see also Dumoulin et al., 2000), for a more in-depth interpretation of the results it has been necessary for us to know the location of the motion-selective area MT+. It was established by two multi-localizer scans from a different cohort of 21 right-handed participants of similar age (11 females). Areas sensitive to two kinds of visual motion, and to the control of two kinds of hand movements were identified. The right and left hands were always tested separately, and typically on 2 consecutive days, whereas the visual stimuli remained basically the same. These stimuli typically consisted of superimposed radial, and concentric gratings, similar to the ones used by Culham et al. (1999), rotating either clockwise, and/or counter-clockwise in three different 14-s blocks (24 steps of 15◦ rotation per block) or contracting and/or expanding, again in three different 14-s blocks (4 consecutive steps of 1.7◦ forward or backward movement, changing position 24 times per block). During hand movement tasks, participants were asked to either rotate their wrist in four steps during the three different 14-s blocks (clockwise and counter-clockwise in a pace similar to the previously seen visual changes), or to reach out and move the arm back, again in four steps during the three different 14-s blocks (back and forth, in a pace similar to the contraction/expansion of the visual image). All the conditions were pseudorandomized, with one of the two visual conditions being always presented first when it comes to task blocks, and were supplemented with six (6) 14-s blocks of passive viewing of stationary radial, and/or concentric control gratings, and additional six (6) 14-s rest periods, with a fixation dot in the middle of the screen.

# MRI Procedures

In the main experiment, and two localizer scans that immediately followed, a Siemens Allegra 3T MRI system (Siemens, Erlangen, Germany) equipped with echo planar imaging (EPI) capabilities, with a 12-channel phased array transmit/receive head coil, was used for data acquisition at the Lewis Center for NeuroImaging at the University of Oregon. Supplemental localizer scans were acquired at the Nencki Institute of Experimental Biology in Warsaw using a Siemens TRIO 3T Scanner with a 32-Channel Head coil, and very similar imaging parameters. Functional volumes were collected using a T2<sup>∗</sup> -weighted, segmented gradient-echo echo planar imaging (time to echo/time to repetition [TE/TR] = 30/2000 ms, flip angle [FA] = 80◦ , voxel size = 3.125 × 3.125 mm; field of view = 384 mm). Each volume was made up of 32 contiguous slices of 3.5-mm thickness. The initial first four volumes in each scan series were discarded. In the main "haptic" experiment of this project, each participant performed five functional runs composed of 275 volumes each. AIPS localizer scans involved the acquisition of 196 volumes per run, during the LO localizer scans only 156 volumes were obtained, whereas during the MT+/hand-movement multi-localizer 225 volumes were acquired on each day. Highresolution anatomical scans were collected using a 3D T1 weighted MPRAGE sequence (TE/TR = 4.38/2500 ms; FA = 8.0◦ , 176 contiguous axial slices, thickness = 1.0 mm, voxel size = 1.0× 1.0 mm; field of view = 256 mm). Siemen's Auto Align Scout and True FISP sequences were executed for each participant before data collection to ensure that slices were prescribed in exactly the same positions across runs. DICOM image files were converted to FSL NIfTI format using the software called MRIConvert (http:// lcni.uoregon.edu/\simjolinda/MRIConvert/).

# fMRI Data Analyses

Data analyses were performed using the FMRIB Software Library (FSL) version 5.0.6 (Jenkinson et al., 2012). The initial preprocessing steps involved: the use of Brain Extraction Tool (BET) for non-brain tissue removal (Smith, 2002), the application of motion correction MCFLIRT algorithm (Jenkinson et al., 2002), spatial smoothing with a Gaussian kernel of full width half magnitude (FWHM) = 8 mm, and high-pass temporal filtering with a cutoff = 50 s. In the functional data from the main (haptic) experiment, Siemens EPI-navigated prospective motion correction algorithm, followed by automatic retrospective re-acquisition, was applied during data collection, and the use of MCFLIRT was no longer required. (In fact, as indicated before, the use of additional motion correction algorithm would in such a case be detrimental; Króliczak, 2013.)

Whole brain (voxelwise) analyses were performed using FSL's fMRI Expert Analysis Tool (FEAT), part of FSL (Jenkinson et al., 2012). At the first level, each fMRI run was analyzed separately, with each condition modeled with a canonical hemodynamic response function (double-gamma HRF). Nine predictors, in the FSL software referred to as Explanatory Variables (EVs) were used, including two separate EVs—for complex and simple objects, respectively—for the three main conditions, i.e., Exploration (of complex, and simple objects), Grasp Planning (for complex, and simple objects), and Grasp Execution (for complex, and simple objects), as well as one EV for Reaching trials, one for NoGo trials, and finally one for "Rest" periods (i.e., the variable longer intervals between consecutive trials). Temporal derivatives for each explanatory variable were automatically created as additional regressors in order to correct for timing discrepancies (e.g., to correct for slice timing alignment).

Except for the Grasp Planning activity, which was modeled as the 3.5-s period beginning with the onset of the instructional cue (i.e., presented visually for 1.5 s, though aurally only for 0.75 s) and lasting through the end of the shortest (2.0 s) delay interval (as in Króliczak and Frey, 2009; and Króliczak et al., 2011; see also **Figure 2**), Exploration, Grasp, Reach, and NoGo conditions, as well as the baseline "Rest intervals" were modeled for their entire durations. Note also that the variable delay introduced between the grasp planning and execution phases substantially reduced the temporal coupling of the two phases, thus enabling an easier deconvolution of the signal from these disparate tasks (e.g., Króliczak and Frey, 2009; cf. Marangon et al., 2011.) The non-modeled Return intervals following exploration played the same role (here: clearly separating the exploration and planning-related signals), and together with the non-modeled "tails" of delay intervals for planning, as conditions of no interest, contributed to the calculation of mean activity in the run (the so-called implicit baseline). While testing for the main effects of tasks vs. explicitly-defined baseline activity, i.e., exploration vs. rest, plan vs. rest, etc., regardless of object type involved, greater weights were actually put on activity related to more complex tasks (i.e., +0.75 for complex objects, and +0.25 for simple disks, vs. −1 for rest). Of course, during testing for simple main effects of each task, and in all direct contrasts between the conditions, the balanced weighting was applied to each of the contrasted conditions.

The resulting first-level contrasts of parameter estimates (COPEs) served as inputs to the second-level analyses (within subjects, across individual runs) using a Fixed Effects model. The resulting second-level COPEs were then used as inputs to the third-level analyses (across participants), performed using a mixed-effects model, with the random-effects components of variance estimated with the default FSL's procedure, the socalled FLAME stage 1 (Beckmann et al., 2003). For the two critical contrasts, i.e., Grasping Complex object vs. Reaching, and Grasping Complex vs. Simple objects, additional analyses were also run with the more time consuming but therefore more robust FLAME stage 1 + 2 procedure (Beckmann et al., 2003). The outcomes from the whole brain (voxelwise) analyses are depicted in figures showing only significant clusters of signal modulations, typically in the form of increased brain activity. Inclusive contrast masking procedure was applied to identify areas significantly activated across two comparisons. In all the renderings, Z-statistic (Gaussianized T/F) images were thresholded with the use of Z > 2.3 and a corrected cluster significance threshold of P = 0.05 (Worsley, 2001). These are the default settings in the FSL's FEAT fMRI analysis tool, where significance level for each cluster is first estimated from Gaussian Random Field theory, then compared with the cluster probability threshold, and corrected accordingly. FSL LInear Registration Tool (FLIRT; as described by Jenkinson and Smith, 2001) was used to implement registration of functional images to high-resolution and standard space images (from the Montreal Neurological Institute [MNI-152] 1 mm brain template).

Anatomical localization of clusters with significant brain activity was always verified by manual comparison with an atlas (Damasio, 2005), and by projecting and visualizing these maps using the standard mapping algorithm implemented in the Caret software (http://www.nitrc.org/projects/caret/), where the group statistical imaging maps can be conveniently overlaid onto a population-average, landmark- and surface-based (PALS) human brain atlas (Van Essen, 2005). Overlays of activity were obtained with the Caret "convert metric to RGB" function, followed by additional adjustments and mixing of the overlaid colors in the three surface renderings.

# Region Of Interest (ROI) Analyses

A total of eight ROIs were selected and/or defined based on voxelwise group results from the main study, the outcomes from the two functional localizer scans, and a combination of thereof with the Juelich cytoarchitectonic maps and/or anatomical regions from the Harvard-Oxford probabilistic atlas included in the FSL package. Indeed, the additional use of the probabilistic maps also helped verifying the anatomical locations of our ROIs. In order to ensure extraction of separate clusters in a given brain region, the probabilistic maps were thresholded at (i.e., zeroed below) the 30% of their lower probability tails. (This way, for example, the middle frontal gyrus ROI did not include any of the voxels belonging functionally to the premotor cortex of the precentral gyrus.) Notably, although the selection of separate functional ROIs is very easy to perform manually using the "paint" tools in the FSLview package, the application of probabilistic atlases to extract ROI "masks" (volumes covering distinct regions) from group-average contrast activity also allows for an objective demarcation of clusters which are connected, and the removal of voxels at the borders with white matter. If necessary, additional localizers from the on-going projects in the lab were used for a comparison and/or clarification.

The primary goal of the ROI analyses was to determine the relative contribution of each selected area to all major studied tasks (exploration, planning, grasping), including task difficulty related to object type (complex, simple). To this end, a 3 (task) × 2 (object type) ANOVA was run on brain activity from their respective contrasts vs. the resting baseline, including the removal of signal related to instruction processing (i.e., exploration of complex objects vs. rest and instruction processing, exploration of simple objects vs. rest and instruction processing, grasp planning for complex objects vs. rest and instruction processing, etc.). The most common level of significance was adopted, i.e., α = 0.05. Where necessary, the required post-hoc tests were Bonferroni adjusted (marked as "Bf-p" to indicated that the P-value was corrected for multiple comparisons).

We focused only on left-hemisphere parietal and frontal areas which are typically linked to higher order manual skills (including action planning and execution; e.g., Frey et al., 2005; Króliczak and Frey, 2009; Jacobs et al., 2010; Marangon et al., 2011), and several ventral areas, such as MTG or LO expected to play a role in less rehearsed or delayed actions (e.g., Króliczak et al., 2007; Singhal et al., 2013).

# RESULTS

# Haptic Object Exploration

After accounting for instruction processing, and when compared to the resting baseline, the haptic exploration of target objects was associated with a bilateral engagement of both the parieto-frontal networks, and the occipito-temporo-insular networks (consistent with James et al., 2002; Dijkerman and de Haan, 2007). The contribution of the left hemisphere was greater for three reasons: (1) the lower-level sensorimotor activity was, due to the use of the right hand, almost exclusively left lateralized, (2) except for the subcortical and medial cortical structures, such as the presupplementary and the cingulate motor areas (pre-SMA, and CMA respectively), the clusters of activity were typically larger on the lateral surfaces of the left hemisphere, including aIPS, the anterior division of the supramarginal gyrus (aSMG), secondary somatosensory cortex (SII), and the ventral premotor cortex (PMv), and finally, (3) the very rostral subdivision of the middle frontal gyrus (rMFG) was engaged exclusively on the left. The clusters of significant activity revealed by this contrast are shown in the form of surface renderings, and in the most representative slices in **Figure 3A**.

Of note is the widespread activity on the medial surfaces in the striate and extrastriate areas of the occipital lobe (early visual cortices or EVCs, as in Singhal et al., 2013), and in the lateral temporo-occipital cortices (TOC), including its posterior division belonging to MT+ (as revealed by an overlap with voxels having >50% probability of belonging to the cytoarchitechtonic map of V5 from the Juelich atlas, and an MT+ localizer from our lab). Moreover, the more medial clusters were connected, via the parahippocampal gyrus, to the thalamic activity, which in turn was linked to the mid-to-anterior insular cortices, and closely related clusters in anterior divisions of the superior temporal gyri. In the left precuneus, the observed signal modulations overlapped with the antero-dorsal divisions of the superior parieto-occipital cortex (adSPOC, cf., Hutchison et al., 2015),

renderings in each of the panels illustrate significant group averages from the selected contrasts overlaid on the PALS atlas, whereas the lower axial slices illustrate this same activity in the most informative slices of an average brain obtained from all study participants' T1-weighted anatomical scans. All the images are displayed in neurological convention (i.e., right hemisphere is on the right), and are shown in hues corresponding to the color bars at the bottom of the panels. (A) Haptic exploration: there were significant increases of activity in all the major areas of the praxis representation network (PRN). They include the anterior supramarginal gyrus (aSMG), anterior intraparietal sulcus (aIPS), ventral premotor cortex (PMv), dorsal premotor cortex (PMd), supplementary motor areas (SMA complex), cingulate motor cortex (CMC), rostral middle frontal gyrus (rMFG), and the middle temporo-occipital cortex (TOC). The remaining clusters were observed in the early visual cortices (EVCs), superior parieto-occipital cortex (SPOC), including its anterior and dorsal subdivisions (adSPOC), primary and secondary somatosensory cortices (SI, SII), the thalamus, the insular cortex, and the neighboring superior temporal cortex. Except for the left sensorimotor cortex, most of the areas were engaged bilaterally. (B) Haptic exploration of complex vs. simple objects: the significant increases of activity were exclusively right lateralized and involved adSPOC, rostral superior parietal gyrus (rSPG), aIPS, aSMG, the fundus of the ventral postcentral gyrus (vPCG), and possibly SII. (C) Haptically-guided grasp planning: in addition to bilateral EVCs, extending into posterior fusiform gyrus (pFusG) on the left, the remaining activity was exclusively left lateralized and involved anterior-to-mid IPS (amIPS), posterior middle temporal gyrus (pMTG), caudal superior temporal gyrus (cSTG), and more anterior divisions of middle and inferior temporal gyri (aMTG and aITG). (D) Haptically-guided grasping of complex objects vs. reaching toward them: the significant modulations of activity involved aIPS, sensorimotor cortices, and a very small PMd cluster on the left, as well as SI, SII, aSMG, rSPG, and PMv on the right.

whereas in the right precuneus they were more rostral and dorsal. Finally, there was a clear involvement of the dorsal premotor (PMd) cortex, although in the right hemisphere the activity extended more onto the superior frontal gyrus.

# Haptic Exploration of Complex vs. Simple Objects

Consistent with earlier studies (Binkofski et al., 1999a; Reed et al., 2005), the haptic exploration of complex objects (vs. simple circular disks) was associated with significant signal modulations in anterior divisions of the posterior parietal lobe, spanning both its superior and inferior lobules, including the rostral superior parietal gyrus (rSPG), aIPS and aSMG, but also extending slightly onto SII. Notably, this single, dorso-ventrally stretched cluster of activity was exclusively right lateralized. This effect is shown in **Figure 3B**, again on surface renderings and in the most representative slices.

# Haptic Exploration of Simple vs. Complex Objects

None of the areas from the parieto-frontal action network showed significantly greater activity in this contrast. Conversely, a widespread and often interrelated net of clusters resembling the default mode network was revealed. Because all these regions and/or most of their subdivisions were not even activated when compared to the resting baseline, the observed effects result primarily from greater inhibition of this network during a more difficult task. These findings are illustrated in **Supplementary Figure 2** in the most representative slices.

# Grasp Planning Based on Haptically Obtained Information

After the subtraction of signal related to instruction processing, and when compared to the resting baseline, the significant grasp planning activity was localized primarily to the occipital and temporal cortices, and mainly to the left hemisphere. Interestingly, there was also a relatively small cluster of activity observed in the left amIPS, and in the premotor cortex (also on the left). Yet, the premotor cluster was conspicuously extended into the white matter, and there were spurious signal modulations in the vicinity of the corpus callosum. For these reasons, this contrast was re-run in the brain mask deprived of white matter (including the corpus callosum itself). The premotor cluster turned out to be too small to reach significance threshold. The remaining significant cortical activity was unaffected by this reanalysis, and is shown in **Figure 3C**.

In addition to the bilateral involvement of EVCs, including the dorso-medial striate and extrastriate cortices, the planningrelated activity—regardless of object type—was observed in the left ventro-lateral divisions of the occipital lobe, most likely including ventral visual area 4 (V4v), extending into the posterior fusiform gyrus (pFusG), and further into the occipito-temporal sulcus. The most conspicuous cluster was found in the posterior middle temporal gyrus (pMTG), although its caudal division extended dorsally (via the superior temporal sulcus) to the caudal superior temporal gyrus (cf. Glover et al., 2012). A smaller cluster of activity was also observed in the more anterior division of the inferior temporal sulcus and gyrus.

# Grasp Planning for Complex vs. Simple Objects (and Vice Versa)

None of the planning related significant activity was sensitive to object type because the contrast of planning the grasp for complex vs. simple objects was empty. The inverse contrast also revealed lack of significant differences with one exception: significantly weaker inhibition of the primary motor cortex on the left during the planning of simple grasps (cf. the exploration of simple vs. complex objects above).

# Haptically-Guided Grasp Execution

Similarly to the exploration task, when compared to the resting baseline and with instruction processing accounted for, the haptically-guided grasping of objects was again associated with a bilateral engagement of both the parieto-frontal networks, and the occipito-temporo-insular networks (with activity pattern quite similar to the one shown in **Figure 3A** for the exploration task). Interestingly, in addition to being more symmetrical, the activity was larger in its extent, and included regions that were not invoked during the exploration of objects, such as the superior temporal sulci and gyri (with their large bilateral involvement). Moreover, the left caudal intraparietal sulcus (cIPS), and the SPOC region were also clearly involved (cf. Gallivan et al., 2009; Monaco et al., 2015). However, when directly compared with activity from the exploration, neither of the parieto-occipital nor the temporo-occipital regions was more significantly engaged in grasping.

# Haptically-Guided Grasping vs. Reaching

To enable comparisons with earlier studies on visually guided grasping, we first ran a balanced contrast of Grasping [of Complex and Simple Objects] > Reaching. This contrast was empty. However, since reaching always involved the presence of complex objects, and the participants did not know ahead of time that it was going to be a reaching trial, the reaching task was quite demanding (i.e., required the processing of object features and a change in cognitive decision/manual performance). Therefore, a more appropriate comparison would involve a balanced contrast between Grasping of Complex Objects vs. Reaching toward Complex Objects. This was indeed the case. In addition to the expected greater involvement of the sensorimotor cortex on the left, a widespread somatosensory (primary or SI, and SII) engagement on the right, this contrast also revealed a familiar contribution of left aIPS (and to a lesser degree its right hemisphere counterpart), as well as rSPG, aSMG, and PMv exclusively on the right. In fact, except for the missing bilateral PMd and left SPOC contribution, this dorsal stream activity was very similar to the one observed for grasping vs. reaching in a study by Króliczak et al. (2008). The observed significant clusters of activity are shown in **Figure 3D**.

# Haptically-Guided Grasping of Complex vs. Simple Objects

Another important effects were revealed by a balanced contrast of the two grasping conditions (namely, Grasping Complex > Grasping Simple objects), as it shows all the brain areas sensitive to critical object features during grasp performance and/or how they are translated into appropriate grip scaling. As it turns out, nearly all the right-hemisphere PPC voxels that were sensitive to object shape during their exploration (see **Figure 3B**) were now re-recruited for the grasping of these objects. This activity was accompanied by significant signal increases in the sensorimotor areas of the left hemisphere contributing to hand guidance. But even more importantly, left aIPS, and PMd, as well as voxels anterior to classically defined area SPOC, also showed sensitivity to object features during grasping. All significantly activated clusters during grasping of complex objects—i.e., showing sensitivity to object shape—are shown in **Figure 4A**. Of particular note is the contribution of the left superior (dorsodorsal) parieto-frontal network, and left aIPS.

# Haptically-Guided Grasping of Simple vs. Complex Objects

This contrast was empty. No area showed significantly greater modulation in this simpler task.

# Haptically-Guided Grasping vs. NoGo

All the nodes forming the bilateral parieto-frontal and temporoinsular networks involved in haptically-guided grasping regardless of whether they are sensitive to object features or not—are revealed by a contrast of grasping vs. the NoGo condition. The obtained significant signal modulations are shown in **Figure 4B**. Although the network of areas has now expanded substantially, and includes bilateral parietal opercular and temporal clusters on the other side of the Sylvian Fissure, bilateral PMv, and on the medial surfaces SMA, pre-SMA, and the nearby

FIGURE 4 | Further contrasts and comparisons showing neural activity associated with haptically-guided grasping. (A) Grasping of complex vs. simple objects: the significant clusters involved left aIPS, rSPG, adSPOC, PMd, and the sensorimotor cortex, whereas in the right hemisphere all the voxels sensitive to object features and involved in related processing (of complex vs. simple objects) were re-recruited, including adSPOC, rSPG, aIPS, aSMG, vPCG, and SI. (B) Grasping vs. NoGo: in addition to all the areas, or their subdivisions, significantly sensitive to object type [see (A)], this contrast also revealed bilateral involvement of SII, PMv, SMA complex, CMC, middle and anterior insular cortices and the neighboring STGs, as well as the thalamus, and caudate nucleus. (C) Grasp execution vs. Grasp planning: this contrast revealed significant involvement in of all the areas from the previous contrast, supplemented by a small contribution from left rMFG, the orbital division of the right inferior frontal gyrus, left EVCs, and the lateral occipital complex, encompassing its primarily visual lateral occipital division (LO), the multimodal tactile and visual division (LOtv), and the motion sensitive area MT+. As in panel (C), these additional areas did not show much sensitivity to haptically processed object features that are of critical importance for haptically-guided grasping. (D) The dorso-ventral gradients of grasp-related activity associated with different kinds of processing: All the areas depicted in magentas represent grasp-relevant features of the unseen targets and belong to the dorso-dorsal (lateral, but also medial) parieto-frontal networks. All the areas illustrated in blues contribute to monitoring of hand pre-shaping or grasp kinematics, and belong to the ventro-dorsal networks. Finally, the areas depicted in violets, including left aIPS, show significant sensitivity to both grasp-relevant target features and the characteristics of the required grasp.

CMA, none of the occipital or temporal regions involved in object processing was identified by this contrast. Importantly, neither the temporo-insular clusters nor, even more surprisingly, the bilateral PMv and SMA complex, showed substantial sensitivity to object type during grasping (see **Figure 4A**).

# Haptically-Guided Grasping vs. Grasp Planning

One of the most surprising outcomes so far has been the conspicuous absence of the LO cortex contribution to the haptically-guided grasping task in any of the direct contrasts between the major studied conditions, including "grasping vs. NoGo" (except for the comparisons of exploration, and grasping, vs. the resting baseline). Indeed, LO was not even involved in the planning of grasp (vs. the resting baseline) either. To shed some more light on this issue, we directly compared the graspexecution phase with the grasp-planning phase. In this contrast, LO contribution has been revealed, in addition to the widespread differences in nearly all the areas mentioned thus far in the context of grasping, and exploration. This outcome is depicted in **Figure 4C**. It must be emphasized that, although this effect is not driven by the LO inhibition during grasp planning—but rather weak, non-significant fluctuations of activity around the resting baseline—its contribution to grasping is marginal at best. After all, consistently with earlier studies (e.g., Króliczak et al., 2007), the signal modulations observed in this area do not depend in any way on object type, and as such they cannot contribute directly to the control of grasping.

# Haptically-Guided Grasp Planning vs. Grasp Execution

The contrast was nearly empty, except for a cluster of weaker inhibition observed bilaterally in the medial frontal cortex, which was not even invoked in a contrast of grasp planning vs. rest.

The anatomical locations of all the major clusters revealed in the contrasts described above, the MNI coordinates, as well as the statistical values of the peak voxels can be found in **Table 1**.

# Dorso-Ventral Gradients of Sensitivity to Object Features and Finger Pre-shaping during Haptically-Guided Grasping

**Figure 4D** shows the results of two critical comparisons involving grasping tasks mentioned above, namely the contrast of grasping vs. reaching (both tasks performed in the presence of complex objects, but requiring completely different movement kinematics), and the contrast of grasping complex vs. simple objects (with the former requiring at least increased processing of axis of elongation, and the actual object length). The difference between **Figures 3D**, **4A**, respectively, and the overlays presented in **Figure 4D** is such that for obtaining the latter effects, the more laborious, but also robust, mixed-effects model, with the random-effects components of variance estimated with FSL's Flame 1 + 2 procedure was used for statistical analyses, hoping that it would also reveal ventral-stream contributions to these tasks. This was not the case. Interestingly, a contrast of grasping complex vs. simple objects profited much more from this TABLE 1 | Major contrasts from the Main Haptic Experiment and the Localizer Scans.


#### MAJOR CONTRASTS FROM THE MAIN HAPTIC EXPERIMENT A. Haptic object exploration vs. rest (Z > 2.3, p = 0.05 cluster corrected) Right pre-Supplementary Motor Area (pre-SMA) 4 4 54 3.16 Right Cingulate Motor Area (CMA) 4 16 31 3.12 Left anterior Intraparietal Sulcus (aIPS) –32 –49 29 3.90 Left anterior Supramarginal Gyrus (aSMG) –56 –36 30 3.15 Left Secondary Somatosensory Cortex (SII) –55 –23 9 3.05 Left dorsal Premotor Cortex (PMd) –26 –11 51 3.22 Left ventral Premotor Cortex (PMv) –57 10 23 3.18 Left rostral Middle Frontal Gyrus (rMFG) –38 39 13 3.60 Posterior Calcarine Sulcus 0 –83 1 3.30 Left Lateral Temporal-Occipital Cortex (TOC) –44 –70 2 2.77 Left posterior Middle Temporal Gyrus (pMTG) –50 –55 –5 2.82 Left Precuneus –3 –79 42 3.09

B. Haptic exploration of complex vs. simple objects (Z > 2.3, p = 0.05


#### C. Grasp planning vs. rest (Z > 2.3, p = 0.05 cluster corrected)


#### D. Haptically-guided grasp execution vs. rest (Z > 2.3, p = 0.05 cluster corrected)

Most of the areas and their coordinates are the same as in A. Haptic exploration vs. rest. Additional regions are listed below.


(Continued)

#### TABLE 1 | Continued


Average peak coordinates (in MNI space) and their peak values (Z statistics) in functional areas (or regions) identified in major contrasts from the main study, and localizer scans.

approach because not only the superior parieto-frontal activity was now revealed in the right hemisphere but, additionally, it showed bilateral contributions from small subdivisions of the SMA complex and CMC. Moreover, it is quite apparent from the inspection of **Figure 4D** that the location of clusters showing sensitivity to object features, and sensitivity to hand preshaping is quite different, more dorsal and ventral, respectively. Yet, there is also a substantial area of overlap. Indeed, as shown by inclusive contrast masking procedure carried out in both directions for the two contrasts, all the voxels and, more importantly, only the voxels in the overlapping regions do show significant sensitivity to both object features and hand preshaping (movements kinematics) during grasp performance. In sum, the more superiorly located the area the greater sensitivity to object features during grasping, whereas the more inferiorly located the area the greater sensitivity to actual finger movement kinematics, rather than objects themselves. Of course, in the overlapping regions there is significant sensitivity both to the critical features of the grasped stimuli and the associated hand actions.

# The Results of ROI Analyses

The selected regions include areas significantly involved in the major tasks alone, such as object exploration—rMFG, PMv, aIPS, and grasp planning—pMTG, or to some extent involved in two tasks, e.g., exploration and grasping—TOC. Moreover, a few distinct functional subdivisions within the lateral temporooccipital cortex were chosen for more theoretical reasons, including the human homolog of motion sensitive area MT+, a subdivision of the lateral occipital cortex sensitive both to tactile and visual processing (LOtv, revealed in our haptic aIPS/LOtv localizer), and finally the more posterior division of LO (pLO), revealed here by its exclusive sensitivity to intact vs. scrambled objects.

In the **rMFG ROI**, a 3 (task) × 2 (object type) ANOVA revealed a main effect of task [F(2, 18) = 9.3, p < 0.01], such that object exploration was associated with significantly stronger activity than both grasp planning, and grasp execution (Bf-p < 0.05 in both cases; whereas the grasp related activity showed only a trend toward being stronger than during grasp planning, Bf-p = 0.09). There was also a main effect of object [F(1, 9) = 6.2, p < 0.05], such that performing any task involving complex objects resulted in significantly higher activity than performing the tasks with simple objects. Finally, the task by object interaction was not significant [F(2, 18) = 1.8, p = 0.19]. The observed pattern of activity is shown in **Figure 5A**.

The **PMv ROI** showed a rather different pattern. Although a 3 × 2 ANOVA revealed a main effect of task [F(2, 18) = 15.9, p < 0.001], such that object exploration was again associated with significantly stronger activity than both grasp planning, and grasp execution (Bf-p < 0.01, and Bf-p < 0.05, respectively), the grasp related activity was also significantly stronger than during grasp planning (Bf-p < 0.05). As above, there was a main effect of object [F(1, 9) = 10.9, p < 0.01], such that performing tasks involving complex objects resulted in significantly stronger activity than performing the tasks involving simple objects. However, the task by object interaction was also

significant [F(2, 18) = 14.9, p < 0.001], and clearly indicated that even though exploration, and grasping of complex (vs. simple) objects did result in greater activity (Bf-p < 0.05 in both cases), this pattern was inverted for grasp planning, but the difference was not significant (Bf-p = 0.57). These results are shown in **Figure 5B**.

A somewhat similar pattern of activity emerged in the **aIPS ROI**. As in PMv, there was a main effect of task [F(2, 18) = 7.8, p < 0.01], yet object exploration was linked to significantly stronger activity only when compared to grasp planning (Bf-p < 0.05), but not grasp execution (Bf-p = 0.33), with the grasprelated activity also being significantly stronger than during its planning (Bf-p < 0.05). The familiar main effect of object was revealed again [F(1, 9) = 11.0, p < 0.01], with tasks involving complex objects associated with significantly higher activity than tasks involving simple objects. Finally, the task by object interaction was also significant [F(2, 18) = 11.0, p < 0.001], in which only exploration, and grasping of complex objects was linked to significantly greater activity than the same tasks performed with simple objects (Bf-p < 0.05, and Bf-p < 0.01, respectively), whereas there was no object related difference whatsoever in the activity associated with grasp planning (Bf-p = 0.63). The observed effects are shown in **Figure 5C**.

The **pMTG ROI** activity pattern was among the least expected. Neither a main effect of task [F(2, 18) = 0.39, p = 0.69], nor object type [F(1, 9) = 0.92, p = 0.36] was significant. So was not the task by object interaction [F(2, 18) = 1.79, p = 0.19]. This result is displayed in **Figure 5D**.

This means that only when tested in isolation, the grasp planning activity vs. rest was significantly different from baseline.

In the **TOC ROI** the pattern of activity was quite similar to the one observed in PMv. A main effect of task [F(2, 18) = 10.8, p < 0.001] was observed, in which object exploration was associated with significantly stronger activity than both grasp planning, and grasp execution (Bf-p < 0.05 in both cases), but despite a trend, grasp execution did not engage this area more than grasp planning (Bf-p = 0.11). A main effect of object [F(1, 9) = 10.9, p < 0.01] was also significant and, again, performing tasks involving complex objects was associated with higher activity than performing the tasks involving simple objects. Finally, the task by object interaction was also significant [F(2, 18) = 6.9, p < 0.01], but now only the exploration of complex objects resulted in significantly higher activity than exploration of simple objects (Bf-p < 0.05), with no difference between objects whatsoever for grasp planning (Bf-p = 1.0), and even lack of substantial trend for grasping complex vs. simple objects (Bf-p = 0.25). The results are shown in **Figure 5E**.

In the more posterior **MT+ ROI** the pattern of activity resembled that of aIPS. The analysis showed a main effect of task [F(2, 18) = 10.0, p < 0.001] such that only object exploration, and grasp execution, was associated with significantly stronger activity when compared to grasp planning (Bf-p < 0.05 in both cases), whereas exploration and grasp execution activity did not differ (Bf-p = 0.19). The familiar main effect of object was revealed as well [F(1, 9) = 10.5, p < 0.01], where tasks involving complex objects were associated with significantly stronger activity than tasks involving simple objects. Finally, there was also a significant task by object interaction [F(2, 18) = 4.4, p < 0.05], yet only exploration of complex objects was linked to significantly greater activity than exploration of simple objects (Bf-p = 0.05), whereas the effect was absent for grasp planning (Bf-p = 1.0), and almost non-existent for grasp execution (Bfp L= 0.24). These changes in activity are shown in **Figure 5F**.

The pattern was nearly the same for the **LOtv ROI**. A main effect of task was again significant [F(2, 18) = 11.5, p < 0.001], wherein object exploration, and grasp execution, was associated with significantly stronger activity than grasp planning (Bf-p < 0.05 in both cases), but exploration also invoked stronger activity than grasp execution (Bf-p = 0.05). The main effect of object was revealed again [F(1, 9) = 15.1, p < 0.01], wherein tasks involving complex objects invoked significantly higher activity than simple objects. Finally, a significant task by object interaction [F(2, 18) = 10.1, p < 0.001] was such that both exploration and grasping of complex objects was linked to significantly stronger activity than exploration and grasping of simple objects (Bf-p = 0.05 in both cases), whereas the effect was absent for grasp planning (Bf-p = 1.0). The observed pattern of activity is shown in **Figure 5G**.

Finally, in the **pLO ROI**, similarly to pMTG ROI, neither a main effect of task [F(2, 18) = 1.4, p = 0.27], nor object type [F(1, 9) = 0.5, p = 0.5] was significant. Similarly, there was no significant task by object interaction [F(2, 18) = 0.86, p = 0.44]. The result can be seen in **Figure 5H**.

# DISCUSSION

In this study, to our knowledge, for the first time, we examined the patterns of neural activity associated with grasping of complex objects that do not have any prior visual representations in the brain. To this end, participants first explored the novel targets haptically in order to determine their shapes and orientations, planned grasping these objects a couple of seconds later, and following a short variable interval, executed the preplanned grasps.

The activity associated with haptic exploration of the targets included the fronto-parietal, temporo-occipital, and insular cortices (Binkofski et al., 1999a; Deibert et al., 1999; James et al., 2002; Sathian, 2005). Interestingly, given the ultimate goal of the exploratory phase, i.e., preparation for later grasping, the engaged networks comprised of all the areas commonly associated with the praxis representation network or PRN (e.g., Frey, 2008; Króliczak and Frey, 2009; see also Króliczak et al., 2008; cf. Snow et al., 2015). Yet, the observed signal changes were less left lateralized and the clusters devoted significantly to the processing of object shape were lateralized exclusively to the right hemisphere. Indeed, they were restricted primarily to the anterior and rostral divisions of the posterior parietal cortex.

To our surprise, although the areas involved in grasp planning belonged largely to the left hemisphere, there was almost no overlap with those involved in object exploration. Namely, the temporal clusters were more anterior, and the single intraparietal cluster was more posterior and, even more unexpectedly, none of them showed any sensitivity to object features (cf. Valyear et al., 2007; Glover et al., 2012; Singhal et al., 2013).

Even though the networks re-recruited for grasp execution were similar to those involved in object exploration, only the dorso-dorsal (Rizzolatti and Matelli, 2003) and predominantly bilateral parieto-frontal networks showed clear modulations depending on object complexity (cf. Binkofski et al., 1999a; Reed et al., 2005). These networks included nearly all the right-hemisphere voxels that revealed object sensitivity in the exploration phase. The region of interest analyses further corroborated these results, demonstrating—as in the study by Króliczak et al. (2007)—no task/object selectivity during grasp performance in areas typically associated with visual perceptual processing of shape and size, or object affordances, such as the lateral occipital or TOC (e.g., James et al., 2003; Vingerhoets, 2008).

Because neither the visual nor multimodal perceptual, ventralstream regions showed any pronounced sensitivity to object shape and orientation in the separate phases of the paradigm used here, similarly to visually-guided actions (e.g., Culham et al., 2003; Króliczak et al., 2008; for a recent review see Gallivan and Culham, 2015) their contribution to the haptic control of grasping is marginal at best. Analogous conclusions can be also drawn about the involvement of the insular, and secondary somatosensory cortices. While greater engagement of selected ventral ROIs was observed during processing of complex shapes, most of the time these effects were quite weak and task independent (i.e., required collapsing across the exploratory and grasp phases). Instead, this study demonstrates that the critical haptic processing of object features for future manual actions takes place primarily in the right superior parietal lobule, and extends via aIPS to aSMG of the inferior parietal lobule (cf. Binkofski et al., 1999a). All these areas get re-recruited for grasp performance, and their input is dispensed bilaterally, with the inclusion of the contralateral left aIPS (and to some extent the interconnected left PMv ROI), and utilized by the parieto-frontal networks for haptic grasp guidance.

As such, these results suggest that a substantial portion of what has been taken as evidence for visual perceptual processing in the parietal lobe can reflect primarily a conversion from visual—object-related processing of shape for action—to haptic codes for the on-line control of the grasping hand (cf. Cohen and Andersen, 2002; see also Culham et al., 2006). A clear support for this proposal comes from our observation of the dorso-ventral gradients of haptic sensitivity to object features and finger pre-shaping, respectively, during haptically guided grasping. The area most commonly studied in the context of grasp performance, namely aIPS, is located somewhere in the middle of this gradient, and shows significant sensitivity to both object characteristics and the required kinematics (even though the targets were never seen before).

# Haptic Object Exploration Involves the Use of Praxis Skills, and Visual Encoding

In addition to the major nodes of the PRN, the activity associated with haptic exploration of the novel objects involved both the medial and lateral occipital cortices, and the more anterior, mainly lateral temporo-occipital regions. Such a pattern of results is not surprising given that, with their exploratory finger movements, participants were to look for object features that were most diagnostic (i.e., characteristic/important) for later performance of the grasping task, and these features could arguably be shape and orientation. Moreover, the use of such skills as executing initial exploratory "grasp-type" enclosure on an unknown target, dynamic molding to and/or following of its contours, and finding the axis of elongation does not only permit efficient extraction of an object form but is typically associated with the visual encoding bias (Lederman et al., 1996). Indeed, as shown by Lederman et al., such a bias is even more likely when variations in shape are more perceptually accessible than any other properties of the studied objects. This was definitely the case for the novel stimuli used in our study. Thus, the simultaneous engagement of PRN and visual regions during haptic exploration is consistent with the use of the grasp-like, and contour-tracking exploratory strategies, and the closely associated inclination for visual encoding.

It was rather unexpected, though, that this kind of processing would not result in more wide-spread differential signal changes reflecting object complexity (cf. Valyear et al., 2006 for the visual modality). Yet, the ultimate goal of the exploratory movements was a preparation for grasping, not object discrimination (or recognition), and encoding of all the details related to object shape was not even necessary. Indeed, the prerequisite of skilled grasping in this paradigm was finding the orientation of the object—basically its axis of elongation, and then encoding its extent (length) along this particular dimension. Similarly, the thorough coding of perceptual properties of objects, and their relations to other targets, could have been disregarded (cf. Hesse et al., 2008).

The less focus on haptic processing of details, the less differential object-related activity would be expected in the left hemisphere, particularly in the ventral stream. Conversely, the crude or more global haptic processing of shape (e.g., finding only appropriate "grasp points") is expected to involve the right superior parietal lobule (Dijkerman and de Haan, 2007; e.g., Leisman and Melillo, 2007; see also Milner and Goodale, 1995). Consistent with this notion is an observation that only a small LOtv cluster, as well the more posterior subdivision of the left TOC, namely MT+, exhibited some selectivity to object shape in the ROI analyses (For LOtv it was found both during exploration and grasping, and for MT+ only during object exploration, resulting in a similar effect in the whole TOC ROI for object exploration). Yet, such sensitivity was not revealed by any of the whole brain contrasts from the main study. Moreover, this kind of response selectivity could be also accounted for by some tuning of this area to more complex patterns of exploratory finger movements, rather than shape processing per se (cf. Amedi et al., 2002; see also Lederman et al., 1996).

In sum, the putative visual encoding bias in haptic exploration of object contour and extent has been insufficient for generating reliable and wide spread object-shape sensitivity in the ventral processing stream. Although the results of ROI analyses indicate that object selectivity may nevertheless be found in subdivisions of the lateral and ventral temporo-occipital cortex, it can be associated with multimodal interactions of shape. That is, it can be linked to moment-to-moment finger postures, or even to monitoring of the more complex finger movements during contour tracking. Only the more anterior, and often rostral divisions of the right posterior parietal lobe have shown indisputable haptic shape/orientation sensitivity (cf. Binkofski et al., 1999a; James et al., 2002; Reed et al., 2005; Dijkerman and de Haan, 2007), and these regions most likely provide the critical input for the execution of later grasping.

# Haptically-Guided Grasp Planning Does Not Invoke Regions Sensitive to Visual Object Shape

To our surprise, unlike in visually-guided delayed grasping, where specialized dorsal-stream areas contribute both to planning of action and maintenance of its goals (e.g., Singhal et al., 2013), none of the dorso-dorsal nor ventro-dorsal networks (Rizzolatti and Matelli, 2003) were re-recruited here during the planning phase. Yet, consistent with the idea that in the ventral stream of information flow the inputs pass through progressively more complex stages of processing (resulting in global object representations linked to memory), during the grasp planning phase we observed significant recruitment of the left lateral and ventral temporal cortices anterior to the TOC region engaged in object exploration. This outcome is also in agreement with a notion that the lateral occipito-temporal cortices appear to be less task-specialized and may play associative functional roles (e.g., Monaco et al., 2014), particularly during action planning rather that its execution (for a review, see Króliczak et al., 2012). Even more importantly, though, grasp planning under haptic guidance has been also associated with sustained bilateral signal in EVCs (particularly with relatively early visual cortex signal modulations; see Singhal et al., 2013). Although this effect is consistent with the employment of the visual encoding bias (Lederman et al., 1996), it must reflect some pretty basic "visualization skills" because only the more posterior medial and ventral occipital areas have shown any overlap with those involved in object exploration.

It is of particular note that neither the areas with the sustained, nor the ones with newly induced significant signal changes showed any object shape selectivity (cf. Valyear et al., 2007; Glover et al., 2012; Singhal et al., 2013), a finding that was also corroborated by the less stringent ROI analyses. Indeed, with the exception of left pMTG, the observed signal changes were characteristically very small (<0.1% of BOLD signal change) and, oftentimes, showed activity patterns going in the direction opposite to neural responses typically observed for complex vs. simple objects. Thus, even though the observed signal modulations may reflect some preparatory set activity (cf. Connolly et al., 2002; Valyear and Frey, 2015), it is quite unlikely that its role is to uniquely link the parieto-frontal grasp networks with the temporo-occipital visual/multimodal areas (cf. Borra et al., 2008; see also Króliczak et al., 2008). This activity may nevertheless play an important role in the later re-recruitment of the parieto-frontal networks for the proper grasp type and hand orientation.

# Haptically-Guided Grasping is Associated with fMRI Activity in Dorsal but Not Ventral Stream Brain Areas

Counter to grasp planning, the actual execution of the grasp based on the haptic information obtained a few seconds before was accompanied by re-recruitment of areas associated with haptic object encoding, and extensive bilateral engagement of the dorsal, parieto-frontal networks. While in the haptic domain this outcome does not necessarily indicate a dissociation between perceptual- and action-related processing (cf. Binkofski et al., 1999b; Reed et al., 2005; see also Whitwell et al., 2014; and Whitwell et al., 2015), it is inconsistent with the notion that the control of manual actions requires both dorsal and ventral stream contributions (e.g., as in delayed visually-guided actions; Singhal et al., 2013).

It is worth emphasizing that the ventral-stream contribution to grasping was likely to occur (when compared to resting baseline) given the observed engagement of EVCs during haptic exploration. Yet, while pointing to the use of the representations based on the preceding visual encoding bias (Lederman et al., 1996; cf. Amedi et al., 2002), neither the EVCs nor the more anterior temporal regions showed any object shape sensitivity. Therefore, their contribution to grasp guidance could have been only of a very general nature. Indeed, this study revealed fast and substantial decay of fMRI activity in temporo-occipital regions when object exploration was complete. Notably, the rerecruitment of their more specialized subdivisions (i.e., LOtv, MT+) was revealed neither in the planning nor the execution phase in the whole brain analyses. Although, some sensitivity to object features was identified in the left LOtv ROI during graspexecution, it more likely reflects the control of the on-going finger preshaping and the ultimate grasp enclosure on the shaped object contour, rather than the overall perceived object shape.

The lack of substantial ventral contribution to hapticallyguided grasping is not that surprising because even in studies on visually-guided actions it is quite controversial whether or not the ventrally encoded perceptual representations are used for the guidance of the grasping hand (e.g., Cavina-Pratesi et al., 2007; Króliczak et al., 2007; cf. Króliczak et al., 2008). Indeed, a large body of evidence suggests that, at least in the case of hand movements directed at simple targets, the remembered (ventrally-encoded) information on object shape, size and/or its relative location seem to play a marginal role when the planning and/or execution of actions takes place in full vision (e.g., Monaco et al., 2010, 2015; see also Culham et al., 2003; Króliczak et al., 2006; cf. Hesse and Franz, 2009; Valyear and Frey, 2015; for a review see Goodale et al., 2005; Króliczak et al., 2012). Although such on-line action guidance can be typically handled almost exclusively by the dorsal, visuo-motor processing stream (Goodale and Milner, 1992; Milner and Goodale, 2008; see also Goodale et al., 2008; Goodale, 2014), when grasp planning and/or its execution is only briefly delayed, or vision is fairly limited, the visuo-motor system can hardly operate without such stored visual input (e.g., Goodale et al., 1994; Milner et al., 2001; Westwood and Goodale, 2003; see also Monaco et al., 2010; see also Whitwell et al., 2014, 2015). For the same reasons, when substantially longer delays are introduced after object viewing the re-recruitment of areas in the visual perceptual stream becomes even more essential for the performance of grasping actions (Singhal et al., 2013). Yet, whatever mechanisms are involved in the guidance of the grasping hand based on previous visual input, they are clearly less relevant when grasp performance is based entirely on the just acquired, and transiently stored, haptic input.

Of utmost importance is the observation that the re-recruited lateral and medial dorso-dorsal (Rizzolatti and Matelli, 2003) parieto-frontal networks of the left hemisphere did not show any sensitivity to object features during earlier phases. Yet, during grasp performance nearly all their critical nodes, such as adSPOC, rSPG, PMd, SMA, and CMC already operated on such representations. Thus, although one of the roles of the dorso-dorsal stream might be the provision of inputs for the comparison of the somatosensory/proprioceptive feedback with a forward model of the programmed grasp (cf. Makoshi et al., 2011; Singhal et al., 2013), in our opinion such a role cannot be effectively fulfilled in the absence of object coding performed elsewhere. This hypothesis is based on our findings that the source of object sensitivity based exclusively on haptic information was in fact located predominantly in the anterior right PPC. Yet, regardless of the origin of this sensitivity, all the aforementioned dorso-dorsal areas have been shown capable of processing the most critical object dimensions, and providing or even transforming their input to action codes for the ventro-dorsal areas that control the on-going grasp movement kinematics.

Our outcomes clearly demonstrate that the dorso-dorsal parietal and frontal areas of the left hemisphere show greater sensitivity to object features, whereas the ventro-dorsal parietal areas are more sensitive to the actual grasp kinematics. Of course, left aIPS is capable of processing relevant target features and selecting and/or monitoring action kinematics both under visual (Króliczak et al., 2008) and haptic guidance. The overall gradients of activity within the right hemisphere, including aSMG were quite similar, including the medial dorso-dorsal divisions and their projections to the frontal cortex (Rizzolatti and Matelli, 2003). Yet, judging by the overall distribution of this activity, it seems that the right hemisphere parietal areas might be more involved in the comparisons of the somatosensory and proprioceptive feedback with predicted movement plans. It should be re-iterated, though, that the haptic sensitivity to object features has its source here, and during haptically-guided grasping is not only retained but also extends to functional areas that were not involved in—or perhaps were even actively suppressed during—object exploration (cf. Binkofski et al., 1999a; Reed et al., 2005).

Given how efficiently the localized, right-hemisphere activity can be distributed across the bilateral parieto-frontal networks, the outcomes of our study also shed a new light on the mechanisms involved in ego-centric coordinate transformations taking place in the parietal lobe (Cohen and Andersen, 2002; Milner and Goodale, 2008). Indeed, instead of representing purely visual transformations between different frames of reference, the parietal lobe activity can often reflect primarily a conversion from visual, haptic and/or modality-independent object coding to (egocentric) haptic and even kinesthetic codes for the control of the grasping hand (cf. Whitwell et al., 2014; Leoné et al., 2015; Whitwell et al., 2015).

# Limitations of the Study

One of the potential limitations is the lack of a visual control task. Yet, haptic exploration typically takes time, while visual exploration would be effective almost instantaneously. Moreover, if we used a control task that would require later object discrimination (or even its recognition), perhaps more areas sensitive to object complexity would be revealed, especially in the ventral stream. In terms of timing, if the exploration phase was shorter, which would also make it harder, and subsequent phases were delayed in time, more clear-cut differences in activity could emerge for complex vs. simple objects. It is also of note that our participants were required to explore the circular disks for the entire 5 s, which could in some areas result in steady increases of the signal, leading eventually to its saturation and/or ceiling effects. Yet, purely perceptual haptic areas should respond less for circular disks due to adaptation following repeated movements over the same shape. Furthermore, perhaps the outcomes from the planning phase would be more intuitive if grasp planning was separated from the exploration phase by a much longer interval. Finally, the paradigm could profit a lot from a clear distinction between grasping and reaching trials made up front. That is, if participants knew right from the beginning of a trial that they were to reach, and thought about moving the clenched fist toward the target instead, the outcome of the comparison with grasping could likely be even more informative.

# CONCLUSIONS

While the importance of vision for action planning and execution has long been recognized (Helmholtz, 1867/1962), the role of the haptic sense for critical daily interactions with objects has received considerably less attention. While this gap has been recently narrowed (Dijkerman and de Haan, 2007) and more ecologically valid paradigms are being used and considered (Stone and Gonzalez, 2015), there is still a substantial work to be done in the domain of tactile processing and haptic perception (cf. Snow et al., 2015), as well as in the area of haptically-guided actions. Although this study did not take into account common objects, all the phases that are essential for such interactions were investigated here. We found that the most critical aspects of task performance are controlled by the dorsal-stream regions, with much greater—in fact exclusive contribution of the right hemisphere to the haptic processing of object shape (or its exact graspable dimension), and the bilateral involvement of the parieto-frontal networks, including left aIPS, in the control of the haptically-guided grasping. Furthermore, two different kinds of signal processing for grasp performance have been associated with the dorso-dorsal vs. ventro-dorsal parieto-frontal networks (thus forming a dorso-ventral gradient), with the emphasis on representing grasp-relevant features of the unseen targets, and monitoring of grasp kinematics, respectively. Of course, intermediate areas such as aIPS show sensitivity to both object shape and the required grip kinematics. Finally, these outcomes suggest that the transformations for action guidance in the parietal lobe reflect primarily re-coding of object-related into effector-related representations, regardless of the sensory input modality, and independent of the ventral stream. Although there is substantial evidence that in more cognitive tasks haptic memory is supported by dedicated ventral somatosensoryinsular-(pre)frontal cortex pathways (e.g., Burton and Sinclair, 2000; Pasternak and Greenlee, 2005), the role of these and other ventrally-located regions in tasks and paradigms similar to ours will likely be limited to supportive or associative functions (cf. Bidula and Kroliczak, 2015).

# AUTHOR CONTRIBUTIONS

This project was conceptualized by GK and MM. Data was collected by GK, analyzed by GK, MM, AK, and interpreted by all the authors. The manuscript was written by GK, MM, and AK.

# ACKNOWLEDGMENTS

The acquisition of data for this project was possible thanks to the Telemedicine and Advanced Technology Research Center (TATRC)-subsidized Pilot-study Funds awarded to GK by the Lewis Center for NeuroImaging at the University of Oregon. GK was then supported by the "Brain, Biology, and Machine Initiative" (BBMI) Research Fellowship in the Department of Psychology at the University of Oregon. We thank Scott H. Frey for assistance with designing the study, and Sam Linton for help with data acquisition. Additional localizers were acquired with help from the Polish National Science Center (Narodowe Centrum Nauki, NCN) grant Maestro 2011/02/A/HS6/00174 to

# REFERENCES


GK. We thank Piotr Styrkowiec for help with the collection of these data and their analyses. The equipment used during data acquisition was funded by the Ministry of Science and Higher Education (Ministerstwo Nauki i Szkolnictwa Wyzszego, ˙ MNiSW) grant 6168/IA/128/2012 to GK, whereas that for data processing, and their analyses was also funded by the NCN grant Maestro 2011/02/A/HS6/00174 to GK. During the preparation of this manuscript MM, AK, and GK were supported by the Maestro grant. We would also like to thank Stephen R. Arnott for recording and editing all the auditory cues used in this study.

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fnhum. 2015.00691

Supplementary Figure 1 | All the objects, their orders, and orientations used in the main experiment. Note, that there were five runs for each subject and one of the sets was repeated.

Supplementary Figure 2 | Neural activity associated with haptic exploration of simple vs. complex objects. This contrast revealed modulations in a widespread network of areas, resembling the default mode network.


reflect the plan to grasp? Neuroimage 36(Suppl. 2), T94–T108. doi: 10.1016/j.neuroimage.2007.03.031


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Marangon, Kubiak and Króliczak. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.