# REWARD- AND AVERSION-RELATED PROCESSING IN THE BRAIN: TRANSLATIONAL EVIDENCE FOR SEPARATE AND SHARED CIRCUITS

EDITED BY: Dave J. Hayes, Georg Northoff and Andrew J. Greenshaw PUBLISHED IN: Frontiers in Systems Neuroscience

#### *Frontiers Copyright Statement*

*© Copyright 2007-2016 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

> *All copyright, and all rights therein, are protected by national and international copyright laws.*

*The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88919-836-8 DOI 10.3389/978-2-88919-836-8

## About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

## Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

## Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

## What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# **REWARD- AND AVERSION-RELATED PROCESSING IN THE BRAIN: TRANSLATIONAL EVIDENCE FOR SEPARATE AND SHARED CIRCUITS**

Topic Editors: **Dave J. Hayes,** Union College, USA **Georg Northoff,** University of Ottawa, Canada; Taipei Medical University, Taiwan and Hangzhou Normal University, China **Andrew J. Greenshaw,** University of Alberta, Canada

Correlation of resting state oscillation power, seen with near-infrared spectroscopy, and harm avoidance/novelty seeking in eyes open and eyes closed conditions.

Adapted from: Nakao T, Matsumoto T, Shimizu D, Morita M, Yoshimura S, Northoff G, Morinobu S, Okamoto Y and Yamawaki S (2013) Resting state low-frequency fluctuations in prefrontal cortex reflect degrees of harm avoidance and novelty seeking: an exploratory NIRS study. Front. Syst. Neurosci. 7:115. doi: 10.3389/fnsys.2013.00115

Affective brain circuits underpin our moods and emotions. Appetitive and aversive stimuli from our exteroceptive and interoceptive worlds play a key role in the activity of these circuits, but we still do not know precisely how to characterize these so-called reward-related and aversion-related systems. Moreover, we do we yet understand how they interact anatomically or functionally.

The aim of the current project was to gather some translational evidence to help clarify the role of such circuits. A multi-dimensional problem in its own right, the book contains 14 works from authors exploring these questions at many levels, from the cellular to the cognitive-behavioural, and from both experimental and conceptual viewpoints. The editorial which introduces the book provides brief summaries of each perspective (Hayes, Northoff, Greenshaw, 2015).

While questions of how to accurately define affect- and emotion-related concepts at the psychological level are far from answered, here we have attempted to provide some insight into the brain-based underpinnings of such processes. The near future will undoubtedly involve making new inroads and will require the joint efforts of behavioural, brain-based, and philosophical perspectives to do so.

**Citation:** Hayes, D. J., Northoff, G., Greenshaw, A. J., eds. (2016). Reward- and aversion-related processing in the brain: translational evidence for separate and shared circuits. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-836-8

# Table of Contents


and Andrew J. Greenshaw

*117 Altered frontocingulate activation during aversive interoceptive processing in young adults transitioning to problem stimulant use*

Jennifer L. Stewart, Jason M. Parnass, April C. May, Paul W. Davenport and Martin P. Paulus

*132 Mesocorticolimbic monoamine correlates of methamphetamine sensitization and motivation*

Kevin D. Lominac, Courtney L. McKenna, Lisa M. Schwartz, Paige N. Ruiz, Melissa G. Wroten, Bailey W. Miller, John J. Holloway, Katherine O. Travis, Ganesh Rajasekar, Dan Maliniak, Andrew B. Thompson, Lawrence E. Urman, Tamara J. Phillips and Karen K. Szumlinski

*151 Impact of appetitive and aversive outcomes on brain responses: linking the animal and human literatures*

Gregory B. Bissonette, Ronny N. Gentry, Srikanth Padmala, Luiz Pessoa and Matthew R. Roesch

*166 Translational studies of goal-directed action as a framework for classifying deficits across psychiatric disorders*

Kristi R. Griffiths, Richard W. Morris and Bernard W. Balleine

## Editorial: Reward- and aversion-related processing in the brain: translational evidence for separate and shared circuits

Dave J. Hayes <sup>1</sup> \*, Georg Northoff 2, 3, 4 and Andrew J. Greenshaw<sup>5</sup>

<sup>1</sup> Brain, Imaging and Behaviour – Systems Neuroscience, Toronto Western Research Institute, Toronto Western Hospital, University Health Network, University of Toronto, Toronto, ON, Canada, <sup>2</sup> Mind, Brain Imaging and Neuroethics Research Unit, Institute of Mental Health Research, University of Ottawa, Ottawa, ON, Canada, <sup>3</sup> Brain and Consciousness Research Center, Graduate Institute of Humanities in Medicine, Shuang Ho Hospital, Taipei Medical University, Taipei, Taiwan, <sup>4</sup> Centre for Cognition and Brain Disorders, Hangzhou Normal University, Hangzhou, China, <sup>5</sup> Department of Psychiatry, University of Alberta, Edmonton, AB, Canada

Keywords: affective neuroscience, affective disorders, appetitive, aversive, reward, punishment, reinforcement, translational research

The dynamic evaluation of experience is existentially essential. Assigning value to events and objects drives neural development and plasticity and impacts our changing perceptual interpretations of the world and future behaviors (Nelson et al., 2014). The affective foundation of behavior provides more than a mere phenomenological "coloring" of experience. In fact, affect may be an inseparable component of sensation and cognition instead of an oft-considered byproduct (Inzlicht et al., 2015), and translational neuroanatomical evidence suggests that the major brain areas and tracts involved appear largely conserved across species (Panksepp, 2011).

Less clear are the neural underpinnings of valuative processing which give rise to positive and negative affective experience, appetitive/aversive encoding, reward/punishment-related reinforced behaviors, and feelings/emotions. There are many outstanding questions in this field—in addition to contention about precise definitions of terms such as emotion (Izard, 2009; Madan, 2013). Are appetitive and aversive stimuli encoded in similar brain areas? If so, do they share neural circuits and mechanisms? Do they function independently, in parallel, or is their cross-talk more complicated than this?

In this Research Topic, a number of authors have explored themes related to these fundamental questions at very different levels. Chris Madan uses a broad psychological-conceptual perspective with his presentation of the SIMON framework, which considers the interplay between the constructs of affect, reward, and motivation (Madan, 2013). This interplay could help contextualize findings showing that prior exposure to unpleasant images, inducing negative affect, can reduce reward-related responding in a reaction time task, even when motivation to perform is high. This framework also underscores how narrowly-focused experimental designs can advance our understanding of a given concept while also hindering a full appreciation of its real-world relevance.

Cross-conceptual thinking also helps elucidate the context-dependence of affective experience. Stimuli that follow painful events, and signal relief, can share neurophysiological characteristics with rewards but be reported as unpleasant. Andreatta et al. (2013) showed that the prediction of a painful stimulus differentially modulated a person's physiological output and behavioral reports. Both predictable and unpredictable conditioned stimuli following a painful shock acquired implicitly positive valences (i.e., skin responses consistent with relief), but only the predictable stimulus was reported as pleasant, while the unpredictable stimulus was said to be highly unpleasant. This speaks to the subjectivity and malleability of pain experience, and also to the context-dependence and interplay of affect, value and motivation alluded to by Madan.

#### Edited and reviewed by:

Maria V. Sanchez-Vives, ICREA-Institut d'Investigacions Biomèdiques August Pi i Sunyer, Spain

#### \*Correspondence:

Dave J. Hayes dave.hayes@neuroscientist.ca

Received: 06 August 2015 Accepted: 12 October 2015 Published: 29 October 2015

#### Citation:

Hayes DJ, Northoff G and Greenshaw AJ (2015) Editorial: Reward- and aversion-related processing in the brain: translational evidence for separate and shared circuits. Front. Syst. Neurosci. 9:147. doi: 10.3389/fnsys.2015.00147

Authors here have also explored biological mechanisms associated with valuative processing in simpler animals. Sinakevitch et al. (2013) looked at octopamine receptor, AmOA1, distribution in honey bee and fruit fly neurons, as octopaminecontaining neurons (a homolog of dopamine) are involved in reinforcement and neural plasticity. In clever cross-species experiments, they revealed similar expression patterns and highlighted the importance of octopamine on the modulation of local GABAergic interneurons, which could help to clarify the mechanisms underlying food-odor reinforcement. Ducrot et al. (2013) underscored the involvement of glutamatergic AMPA and NMDA receptors in electrical brain self-stimulation reinforced behavior in rats. Blocking AMPA receptor function in the anterior ventral tegmental area decreased appetitive responding, perhaps related to reduced excitatory input to dopaminergic cells, while NMDA receptor blockade in more posterior areas increased appetitive responding, which likely reflects GABAergic disinhibition. Hayes' (2015) work echoes that of Sinakevitch and Ducrot by proposing that GABA-containing cells play a central role in valuative processing and suggests that longand short-range GABAergic circuitry likely contributes to both the integration/cross-talk and differentiation of appetitive and aversive signals.

Human neuroimaging studies asked questions about emotional responsivity, behavioral control, and substance abuse at the whole-brain level. Lee et al. (2013) showed that a vasopressin V1a receptor antagonist could reverse the effects of vasopressin-induced amygdalar BOLD deactivations associated with the presentation of aversive faces. The antagonist also resulted in reduced activation to angry faces in the right temporoparietal junction, precuneus, putamen and medial prefrontal cortex. Nakao et al. (2013) used near-infrared spectroscopy to show that resting state signals in the dorsal portion of the medial prefrontal cortex (which contains the area identified by Lee et al. above) are negatively correlated with harm avoidance (a personality trait characterized by increases in aversive states such as worrying and pessimism), while novelty seeking (a trait reflecting exploratory behavior, impulsivity, and increased substance abuse risk) was reflected in a more ventral area. Brown et al.'s (2015a,b) behavioral-fMRI findings suggest that one must be careful not to conflate high-risk behavior with impulsivity, even if the two are behaviorally correlated, as brain responses associated with each do not largely overlap. Young adults who reported high-risk behaviors showed reduced activations during response inhibition in right orbitofrontal cortex and ventromedial prefrontal cortex, while impulsive people showed reduced activity in right posterior orbitofrontal, dorsomedial prefrontal, and perigenual anterior cingulate cortices. This group also showed increased lateral prefrontal cortex activation for aversive NoGo vs. aversive Go

## REFERENCES

Andreatta, M., Mühlberger, A., Glotzbach-Schoon, E., and Pauli, P. (2013). Pain predictability reverses valence ratings of a reliefassociated stimulus. Front. Syst. Neurosci. 7:53. doi: 10.3389/fnsys.2013. 00053

trials, irrespective of impulsivity or high-risk scores. Adolescents, however, showed very similar BOLD correlations with riskand impulsivity-related scores although greater activation was noted in the lateral prefrontal cortices for neutral vs. aversive distractors—suggesting some differences in emotional-motor processing as a function of age.

Stewart et al. (2013) compared those with problem (n = 18) or stopped (n = 15) drug use and healthy controls in an fMRI study combining a two-choice affective prediction task with interoceptive challenges. Blunted frontocingulate activations during aversive interoceptive stimuli, and increased inferior frontal gyrus activation during punished feedback trials, in those with problem drug use may reflect goal-directed impairments in the face of negative external and internal challenges. Lominac et al. (2014) looked at stimulant-induced neural changes in biobehavioral experiments in mice. They showed that low subchronic doses of methamphetamine are capable of inducing changes within the mesocorticolimbic system, and that preexisting differences in accumbens dopaminergic signaling—as seen in mice bred for high vs. low consumption—may predict a resistance to addiction.

Two final papers provide a more holistic view by reviewing the literature on valuative processing from a basic and psychiatric translational perspective. Bissonette et al. (2014) piece together the few animal and human studies employing both appetitive and aversive stimuli to help distinguish value- and salience-related neural mechanisms, finding a cross-species network of cortical (e.g., orbitofrontal to parietal) and subcortical (e.g., ventral tegmental area to substantia nigra pars compacta) regions which show preferences for each. Griffiths et al. (2014) used a similar translational approach, but instead considered that dysfunctional value-related decision making circuits may be the lynchpin to common psychiatric symptoms. They raise concerns about considering "decision-making" as a unified concept, and suggest that the strongest findings are from studies with translatable processes, such as those involving associative learning and goaldirected action tasks.

Advancements about "reward" and "fear" circuits, which have dominated the literature, have gradually paved the way for a more nuanced conceptualization of valuation in the brain. The mesocorticolimbic system can no longer be categorized as a "reward" or a "dopaminergic" circuit, nor can the amygdala be deemed the "fear center." Importantly, identification of the interplay between positive and negative affective brain circuits, noted here and elsewhere (Vickery et al., 2011; Hayes et al., 2014; Lindquist et al., 2015), is highlighting the complexity of such networks. The primary goal of this Research Topic was to help identify some of the strengths of this approach and to help generate new hypotheses about how to better apprehend affective circuits.


clinical high-risk behaviour in adolescents. Front. Syst. Neurosci. 9:124. doi: 10.3389/fnsys.2015.00124


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Hayes, Northoff and Greenshaw. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

## Toward a common theory for learning from reward, affect, and motivation: the SIMON framework

### *Christopher R. Madan\**

*Department of Psychology, University of Alberta, Edmonton, AB, Canada*

#### *Edited by:*

*Dave J. Hayes, University of Toronto, Canada*

#### *Reviewed by:*

*Raúl G. Paredes, National University of Mexico, Mexico Nicole El Massioui, National Center of Scientific Research, France*

#### *\*Correspondence:*

*Christopher R. Madan, Department of Psychology, University of Alberta, P-217 Biological Sciences Building, Edmonton, AB T6G 2E9, Canada e-mail: cmadan@ualberta.ca*

While the effects of reward, affect, and motivation on learning have each developed into their own fields of research, they largely have been investigated in isolation. As all three of these constructs are highly related, and use similar experimental procedures, an important advance in research would be to consider the interplay between these constructs. Here we first define each of the three constructs, and then discuss how they may influence each other within a common framework. Finally, we delineate several sources of evidence supporting the framework. By considering the constructs of reward, affect, and motivation within a single framework, we can develop a better understanding of the processes involved in learning and how they interplay, and work toward a comprehensive theory that encompasses reward, affect, and motivation.

**Keywords: reward, affect, motivation, movements, emotion, context, arousal, valence**

#### **INTRODUCTION**

Reward, affect, and motivation are three highly related constructs, but are often investigated in isolation despite using similar experimental procedures. As an example, contextual fear conditioning is a common task used to investigate affective learning in rats. In this task, a rat is kept in a two-compartment chamber. Over time the rat gradually learns that when a tone is presented, the floor of one compartment of the chamber will deliver an electric shock. With respect to the affect construct, this task is described as eliciting a fear-related response (i.e., fleeing or freezing) when the tone occurs. However, this procedure is nearly identical to a conditioned place preference task, where the dependent measure is the proportion of time that the rat spends in each compartment, after the rat has been conditioned with shocks. Here the task is described as measuring motivational effects, e.g., approach vs. avoidance. Furthermore, it is important that an integral aspect of the task is the use of shocks, an aversive stimulus with respect to the reward construct, to elicit learning.

While it is possible to disentangle these constructs experimentally, they often coincide in real-world experiences and can converge and conflict in important ways. Here we briefly define each construct and discuss how they function in concert, as described by the proposed SIMON framework. By discussing the interplay of the constructs, we can lay the foundation for the development of a common theory encompassing reward, affect, and motivation.

#### **DEFINING THE CONSTRUCTS**

Before we can discuss interactions of reward, affect, and motivation, it is important to operationalize the three constructs independently. As the descriptions below are relatively brief, it is suggested to refer to the cited reviews for further details.

#### **REWARD**

Reward is the most clearly defined of the three constructs, particularly when viewed from an operant conditioning perspective: an organism learns that by responding (R) to a stimulus (S), an outcome (O) is presented. The outcome can either be appetitive (i.e., elicit an approach response), such as food, or aversive (i.e., elicit avoidance), such as an electric shock. Thus, in this simplified form, reward-based learning can be described as a S–R–O association (i.e., operant conditioning). To clarify the reward construct, it is important to note that while often used interchangeably, "reward" and "reinforcement" do not have identical meanings (White, 1989; Berridge and Robinson, 1998). Specifically, while rewards (i.e., appetitive stimuli) elicit approach responses, reinforcement should be used to describe the increase of responses to a stimulus. For further clarity, we will define "reward" as the construct itself, where outcomes can be either "appetitive" and "aversive," rather refer to outcomes as being "rewarding" (i.e., appetitive). It is also important to note that many different types of stimuli can be appetitive, such as monetary, food, and erotic stimuli (see Sescousse et al., 2013 for a review), while aversive stimuli usually are either monetary losses or electric shocks. Kirsch et al. (2004) provide a comprehensive discussion on the role of reward-based learning (specifically, conditioning) on cognition.

It is unarguable that rewards can vary along at least one dimension: value; when gains vs. losses are included, this dimension is often referred to as reward valence. However, recent findings suggest that reward is coded in the brain along two orthogonal dimensions: valence and salience (**Figure 1A**). Briefly, reward valence ranges from appetitive to aversive. Reward salience is defined by a quadratic relationship relative to value, such that the highest and lowest values experienced are highest in salience. Evidence for separable coding of reward salience is mainly supported by neural activations that correlate with the magnitude of outcomes, independent of the valence (Zink et al., 2004; Jensen et al., 2007; Cooper and Knutson, 2008; Litt et al., 2011). Recent behavioral studies have supported the notion of reward salience, even when the range of experienced outcomes is constrained to only the gain or loss domain

[(Ludvig et al., 2013; Madan and Spetch, 2010); e.g., **Figure 1A**, dashed line].

Neuroimaging results suggest that reward-related activations in the medial orbitofrontal cortex, rostral anterior cingulate cortex, and dorsal posterior cingulate correspond to reward value, while activations in the dorsal anterior cingulate cortex and insula correspond to reward salience (Litt et al., 2011). Activations in the ventral striatum, and particularly the nucleus accumbens, correspond to a mixture of reward value and salience, with salience playing a stronger role. There is also evidence of dissociable brain regions associated with gain vs. loss outcomes (Yacubian et al., 2006; Eppinger et al., 2013).

#### **AFFECT**

Affect can be defined as the conscious experience of emotions (Panksepp, 2000; Yik et al., 2011), though affect and emotion are often used synonymously [also see Kleinginna and Kleinginna (1981a), Lang (2010), and Izard (2010), for in-depth definitions of emotion]. To describe the affective space, Russell (1980) proposed the circumplex model of affect (also see Yik et al., 2011), which suggests that affect is comprised of two orthogonal dimensions: valence and arousal. Valence ranges from unpleasant to pleasant, while arousal ranges from bored to excited. The orthogonality of these two dimensions is also supported by neuroimaging results (Kensinger and Corkin, 2004; Posner et al., 2009): the valence-specific network was associated with the insula, the arousal-specific network with the medial prefrontal cortex and posterior cingulate, while both networks included the dorsolateral prefrontal cortex, anterior cingulate cortex, and the amygdala.

Within an experimental setting, words and images are used to elicit affect within the participant, most commonly using the International Affective Picture System (IAPS; Lang et al., 2008) and Affective Norms for English Words (ANEW; Bradley and Lang, 1999; but also see Warriner et al., 2013) databases. While affective states fill the complete circumplex space, stimuli often show a U- or boomerang-shaped distribution [Lang et al. (1998); e.g., **Figure 1B**].

#### **MOTIVATION**

Here we define motivation primarily based on the hedonistic principle (e.g., Young, 1959; White, 1989): motivation is the process of maximizing pleasure (i.e., appetitive, positive affect) and minimizing pain (i.e., aversive, negative affect). The ends of this motivational valence continuum correspomd to approach and avoidance behavior [see (Young, 1959), Kleinginna and Kleinginna (1981b), and Elliot and Covington (2001), for detailed definitions of motivation]. Based on this definition, it is clear that motivation is closely related to reward and affect. Additionally, motivation is intrinsically defined as motor movements, to either approach or avoid a stimulus. This perspective also overlaps highly with the idea of motor affordances (Gibson, 1977; Cisek and Kalaska, 2010).

It is also important to note that motivation incorporates contextual information that may influence stimulus processing outside of what could be explained by reward and affective processing alone [e.g., Berridge (2004); see **Figure 1C**]. As an example, money can be used as a appetitive outcome, but an individual's drive to obtain money is not always constant. A simpler example would be one's drive for food and water, both of which are appetitive, but an individual is not always hungry/thirsty and may be over-satiated and temporarily not want to consume more food or water, and thus be not approached. Other stimuli may be generally aversive, such as electric shocks; stimuli that reliably predict shocks will lead to avoidance behavior after sufficient learning. However, there are individual differences in approach–avoidance motivation. For instance, smoking is highly aversive to many, but considered appetitive to some. Foods like donuts and bacon can demonstrate even more inter-individual variability: despite being foods and thus generally appetitive, an individual who is dieting should avoid donuts and a vegetarian would actively avoid the bacon.

#### **SUMMARY**

Reward, affect, and motivation are related constructs. However, all constructs are discrete and dissociable: affect is largely an internal state, whereas a reward is related an outcome to be obtained (i.e., a goal) or avoided. While obtaining the outcome, e.g., food, likely also leads to a positive affective state, these are two dissociable processes, such as in the case of over-satiation. The food is still appetitive, but due to over-satiation, motivation is attenuated and the resulting affective state changes accordingly. Despite the strong intrinsic relationships between the constructs, they do not co-vary together in all situations. Studying these instances of disagreement are critical for the development of a comprehensive theory that encompasses all three constructs.

#### **THE SIMON FRAMEWORK**

While prior studies have discussed portions of their interplay, all three have not been discussed within the same framework. The SIMON framework serves this purpose by delineating a simple framework where the constructs are considered in concert. Here we propose the structure of the SIMON framework and describe prior evidence supporting portions of the framework:

The proposed SIMON framework suggests that after a [S]timulus is presented, it leads to an [I]nternal affective state. The stimulus and the elicited affective state both influence the resulting [M]otivated movement (i.e., response) where the individual responds to the stimulus. Based on the movement (or lack there of), an [O]utcome occurs that then also leads to an i[N]ternal affective state. See **Figure 2** for a graphical representation of the SIMON framework. Here we have the three constructs overlaid such that the reward construct is described by the Stimulus–Movement–Outcome (S–M–O) portion of the framework, which is a S–R–O association, i.e., operant conditioning. The affect construct is denoted by the S–I(–M) and O–N portions of the framework, where presented stimuli, including the outcome itself, elicit an affective state, and that this can also lead to a response. The motivation construct is described by the S–I–M portion of the framework, where a stimulus and it's resulting affective state both lead to a motivated movement.

#### **EVIDENCE FOR REWARD → MOTIVATION: CAN REWARD LEARNING LEAD TO MOTIVATED MOVEMENTS?**

Learning that a specific action leads to a reward-related outcome is the basis of operant conditioning and much of animal learning as a field (e.g., Balleine and Dickinson, 1998). Additionally, in certain circumstances, this type of learning can lead to the

development of superstitious behaviors in both human and nonhuman animals (e.g., "lucky numbers"; Brown, 1986). An illustration of this type of learning is outlined in (Cardinal et al., 2002, **Figure 2**), where the behavior resulting from the learning of a stimuli–outcome association are the elicitation of motivated movements, such as lever-pressing and locomotor approach.

Consider two theoretical perspectives that bear on the relation of these two constructs: from a reinforcement learning perspective, an agent's goal is to obtain as much reward (i.e., appetitive stimuli) as possible, by learning from the outcomes of prior actions Woergoetter and Porr (2008). In other words, seeking of rewards drives motivated movements, a notion supported by a number of studies (e.g., Hikosaka et al., 2008), and supporting the S–M–O portion of the SIMON framework. This rationale is also supported by the motor chauvinist perspective: the purpose of the brain is to produce movements, and sequences of motor actions are constructed to achieve high-level goals, such as acquire appetitive outcomes (Wolpert et al., 2001). Despite markedly different theoretical backgrounds, both perspectives suggest that at a basic level, motor movements are important to acquiring appetitive outcomes and that learning from reward-related experiences can reinforce the production of preceding movements.

#### **EVIDENCE FOR AFFECT → MOTIVATION: CAN AFFECT DRIVE MOTIVATED MOVEMENTS?**

Stimuli can often elicit affective states, and the combination of the stimuli and affective states can lead to a motor response (I–M portion of the SIMON framework). Well-known examples of this phenomena are reflex potentiation (fight-or-flight response) and freezing, and that affective states can influence physiological measures such as pupil dilation, heart rate (e.g., fear bradycardia; Campbell et al., 1997), and skin conductance (Bradley et al., 2008). Furthermore, research has shown that a variety of mammals use similar facial expressions (i.e., movements of the facial muscles) as humans to express positive/appetitive and negative/aversive states (e.g., Berridge, 2004). Lang and Bradley (2010) discuss evidence that affective stimuli can lead to greater motor potentiation, as measured by neural activation in supplementary motor area, among other brain regions, indicating higher-level cortical involvement, rather than only reflexive motor potentiation due to affect. Also see Carver(2006) and Harmon-Jones et al. (2013) for further discussions of the coupling between affect and motivation/motor-actions.

#### **EVIDENCE FOR REWARD → AFFECT: CAN REWARDS LEAD TO AFFECTIVE STATES?**

Rewards and affect both have important influences on learning, but are often discussed in isolation and use different procedures: studies of reward learning usually use a conditioningbased approach, where the task is learned through trial-anderror with the goal of obtaining the maximal cumulative reward. Studies using affective stimuli usually simply present the affective words/images, though there are instances where affect is conditioned (e.g., Mather and Knight, 2008; Schwarze et al., 2012). Nonetheless, one would expect that that earning an appetitive stimulus should elicit positive affect, whereas a negative outcome, such as a shock, is both aversive and elicits negative affect. This

affect (pink), and motivation (blue).

notion also suggested by Rolls (2000), and would comprise the O– N portion of the framework. Results from Dixon et al. (2010) also support this idea, where physiological measures of arousal were greater for appetitive outcomes (also see Bechara et al., 2005). Brown (1986) also suggests that arousal can play an important role in problem gambling.

Another line of research supporting the influence of rewards on affect is decision affect theory (Mellers et al., 1997). Here participants were presented with pie charts denoting probabilities of either gaining/losing a specified amount of money or receiving an outcome of \$0. After each trial, participants were asked to provide a rating of the emotional state on a Likert scale, ranging from extremely elated (+50) to extremely disappointed (−50), and affective responses were found to follow directly from predictions based on the reward outcome obtained. According to this line of research, in the event of a choice, "elation" is experienced if the outcome exceeds expectations, but "disappointment" is experienced if the outcome is worse than expected (Bell, 1985). If feedback on the forgone/unchosen option is also presented, "regret" is experienced if the chosen option is worse than the unchosen option's outcome, while "relief " is experienced when the chosen outcome led to the better outcome (Bell, 1982; Bryne, 2002). In other words, affective responses are operationalized based on outcomes experienced.

#### **EVIDENCE SUPPORTING THE FRAMEWORK**

By considering the relations between reward, affect, and motivation, a myriad of behavioral findings support the notion of a single framework. Here we outline a handful of such examples, along with their underlying principles as outlined by the SIMON framework.

#### **AFFECTIVE STIMULI AND MOTOR MOVEMENTS**

One of the most straightforward sources of evidence for the SIMON framework is that positive stimuli should be more congruent with an approach response, while negative stimuli should be more congruent with an avoid response. In lexical decision, where participants are presented with a letter-string that may or may not be a word and must judge it's lexicality, participants have been shown to respond relatively faster to positive words and relatively slower to negative words, when compared to neutral words (Estes and Adelman, 2008). Furthermore, in a go/no-go task, participants were slower to respond to images of fearful faces relative to happy faces (Hare et al., 2005). In both instances, participants exhibited a tendency to avoid negative stimuli, in conflict with the instruct to respond (i.e., approach) the stimuli. However, Hare et al. (2005) also reported that when participants are asked to inhibit their responses (i.e., no-go trials), false alarm rates are higher for happy than fearful faces as it is more difficult to suppress the approach response to the positive stimuli. Through similar principles, it has been suggested that approach/avoidance movements can provide information about an animal's affective preferences (Kirkden and Pajor, 2006).

#### **MOTOR MOVEMENTS AND REWARDS**

Another interesting line of evidence for the SIMON framework is intrinsic relationship between motor actions and rewards. One example of this is motor movements should optimize rewards earned in the task. Consider a reaching task where there are two target areas, each associated with a different reward value, e.g., similar to a dartboard (see Trommershäuser et al., 2008; Cisek, 2012). Participants have been found to aim for a point that would maximize their earnings and minimize potential losses, while also accounting for variability in precision.

A second interaction of motor and rewards can be observed in the influence of medication to treat motor dysfunction on reward-related behavior. It is known that Parkinson's patients taking dopamine agonists are more likely to develop problem gambling behavior (Dodd et al., 2005), a result found to generalize to other disorders also treated with dopamine agonists (e.g., d'Orsi et al., 2011). A likely cause for this interaction is that even though both gain outcomes and motor movements normally rely on the phasic release of dopamine (e.g., Steinberg et al., 2013), dopamine agonists increase the tonic level of dopamine, still leaving a dysregulation of the dopamine system.

#### **CONFLICT IN AFFECT AND REWARD TO IMPROVE EXECUTIVE CONTROL**

Another source of support for the SIMON framework is the use of affective stimuli that conflict with a reward. For instance, cigarette packages in North America often depict graphical images of the negative consequences of smoking, and have been shown to help individuals quit smoking (Farrelly et al., 2012). Extending this to food stimuli, Veling et al. (2011) presented participants with palatable food images in a go/no-go task, but preceded the food images with affective faces. Images of fearful faces were found to increase response time, suggesting that the conflict between reward and affect can be used to increase impulse control. Hollands et al. (2011) used a similar approach, but instead conditioned individuals to form associations between snack images and aversive bodily images (e.g., obese individuals) and found that the intervention improved healthy food choices relative to a control group. Similar interventions have also been used to treat phobias (e.g., Hekmat and Vanian, 1971).

## **CONCLUSION**

In the laboratory we aim to isolate a single construct and research it experimentally. However, in the real world learning is influenced by a multitude of concurrent effects that can be closely inter-related. By considering the constructs of reward, affect, and motivation within a single framework, we can work toward a better understanding of the processes involved in learning and provide an opportunity to refine the definitions of each of the component constructs. Finally, by considering the interplay of these three constructs, several current lines of research can be predicted in a common framework, and we can begin to work toward a comprehensive theory that encompasses reward, affect, and motivation.

#### **ACKNOWLEDGMENTS**

This work was partly funded by an Alexander Graham Bell Canada Graduate Scholarship (doctoral-level) from the National Science and Engineering Research Council of Canada as well as doctoral scholarships from the Alberta Gambling Research Institute. I would like to thank Marcia Spetch, Craig Chapman, and Elizabeth Kensinger for feedback on an earlier draft of the manuscript. I would also like to thank the the following designers from The Noun Project (http://www.thenounproject.com) for the icons used in **Figure 1**: Ben Rex Furneaux (wallet), Tom Glass Jr.

## **REFERENCES**


*and Motivational Processes*, eds P. J. Lang, R. F. Simmons, and M. T. Balaban, (Hillsdale, NJ: Lawrence Erlbaum Associates), 41–67.


(sandwich), Jacob Jalton (bacon, doughnut), Karthick Nagajaran (money), Sergey Shmidt (money), Kate Vogel (carrot), Hakan Yalcin (wallet), along with a number of unknown designers (electrical shock warning, smoking, water), as well as GoSquared Ltd. (http://www.gosquared.com) for the face icons used in **Figure 1B**.

*J. Neurosci.* 33, 9905–9912. doi: 10.1523/JNEUROSCI.2942-12.2013


regulation. *Emot. Rev.* 2, 363–370. doi: 10.1177/1754073910374661


using emotion-denoting words. *Hum. Brain Mapp.* 30, 883–895. doi: 10.1002/hbm.20553


impulsive behavior toward rewarding food objects. *Appetite* 56, 643–648. doi: 10.1016/j.appet.2011. 02.018


Zink, C. F., Pagnoni, G., Martin-Skurski, M. E., Chappelow, J. C., and Berns, G. S. (2004). Human striatal response to monetary reward depend on saliency. *Neuron* 42, 509–517. doi: 10.1016/S0896- 6273(04)00183-7

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 11 July 2013; accepted: 12 September 2013; published online: 07 October 2013.*

*Citation: Madan CR (2013) Toward a common theory for learning from reward, affect, and motivation: the SIMON framework. Front. Syst. Neurosci. 7:59. doi: 10.3389/fnsys. 2013.00059*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2013 Madan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Pain predictability reverses valence ratings of a relief-associated stimulus

#### *Marta Andreatta1 \*, Andreas Mühlberger 1,2, Evelyn Glotzbach-Schoon1 and Paul Pauli <sup>1</sup> \**

*<sup>1</sup> Department of Psychology (Biological Psychology, Clinical Psychology and Psychotherapy), University of Würzburg, Würzburg, Germany <sup>2</sup> Department of Experimental Psychology (Clinical Psychology and Psychotherapy), University of Regensburg, Regensburg, Germany*

#### *Edited by:*

*Dave J. Hayes, University of Ottawa, Canada*

#### *Reviewed by:*

*Jitendra Sharma, Massachusetts Institute of Technology, USA Rongjun Yu, South China Normal University, China*

#### *\*Correspondence:*

*Marta Andreatta and Paul Pauli, Department of Psychology (Biological Psychology, Clinical Psychology and Psychotherapy), University of Würzburg, Marcusstraße 9-11, D-97070 Würzburg, Germany e-mail: marta.andreatta@ mail.uni-wuerzburg.de; pauli@ psychologie.uni-wuerzburg.de*

Relief from pain is positively valenced and entails reward-like properties. Notably, stimuli that became associated with pain relief elicit reward-like implicit responses too, but are explicitly evaluated by humans as aversive. Since the unpredictability of pain makes pain more aversive, this study examined the hypotheses that the predictability of pain also modulates the valence of relief-associated stimuli. In two studies, we presented one conditioned stimulus (FORWARDCS+) before a painful unconditioned stimulus (US), another stimulus (BACKWARDCS+) after the painful US, and a third stimulus (CS−) was never associated with the US. In Study 1, FORWARDCS+ predicted half of the USs while the other half was delivered unwarned and followed by BACKWARDCS+. In Study 2, all USs were predicted by FORWARDCS+ and followed by BACKWARDCS+. In Study 1 both FORWARDCS+ and BACKWARDCS+ were rated as negatively valenced and high arousing after conditioning, while BACKWARDCS+ in Study 2 acquired positive valence and low arousal. Startle amplitude was significantly attenuated to BACKWARDCS+ compared to FORWARDCS+ in Study 2, but did not differ among CSs in Study 1. In summary, predictability of aversive events reverses the explicit valence of a relief-associated stimulus.

**Keywords: backward conditioning, forward conditioning, implicit and explicit responses, pain relief, threat unpredictability**

## **INTRODUCTION**

Reliable predictions of painful or threatening events modulate the perception of such events. Namely, both humans and mice respond to an aversive auditory stimulus with greater amygdala activation when such stimulus was presented unpredictably than when it was predictable (Herry et al., 2007). Moreover, human participants rate a painful stimulus more intense and more negative when they cannot reliably predict its delivery by means of a visual cue (Carlsson et al., 2006). In the same vein, a context in which a painful electric shock was unpredictably delivered induced higher anxiety level and potentiated startle response compared to a context where the same shock was predictable (Fonteyne et al., 2010). Since the startle response is an ancestral defensive reflex, the amplitude of which is modulated by the emotional state of an individual (Lang, 1995), it can be considered as an implicit biopsychological measure of the individual's emotional state. Thus, threatening situations prime defensive responses and cause potentiation of startle amplitude, while appetitive situations cause startle amplitude attenuation (Fendt and Fanselow, 1999; Koch, 1999). Hence, these findings suggest that the simple unpredictability of an aversive event increases the experienced aversiveness as indicated by explicit and implicit measures.

The present study moved one step further to examine if and how the unpredictability of an aversive event affects the relief experienced after its offset. According to previous findings, pain relief is appetitive and organisms react with reward-like responses to stimuli associated with it. Namely, humans show reward-like brain activations (e.g., ventral striatum) to a stimulus temporally contiguous to the decrease (Seymour et al., 2005) or the omission (Leknes et al., 2011) of a painful stimulation. Moreover, conditioned responses to a relief-associated stimulus are similar to those to a reward-associated stimulus. That is, appetitive events or stimuli predicting these events induce attenuation of the startle response (Schneider and Spanagel, 2008), or activation of the ventral striatum (Gottfried et al., 2002). Comparably, fruit flies avoid an odor (conditioned stimulus, CS) which was repeatedly presented before a painful unconditioned stimulus (US; forward conditioning or fear conditioning) but approach an odor which repeatedly followed a painful US (backward conditioning or pain relief conditioning; Tanimoto et al., 2004; Yarali et al., 2008). In the same vein, rats show after the injury of the muscle of a paw conditioned place preference (CPP) for the chamber in which the pain was alleviated by a local anesthesia (Navratilova et al., 2012). Finally, rats and humans respond with startle attenuation to a stimulus associated with pain offset, and such relief-associated stimulus activate striatal regions (Andreatta et al., 2010, 2012). Therefore, the relief reaction which follows a painful stimulation seems to entail appetitive properties.

These results support the opponent-process theory of Solomon (1980) and the relaxation theory of Denny (1971) which assert that aversive or painful events are initially characterized by a negative emotional state determined by the aversiveness of the pain itself. However, as soon as such aversive stimulation terminates, individuals feel an emotional state which entails opponent, namely appetitive properties. In line, the pleasantness of pain relief is linearly correlated with the aversiveness of the preceding painful stimulation that is the more the pain intensity is increased the more positive the following relief is experienced (Leknes et al., 2008). Considering the etiology of anxiety disorders (Mineka and Zinbarg, 2006), avoidance behavior is frequently considered to be maintained by negative reinforcement due to the relief following such behaviors (Kim et al., 2006). Besides such operant conditioning, classical conditioning may play an important role too since stimuli associated with the relief very likely guide behaviors (i.e., safety behavior). In any case, it is of crucial importance to unravel the impact of relief on conditioned responses and behaviors because this may in the long run allow improving therapeutic intervention of anxiety disorders.

Despite the appetitive physiological and neural responses, humans may value a stimulus associated with relief as negatively valenced and high arousing. We found that the verbal and explicit ratings of the participants dissociated from the physiological/neural and implicit responses (Andreatta et al., 2010, 2012). This dissociation can be understood on the basis of psychological theories which posit two systems: an impulsive and a reflective system, which can work in a synergic or antagonist fashion (Strack and Deutsch, 2004). The impulsive system generates behaviors on the basis of automatic processes influenced by simple associative learning mechanisms, while the reflective system generates behaviors on the basis of explicit knowledge about the situation. It is then presumable that the physiological and neural responses to a relief-associated CS are mediated by the impulsive system, but the ratings by the reflective system, which seems to consider the temporal contiguity of the US more important than the ongoing appetitive reaction. Supportively, human participants in our previous studies reported the stimulus presented upon pain termination—that is at the moment of relief—as being temporally linked to the painful US (Andreatta et al., 2010, 2012).

Because pain aversiveness is modulated by its predictability and because appetitive properties of pain relief depend on pain aversiveness, we assume that the predictability of pain modulates the following relief as well. In line with our previous studies (Andreatta et al., 2010, 2012), we hypothesize that participants show attenuated startle amplitude (i.e., reward-like responses) to a relief-associated stimulus. We further hypothesize that startle amplitude would be more attenuated for stimuli associated with the offset of unpredictable vs. predictable painful USs. Moreover, we expect a dissociation between physiological and verbal responses to a stimulus associated with the offset of an US which is delivered unpredictably as in our previous betweensubjects designed studies; that is negative valence and high arousal ratings (Andreatta et al., 2010, 2012). On the contrary, we expect positive ratings of the relief-associated CS when it follows a predictable US, because in this case participants would not explicitly associate the relief-associated CS with the painful US. In order to investigate these hypotheses, we conducted two studies in which we presented one stimulus as signal for pain onset (FORWARDCS+) and another stimulus upon the moment of the relief (i.e., after pain offset, BACKWARDCS+). In the first study, participant could predict only half of the painful USs, whereas the other half was delivered unwarned. In the second study, participants could reliably predict all painful USs. In both studies, we measured startle responses and skin conductance response (SCR) to conditioned visual stimuli as indices of implicit and physiological learning. In addition, we collected verbal reports for the valence and the arousal of the visual stimuli as indices of explicit and cognitive learning.

## **STUDY 1**

In Study 1 we investigated whether the unpredictability of a painful event (US) would induce reward-like physiological responses but negative reports. In other words, we wanted to replicate the results of our previous between-subjects study (Andreatta et al., 2010) in a within-subjects study. To this purpose, each participants experienced sixteen USs which were presented predictably at the offset of one visual stimulus (FORWARDCS+) and 16 USs which were delivered unpredictably shortly before another visual stimulus (during relief, BACKWARDCS+).

#### **MATERIALS AND METHODS**

#### *Participants*

Forty-one volunteers participated in the study and were recruited through media advertisements. For their participations, individuals received 14C. Eleven participants were excluded from the analysis. Three participants were excluded because they lost the electrodes for the electric shock (US) during the experiment and one because of technical problems. Seven additional participants were excluded from the analysis, one because interrupted the experiment, three because they were coded as non-responders (mean startle amplitude <5μV) and three because they did not have enough startle responses per condition (minimum = 4; for details see Materials and Methods). At the end, we considered 30 participants for the analysis (16 males; mean age: 25.33 years, *SD* = 3.18; range = 20–33 years).

#### *Stimulus material*

The aversive US consisted of a mild painful electric shock (200 ms duration). The shock was an electric pulse delivered with a frequency of 50 Hz. The intensity of the shock was individually assessed with a threshold procedure consisting of two ascending and descending series of electric shocks in steps of 0.5 mA (for details see Andreatta et al., 2010). The electric shock was generated by a current stimulator (Digitimer DS7A, Digitimer Ltd, Welwyn Garden City, UK, 400 V, maximum of 9.99 mA) and delivered by two disk electrodes with 9 mm diameter and spacing 30 mm over the forearm of the dominant hand. Participants rated the subjective painfulness of the US by means of a scale ranging from 0 ("feeling nothing at all") to 10 ("very intense pain") with 4 as an anchor for "just noticeable pain." The mean value of painful intensities was then increased by 1 mA. The mean intensity of the US was 2.32 mA (*SD* = 0.64) while the subjective intensity was 6.30 (*SD* = 1.51).

As visual *conditioned stimuli* (CS) we used yellow geometrical shapes presented for 8 s on a 19- computer screen localized circa 80 cm in front of the participants at the eye level over a black background. Shapes were a square, a triangle, a circle and a hexagon with 7.8 cm width and 7.8 cm height. The interstimulus interval (ISI), defined as the time between CS onset and US onset was as follows: the US was delivered either at the offset of one shape (FORWARDCS+; ISI = 8 s) or 6 s before the onset of another shape (BACKWARDCS+; ISI = −6 s). During the conditioning phase, three shapes were presented: FORWARDCS+, BACKWARDCS+ and a third shape (CS−), which was never associated with the US. During the test phase, four shapes were presented: FORWARDCS+, BACKWARDCS+, CS−, and a novel shape (NEW) as control stimulus. Shapes were counterbalanced across participants.

The *startle probe* was a burst of white noise of 98 dB with duration of 50 ms. The acoustic stimuli were presented binaurally over headphones and occurred randomly 3–7 s after shape's onset.

Two *questionnaires* were used as indicators for anxiety traits and the actual emotional state of the participants. The German version of the State-Trait Anxiety Inventory (STAI, Laux et al., 1981) is an inventory to assess the trait and/or the state anxiety of the participants. Both the trait and the state version consist of 20 items, respectively. Participants had to rate on a 4-point Likert scale from 1 ("almost never") until 4 ("almost always") how much the item would describe their anxiety. Higher scores indicate greater anxiety. Participants' anxiety level before and after the experiment did not change significantly [35.4 ± 4.8 vs. 35.7 ± 4.6; *t*(29) = 0.31, *p* = 0.758]. Trait anxiety scores in the current sample ranged between 20 and 58 (mean = 36.7, *SD* = 8.64), which is comparable to the published normal range of adults (Laux et al., 1981). The Positive and Negative Affect Schedule (PANAS; Krohne et al., 1996) is a questionnaire to asses participants' mood. High scores on the PA scale reflect positive affectivity and individuals are disposed to emotions such as enthusiasm. While high scores on the NA scale represent negative affectivity and individuals are disposed to emotions such as distress. Participant had to indicate to what extend he/she feels a particular emotion on a scale ranging from 1 ("very slightly") to 5 ("extremely"). Participants negative affect did not change throughout the experiment significantly [13.07 ± 5.4 vs. 11.60 ± 2.6; *t*(29) = 1.56, *p* = 0.129], but they reported less positive mood at the end of the experiment in comparison to the beginning [28.9 ± 4.7 vs. 24.97 ± 5.3; *t*(29) = 4.42, *p* < 0.001]. Such decrease of participant's positive mood might have depended on the unpleasantness of the paradigm (painful electric shock as well as an aversive white noise were presented).

#### *Procedure*

Upon the arrival in the laboratory, participants read and signed an informed consent approved by the ethics committee of the Deutsche Gesellschaft für Psychologie (DGPs), in which they were informed that a series of geometrical shapes, an electric shock and loud noises will have been presented and that they should keep the shapes in their visual focus. We did not mention the contingency between CSs and US. After having filled in the questionnaires, the electrodes were attached and the pain threshold procedure was performed as described above.

The experiment consisted of two phases: The conditioning and the test phase separated by subjective ratings. During the *conditioning phase* (**Figure 1A**) participants saw three out of four geometrical shapes 16 times each. Altogether, there were 48 trials, 16 CS− trials, 16 FORWARDCS+ trials, and 16 BACKWARDCS+ trials. The inter-trial interval (ITI) defined as the time between stimulus offset and the subsequent stimulus onset varied between 20 and 30 s (mean = 25 s). The choice of this relatively long ITI

One shape (FORWARDCS+) was presented before a painful electric shock (unconditioned stimulus, US), one shape (BACKWARDCS+) was presented after the US, and another shape (CS−) was never associated with the US. In Study 1 **(A)** 16 USs out of 32 were predicted by the FORWARDCS+, whereas the other 16 USs were delivered unwarned before the BACKWARDCS+. In Study 2 **(B)** all USs were preceded by the FORWARDCS+ and followed by the BACKWARDCS+.

was made in accordance with our previous study (Andreatta et al., 2010) as well as to avoid carry-over effects from one trial to the following one. Stimulus presentation was randomized with the only restriction that the same stimulus may not be presented more than twice in a row. No startle probe was presented during conditioning.

Before the test phase, 7 white noises were delivered every 7–15 s in order to decrease the initial startle reactivity. During the *test phase* participants saw four geometrical shapes, that were the three CSs (FORWARDCS+, BACKWARDCS+, and CS−) and a novel neutral shape (NEW) as control stimulus. No US was delivered during the test phase. Each stimulus was presented 16 times in a pseudorandom order (i.e., the same stimulus was not presented more than twice consequently), so altogether there were 64 trials. During the test phase, for 8 of the 16 stimulus presentations a startle probe was delivered between 3 and 7.5 s after stimulus onset in order to provoke the automatic defensive reflex. As in the conditioning phase, the ITIs varied between 20 and 30 s. In order to assure the unpredictability of the startle probes we additionally delivered 8 startling noises during the ITIs.

Before and after the conditioning phase as well as after the test phase, participants had to rate the valence (pleasantness) and the arousal (excitatory) of the visual stimuli by using two different visual analog scales (VAS) ranging from 1 until 9. One indicates "very unpleasant" for the valence and "calm" for the arousal, while 9 indicates "very pleasant" and "exciting," respectively. In addition, after the conditioning phase we verified participants' contingency awareness with a VAS ranging from 0 (no association) until 100 (perfect association). The contingency awareness indicates participant's ability to verbally report the association between the FORWARDCS+, BACKWARDCS+ or CS−, and the US.

#### *Physiological recording and data reduction*

Physiological responses were recorded with a V-Amp 16 amplifier and Vision Recorder V-Amp Edition Software (Version 1.03.0004, BrainProducts Inc., Munich, Germany). A sampling rate of 1000 Hz and a 50 Hz notch filter were applied. The offline analyses of these responses were conducted with Brain Vision Analyzer (Version 2.0; BrainProducts Inc., Munich, Germany).

Startle response was measured by means of electromyography (EMG) at the left *orbicularis oculi* muscle with two 5 mm Ag/AgCl electrodes. According to the guidelines (Blumenthal et al., 2005), one electrode was positioned under the pupil and the second one 1 cm laterally. The ground and the reference electrodes were placed on the right and left mastoids respectively. Before attaching the electrodes, the skin was slightly abraded and cleaned with alcohol in order to keep the impedance below 8 k-. EMG activity was continuously recorded. The electromygraphic signal was offline filtered with a 28 Hz low cutoff filter and a 500 Hz high cutoff filter as well as with a 50 Hz notch filter. Then the EMG signal was rectified and a moving average of 50 ms was applied. As baseline we used the 50 ms before startle probe onset (Grillon et al., 2006). Responses to startle probes were scored manually, and trial with excessive baseline shifts (±5μV) or movement artifacts were excluded from further analysis. Altogether, 19.1% of the trials were rejected, and a minimum of 4 out of 8 startle responses for each condition was required to keep the participant for further analysis. The peak amplitude was defined as the maximum peak relative to baseline during the 20–120 ms time window after startle probe onset. The raw data were then normalized within-subjects using *z*-scores in order to reduce the influence of the individual variability and to better detect the psychological processes. The *z*-scores were averaged for each condition (FORWARDCS+, BACKWARDCS+, CS−, NEW, and ITI). In order to investigate startle potentiation or startle attenuation, the scores for the ITI startle responses were subtracted from the startle responses of each condition.

SCR was recorded using two 5 mm Ag/AgCl electrodes placed on the palm of the no-dominant hand. SCR was continuously recorded with the same V-Amp system, which delivered a constant current of 0.5 V. Sampling rate was 1000 Hz. The galvanic response was offline filtered with 1 Hz high cutoff filter. The SCR was defined as difference (in μS) between the response onset (1–3 s after shape onset) and the response peak (Tranel and Damasio, 1994; Delgado et al., 2011). Trials containing startle probes were not considered for the analysis of the SCR. Responses below 0.02 μS were coded as zero. For SCR analysis of the conditioning phase, two further participants were excluded and for the SCR analysis during the test phase we excluded 10 further participants because they had no detectable SCR (non-responses) in each condition. The skin raw conductance data were then square root transformed in order to normalize the distribution and the scores were averaged for each condition separately for the conditioning (FORWARDCS+, BACKWARDCS+, CS−) and the test phase (FORWARDCS+, BACKWARDCS+, CS−, and NEW).

#### *Data analysis*

All data were analyzed with SPSS for Windows (Version 20.0, SPSS Inc.). Startle amplitude, valence, arousal and contingency ratings were separately analyzed with multivariate analysis of variance (MANOVA). For all dependent variables MANOVAs had as within-subjects factor stimulus (FORWARDCS+, BACKWARDCS+, CS−, and NEW). The SCR was separately analyzed for the conditioning (FORWARDCS+, BACKWARDCS+, and CS−) and the test phase (FORWARDCS+, BACKWARDCS+, CS−, and NEW) having stimulus as within-subjects factor. In the analysis for the valence and the arousal ratings, the within-subjects factor phase was added (T1: before conditioning, T2: after conditioning, T3: after test phase), as well as for the contingency ratings (T1: after conditioning, T2: after test phase). The alpha (α) level was set at 0.05 for all analyses. The effect size is reported as partial η2.

#### **RESULTS**

#### *Valence ratings (Figure 2A, left panel)*

The valence of the four CSs was differentially affected by conditioning as confirmed by the significant Stimulus × Phase interaction [*F*(6, <sup>23</sup>) <sup>=</sup> <sup>2</sup>.77, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.035, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.42]. According to follow up *t*-tests, the valence of the four geometrical shapes did not differ before conditioning (all *p*s > 0.42) and were rated as neutral (i.e., 5; all *p*s > 0.40). After conditioning, the FORWARDCS+ and the BACKWARDCS+ had comparable valence [*t*(28) = 0.56, *p* = 0.579] which was significantly more negative than the valence of both the CS− [FORWARDCS+: *t*(28) = 3.48, *p* = 0.002; BACKWARDCS+: *t*(28) = 2.73, *p* = 0.011] and the NEW [FORWARDCS+: *t*(28) = 3.08, *p* = 0.005; BACKWARDCS+: *t*(28) = 2.38, *p* = 0.024]. Valence ratings between the CS− and the NEW did not differ [*t*(28) = 1.65, *p* = 0.110]. After the test phase, the FORWARDCS+ valence [*t*(28) = 1.86, *p* = 0.073] and the BACKWARDCS+ valence [*t*(28) = 2.00, *p* = 0.055] remained slightly more negative than the CS− valence although these comparisons just failed to reach the significance level. The valence ratings for the NEW did not differ from the other stimuli (all *p*s > 0.26) and the valence of the FORWARDCS+ did not differ from the BACKWARDCS+ [*t*(28) = 0.44, *p* = 0.663]. In summary, both the FORWARDCS+ and the BACKWARDCS+ acquired negative explicit valence after conditioning.

#### *Arousal ratings (Figure 2B, left panel)*

The arousal of the four CSs was differentially modulated by conditioning as the significant Stimulus × Phase interaction indicates [*F*(6, <sup>23</sup>) <sup>=</sup> <sup>2</sup>.51, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.051, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.40]. Follow-up *t*tests revealed equal arousal ratings among the geometrical shapes (all *ps* > 0.26) before conditioning. After conditioning, the FORWARDCS+ [*t*(28) = 1.98, *p* = 0.058], the BACKWARDCS+ [*t*(28) = 1.97, *p* = 0.059], but not the NEW [*t*(28) = 1.90, *p* = 0.098] were slightly rated more arousing than the CS−, although these tests just failed to reach the significance level. Moreover, the FORWARDCS+, the BACKWARDCS+, and the NEW did not differ regarding arousal ratings (all *p*s > 0.44). After the test phase, the FORWARDCS+ [*t*(28) = 2.30, *p* = 0.029] and the BACKWARDCS [*t*(28) = 2.39, *p* = 0.024] were rated more arousing than the CS−, but not to the NEW [*t*(28) = 0.94, *p* = 0.354]. Notably, arousal ratings of the FORWARDCS+ did not differ significantly from those of the BACKWARDCS+ [*t*(28) = 0.27, *p* = 0.787] and the NEW [*t*(28) = 1.55, *p* = 0.133] after the test phase, and the BACKWARDCS+ was rated with higher arousal compared to the NEW [*t*(28) = 2.20, *p* = 0.036]. In summary, the FORWARDCS+ and the BACKWARDCS+ were rated as high arousing stimuli after conditioning, and such ratings lasted until the end of the experiment.

#### *Startle response (Figure 3A)*

Analysis of the startle response revealed no significant main effect of stimulus [*F*(3, <sup>27</sup>) <sup>=</sup> <sup>0</sup>.23, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.875, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.03] indicating no differential responses to the FORWARDCS+, the BACKWARDCS+, the CS−, and the NEW. We, however, compared *z*-scores of the startle amplitudes to the four visual stimuli with the mean (i.e., 0) and found that only the FORWARDCS+ induced a significant potentiation of the startle response [*t*(29) = 2.10, *p* = 0.044].

#### *SCR (Figure 4A)*

Analysis of the SCR revealed that the conditioning differentially affected the SCR to the CSs as reflected in the significant main effect of stimulus [*F*(2, <sup>26</sup>) <sup>=</sup> <sup>15</sup>.72, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.547]. Follow up *t*-tests indicated that the FORWARDCS+ elicited higher SCRs compared to the CS− [*t*(27) = 3.69, *p* = 0.001] and to the BACKWARDCS+ [*t*(27) = 5.71, *p* < 0.001]. Moreover, SCRs to the BACKWARDCS+ were significantly lower than to the CS− [*t*(27) = 4.21, *p* < 0.001]. Analysis of the SCR during the test phase indicated successful extinction learning as all stimuli elicited comparable SCRs [*F*(3, <sup>17</sup>) <sup>=</sup> <sup>0</sup>.15, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.928, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.026]. In summary, the FORWARDCS+ elicited enhanced fear responses (i.e., high SCR), whereas the BACKWARDCS+ seems to be less arousing as indicated by low SCR.

#### *Contingency awareness (Table 1)*

Participants were aware about the contingency between the CSs and the US as indicated by a significant main effect stimulus [*F*(2, <sup>27</sup>) <sup>=</sup> <sup>24</sup>.10, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.641]. Follow up *t*-tests indicated that participants reported significant higher contingency ratings for the FORWARDCS+ [*t*(28) = 6.89, *p* < 0.001] and the BACKWARDCS+ [*t*(28) = 5.20, *p* < 0.001] than for the CS−. Furthermore, contingency ratings for the FORWARDCS+ and the BACKWARDCS+ did not differ [*t*(28) = 1.88, *p* = 0.071]. In summary, participants recognized the associations between the FORWARDCS+ and the BACKWARDCS+ and the US.

#### **DISCUSSION**

The main goal of Study 1 was to investigate the modulatory role of the unpredictability of a painful electric shock (US) over relief. To this end, we presented half of the USs predictably after one visual stimulus (FORWARDCS+) while the other half was presented unwarned shortly before another visual stimulus (BACKWARDCS+). In line with our previous findings (Andreatta et al., 2010), we found that both the FORWARDCS+ and the BACKWARDCS+ compared to the CS− stimulus acquired explicit aversive properties through conditioning as indicated by negative valence and high arousal ratings (**Figure 2**). This acquired explicit aversiveness of both the FORWARDCS+ and the BACKWARDCS+ might be due to the cognitive knowledge that these two visual stimuli were temporally presented in association with the painful electric shock as the participants' contingency ratings indicate.

A further interesting and new result of this study is the modulation of the SCR by the CSs (**Figure 4A**). Namely, SCR to the BACKWARDCS+ was significantly lower compared to the FORWARDCS+ and the CS− during conditioning indicating that the relief-associated stimulus (BACKWARDCS+) was less arousing than the pain-signaling stimulus (FORWARDCS+) and even less

and arousing. On the contrary, in Study 2 (right panels) the FORWARDCS+ was rated as negatively valenced and high arousing, whereas the BACKWARDCS+ as positively valenced and low arousing (+*p* < 0.06; ∗*p* < 0.05; ∗∗∗*p* < 0.001).

arousing than the safety signal (CS−). These results are in line with previous human findings of low SCR during relief which also was positively correlated with the intensity ratings of the relief (Leknes et al., 2008). Notably, the physiological and the verbal responses dissociated. Namely, participants rated the BACKWARDCS+ as arousing, but they showed low SCR. Such dissociation is in line with the valence-related dissociation found in our previous studies (Andreatta et al., 2010, 2012). Thus, in the previous studies and the current study we found relief-like physiological responses (attenuation of startle amplitude and low SCR), but fear-like verbal reports (negative valence and high arousal). Apparently, the offset of unpredictable aversive stimuli is valued in an antagonist fashion by the implicit impulsive system and the explicit reflective system (Strack and Deutsch, 2004). On the one hand, the physiological responses reflect the implicit relief-reactions going on after a painful event. On the other hand, the explicit negative valuation may be imposed by the explicit knowledge that the stimulus is somehow associated with pain (see contingency rating, Dunsmoor et al., 2011). Finally, SCR did not differ among the FORWARDCS+, the BACKWARDCS+, the CS−, and the NEW during the test phase and this may be due to processes linked to extinction learning (Phelps and Ledoux, 2005).

**FIGURE 3 | Startle amplitudes to the visual stimuli during the test phase.** Bars (with standard errors) depict the startle amplitude in *z* scores in response to the FORWARDCS+ (light gray), the BACKWARDCS+ (black), the CS− (white), and the NEW (striped). Startle responses did not differ among CSs after conditioning in Study 1 **(A)**. On the contrary, startle response was significantly attenuated (i.e., reward-like conditioned responses) by the BACKWARDCS+ as compared to the FORWARDCS+ and the NEW after conditioning in Study 2 **(B)** ( ∗*p* < 0.05; ∗∗*p* < 0.01).

and the NEW (striped) either during the conditioning or during the test phase. In both Study 1 **(A)** and Study 2 **(B)** the SCR to the BACKWARDCS+ was significantly lower compared to the FORWARDCS+ and the CS− ( ∗∗*p* < 0.01; ∗∗∗*p* < 0.001).

In contrast to our previous between-subjects designed study (Andreatta et al., 2010), we did not find any difference in the participants' startle responses to the CSs (**Figure 3A**). Most likely, differences in the number of the painful electric shocks together with their unpredictability might have played a crucial role here. In fact, we doubled the number of shocks in the present study compared to the previous between-designed studies (32 vs. 16). According to Fanselow and Lester (1988), circa-strike defensive responses depend on the shock density as well as on their imminence. Shock density refers to the number of shocks per time; the more dense the shock schedule is (i.e., the increased number of shocks), the more the animals present circa-strike defensive response (i.e., flight/fight). The imminence refers to the real presence and the vicinity of a danger, the closer a danger is the stronger fear responses are prompted. Referring to the present study, the US preceding the BACKWARDCS+ had no warning signal, which might have provoked a feeling of sustained fear or anxiety (Barlow, 2000; Grillon et al., 2008; Davis et al., 2010; Fonteyne et al., 2010). Moreover, both the USs presented after the FORWARDCS+ and those presented before the BACKWARDCS+ may have been experienced as one single aversive event, which was sometimes predicted, but sometimes not. Therefore, the unreliable prediction of the shock might have provoked a state of uncertainty which consequently induced anxiety rather than fear. Furthermore, the anxious feeling of the individuals in this study might have been stronger than the one induced in the betweendesigned study because of the higher density of the shocks. Hence, we think that the conditioned responses here are induced by a post-encounter stage rather than by a circa-strike stage, in line with Fanselow (1994) and Davis et al. (2010) who assume that post-encounter behavior resembles sustained anxiety, whereas circa-strike behavior is induced by phasic fear. Thus, participants have "encountered" the threat (the US), but because of its relative predictability, such threat is not sufficiently imminent for provoking clear discriminative fear responses (e.g., potentiation of the startle response to the threat signal).

## **STUDY 2**

In Study 2 we investigated the modulatory influence of a predictable pain over pain relief. For this purpose, we associated the painful electric shock (US) during all trials with both a FORWARDCS+ and a BACKWARDCS+. The FORWARDCS+ predicted all USs which were presented at its offset, and the BACKWARDCS+ followed all USs. Thus, here we never delivered an unpredictable painful US before the BACKWARDCS+, all USs were predictable by the FORWARDCS+. We expected appetitive conditioned responses to the BACKWARDCS+ as compared to the FORWARDCS+ such as attenuation of startle response and positive valence ratings. As opposed to Study 1, we did not expect a dissociation between implicit and explicit responses because in this case the FORWARDCS+ signals the US and consequently the BACKWARDCS+ might be explicitly associated with the its termination.

## **MATERIALS AND METHODS** *Participants*

Thirty-three volunteers participated in the study and were recruited through media advertisements. For their participations, individuals received 14C. Three participants were excluded from the analysis: One because of technical problems, one because it interrupted the recording and the third one because it was the only one who was unaware (i.e., she was not able to indicate the association between the stimuli). Two additional participants were excluded from the analysis, because they were coded as no-responders (mean startle amplitude <5μV). At the end, we considered 28 participants for the analysis (9 males; mean age: 22.96 years, *SD* = 1.48; range = 21–26 year). Participants' trait anxiety scores ranged between 24 and 65 (mean = 40.6, *SD* = 8.86), which is comparable to the published normal range of adults (Laux et al., 1981). Participants' anxiety level (STAI state) before and after the experiment did not change significantly [37.46 ± 6.77 vs. 39.75 ± 7.86; *t*(27) = 1.31, *p* = 0.202] as well as negative affect [NA scale from PANAS; 12.50 ± 2.58 vs. 13.64 ± 4.99; *t*(27) = 1.25, *p* = 0.223]. Similar to the Study 1, participants positive mood (PA scale from PANAS) significantly decreased at the end of the experiment compared to their mood at the beginning [28.18 ± 6.04 vs. 25.61 ± 7.23; *t*(27) = 2.14, *p* = 0.041]. Again, this decreased positive mood may have been induced by the aversiveness of the stimuli used or by the boringness of the experiment.

#### *Stimulus material*

The stimulus material was exactly the same as in Study 1. The mean electric shock intensity was 1.84 mA (*SD* = 0.27) and participants' subjective painfulness of the US was 6.39 (*SD* = 1.26; range: 5–9). Importantly, participants still rated the US as painful at the end of the experiment (6.39, *SD* = 1.57; range: 3–10) and the two ratings did not differ [*t*(26) = 0.33, *p* = 0.746].

#### *Procedure*

The procedure of the Study 2 was almost the same as in Study 1; the only difference was the number of USs and their predictability.

During the *conditioning phase* participants saw three out of four geometrical shapes 16 times each. Altogether, there were 32 trials, 16 CS− trials and 16 CS+ trials. The CS+ trials started with the FORWARDCS+ onset, at FORWARDCS+ offset the US was delivered (ISI = 8 s), and 6 s later the BACKWARDCS+ was presented (ISI = −6 s; **Figure 1B**). The CS− trials consisted of CS− presentation. The ITI varied between 20 and 30 s (mean = 25 s) for the same reasons as in Study 1 (see Page 6). Stimulus presentation was randomized with the only restriction that the same stimulus may not be presented more than twice in a row. No startle probe was presented during conditioning. The *test phase* and the *subjective rating* were exactly the same as in Study 1.

In addition, after conditioning we verified participants' awareness about the association between the CSs and the US by means of an open question. That is, participants had to verbally report to which geometrical shape the electric shock was associated. Only one participant recalled the association between the BACKWARDCS+ and the US, one participant was not able to indicate a particular shape (she was then coded as unaware and excluded from the statistical analysis), whereas all other participants recalled the association between the FORWARDCS+ and the US.

#### *Physiological recording and data reduction*

Physiological responses and data reduction worked out in exactly the same way as in Study 1. Notably, 9.2% of the trials were rejected for the analysis of startle response. Moreover, seven further participants were excluded from the analysis for the SCR during conditioning because they had no detectable peaks per condition and 12 for the same analysis during test phase.

#### *Data analysis*

All data were analyzed with SPSS for Windows (Version 20.0, SPSS Inc.) and as for the Study 1 startle amplitude, SCR, valence, arousal and contingency ratings were separately analyzed with MANOVA. Again the alpha (α) level was set at 0.05 for all analyses.

#### **RESULTS**

#### *Valence ratings (Figure 2A, right panel)*

Analysis of the valence ratings revealed a significant Stimulus × Phase interaction [*F*(6, <sup>22</sup>) <sup>=</sup> <sup>5</sup>.32, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.002, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.592]. *Posthoc t*-tests indicated that the valence of the geometrical shapes at the beginning of the experiment was equally rated (all *ps* > 0.24) and that the valence was reported as neutral (i.e., 5; all *p*s > 0.28). After conditioning, the FORWARDCS+ was rated as more negatively valenced compared to the CS− [*t*(27) = 5.97, *p* < 0.001], the NEW [*t*(27) = 5.22, *p* < 0.001] and interestingly to the BACKWARDCS+ as well [*t*(27) = 5.82, *p* < 0.001]. The valence ratings of the CS− did not differ significantly from those of the BACKWARDCS+ [*t*(27) = 1.20, *p* = 0.242] and the NEW [*t*(27) = 1.22, *p* = 0.234], and the BACKWARDCS+ was rated more positive than the NEW [*t*(27) = 2.15, *p* = 0.041]. After the test phase, the valence ratings of the NEW were significant more negative than those of the CS− [*t*(27) = 2.64, *p* = 0.014], but no other significant differences were found (all *p*s > 0.09). Contrarily to the Study 1, the relief-associated stimulus (the BACKWARDCS+) acquired explicit positive valence as opposed to the threat signal (the FORWARDCS+) and similar to the safety signal (the CS−).

#### *Arousal ratings (Figure 2B, right panel)*

Analysis of the arousal ratings revealed a significant modulation of conditioning as indicated by a significant Stimulus × Phase interaction [*F*(6, <sup>22</sup>) <sup>=</sup> <sup>6</sup>.<sup>32</sup> *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.633]. *Posthoc t*-tests indicated equal arousal for all four CSs at the beginning of the study (all *p*s > 0.12). After conditioning, the FORWARDCS+ was rated as more arousing compared to the CS− [*t*(27) = 5.30, *p* < 0.001], the NEW [*t*(27) = 5.14, *p* < 0.001] and to the BACKWARDCS+ [*t*(27) = 6.29, *p* < 0.001], while the BACKWARDCS+, the CS− and the NEW were rated with comparable arousal (all *p*s > 0.08). After the test phase, the FORWARDCS+ was still rated as more arousing than the CS− [*t*(27) = 1.99, *p* = 0.057; despite marginally] and the BACKWARDCS+ [*t*(27) = 2.50, *p* = 0.019], but not more arousing than the NEW [*t*(27) = 0.72, *p* = 0.475] anymore. Moreover, the NEW was rated as significantly more arousing compared to the BACKWARDCS+ [*t*(27) = 3.22, *p* = 0.003] and the CS− [*t*(27) = 2.81, *p* = 0.009]. Notably, the CS− and the BACKWARDCS+ did not differ regarding arousal ratings [*t*(27) = 0.50, *p* = 0.624]. Contrarily to the Study 1, the relief-associated stimulus (BACKWARDCS+) was valued as less arousing than the threat stimulus (FORWARDCS+).

#### *Startle response (Figure 3B)*

Analysis for the startle responses revealed a significant main effect of stimulus [*F*(3, <sup>25</sup>) <sup>=</sup> <sup>3</sup>.85, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.022, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.316]. *Post-hoc t*tests indicated that the startle amplitude to the BACKWARDCS+ was significantly attenuated compared to the FORWARDCS+ [*t*(27) = 2.85, *p* = 0.008] and to the NEW [*t*(27) = 2.45, *p* = 0.021], but not to the CS− [*t*(27) = 1.13, *p* = 0.267]. Moreover, the startle responses to the CS− did not differ significantly from those to the FORWARDCS+ [*t*(27) = 1.53, *p* = 0.137] and to the NEW [*t*(27) = 1.00, *p* = 0.326]. Because we did not find significant discriminative responses to the FORWARDCS+ and the CS−, we compared the *z*-scores of the startle amplitudes to the four visual stimuli with the mean (i.e., 0) in order to verify whether startle amplitude to the FORWARDCS+ was potentiated. Tests revealed significant startle potentiation to the FORWARDCS+ [*t*(27) = 2.93, *p* = 0.007] and to the NEW [*t*(27) = 1.59, *p* = 0.052; despite marginally], but not to the BACKWARDCS+ [*t*(27) = 1.59, *p* = 0.123] and to the CS− [*t*(27) = 0.47, *p* = 0.640].

#### *SCR (Figure 4B)*

Analysis for the SCR during conditioning revealed a significant main effect of stimulus [*F*(2, <sup>19</sup>) <sup>=</sup> <sup>12</sup>.05, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.559]. Similar to the Study 1, *post-hoc t*-tests indicated significant higher SCR to the FORWARDCS+ compared to the CS− [*t*(20) = 3.15, *p* = 0.005] and to the BACKWARDCS+ [*t*(20) = 4.91, *p* < 0.001]. Again, the SCR to the BACKWARDCS+ was lower compared to the CS− [*t*(20) = 3.12, *p* = 0.005]. Same as for Study 1, analysis of the SCR during the test phase did not reveal a significant main effect of stimulus [*F*(3, <sup>13</sup>) <sup>=</sup> <sup>2</sup>.24, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.132, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.34].

#### *Contingency awareness (Table 1)*

Participants' awareness about the association between the visual stimuli and the painful shock was significantly modulated by conditioning as the significant main effect of stimulus indicated [*F*(2, <sup>19</sup>) <sup>=</sup> <sup>92</sup>.87, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.907]. *Post-hoc t*-tests indicated higher contingency ratings for the FORWARDCS+ compared to the BACKWARDCS+ [*t*(20) = 10.99, *p* < 0.001] and to the CS− [*t*(20) = 13.86, *p* < 0.001]. No significant difference between the BACKWARDCS+ and the CS− [*t*(20) = 0.06, *p* = 0.951] was found.

#### **DISCUSSION**

In Study 2 we investigated whether a stimulus associated with the relief from a painful US would acquire reward-like properties even when the aversive event is fully predicted. This is exactly what we found. Thus, when the BACKWARDCS+ followed a fully predictable painful US, participants showed significant attenuation of the startle amplitude (i.e., reward-like responses) to the BACKWARDCS+ compared to the FORWARDCS+ and the NEW stimulus (**Figure 3B**). Therefore, when the onset of the US was predictable, the BACKWARDCS+ appears to acquire implicit positive valence in parallel to our previous findings (Andreatta et al., 2010). Strikingly and in contrast to our previous studies (Andreatta et al., 2010, 2012), the BACKWARDCS+ in this case acquired an explicit positive valence too and low arousal (see **Figure 2**).

Why does the BACKWARDCS+ acquire explicit appetitive properties when presented after a FORWARDCS+, but explicit aversive properties when presented "alone"? Differently from Study 1, participants might have felt in Study 2 less anxious since the threat was fully predictable. Moreover, all participants (except one) explicitly indicated the FORWARDCS+ and not the BACKWARDCS+ as the visual stimulus associated with the US (see contingency ratings, **Table 1**). The absence of an explicit association between the BACKWARDCS+ and the US may have determined its positive valence and its low arousal ratings. Thus, since the FORWARDCS+

#### **Table 1 | Contingency ratings.**


*Scores (standard deviation) indicate the subjective expectancy of the painful US in association with the respective shapes. Zero indicated "no association at all" and 100 "perfect association."*

reliably predicted the US, the BACKWARDCS+ became explicitly associated with the relief only.

We did not find during the test phase discriminative startle responses to the FORWARDCS+ and the CS−. This result is quite puzzling considering the broad literature on classical fear conditioning. However, we should consider that the startle responses were recorded during the test phase in which no USs were delivered. Therefore, it is possible to assume that a new learning (i.e., extinction learning) has started and modulated these responses (Milad and Quirk, 2012). In any case, the strong attenuation of the startle response to the BACKWARDCS+ suggests that relief-conditioned responses undergo slower extinction processes; although, this hypothesis must be further investigated.

Nicely, the SCR findings in Study 2 mirror the SCR results of Study 1. Namely, SCR to the BACKWARDCS+ was significantly lower compared to the other CSs during conditioning (**Figure 4B**). Moreover, the greater SCR to the FORWARDCS+ compared to the CS− confirms previous studies, in which participants showed increased SCR to the threat-predicting CS suggesting greater sympathetic arousal (Büchel et al., 1998; Labar et al., 1998; Weike et al., 2008; Li et al., 2011). Finally, it is conceivable that pain relief does not implicate or does not need strong sympathetic engagement because there is no real need to react since the threat is not imminent anymore (see General Discussion for further interpretations; Fanselow and Lester, 1988; Fanselow, 1994). Discriminative SCRs to the CSs disappeared during the test phase, which may be related to extinction processes.

## **GENERAL DISCUSSION**

The goal of the present studies was to investigate the temporal sequence between pain and its relief and how their contiguity and predictability would affect the individuals' implicit and explicit responses. Because of the dependence of relief pleasantness on pain aversiveness and of pain aversiveness on pain unpredictability, we wondered whether the prediction of a painful stimulus might differentially modulate the responses to a stimulus associated with pain relief. We realized two studies which were similar in most aspects, but differed in the predictability of the painful US. During the conditioning phase of both studies, one geometrical shape (FORWARDCS+) was presented *before* a mild painful electric shock (aversive US), while another geometrical shape (BACKWARDCS+) was presented *after* the US, and a third geometrical shape (CS−) was unrelated to the US. In Study 1 on the one hand, the FORWARDCS+ and the BACKWARDCS+ were presented in different trials meaning that during the FORWARDCS+ trials the US could be predicted, while during the BACKWARDCS+ trials the BACKWARDCS+ followed an unpredicted US. In Study 2 on the other hand, the FORWARDCS+ and the BACKWARDCS+ were presented in one trial meaning that the US could be predicted by the FORWARDCS+ and the BACKWARDCS+ followed this predictable US.

Based on our previous and other fear conditioning studies, we hypothesized that the FORWARDCS+ would acquire negative affective implicit and explicit properties in both studies. This assumption was confirmed. Participants showed increased fear responses to the FORWARDCS+ as indicated by potentiation of the startle response, high SCR, and negative valence as well as enhanced arousal ratings <sup>1</sup> . In other words, the FORWARDCS+ acquired aversive explicit and implicit properties by means of its association with the painful US; it became a signal of danger (Weike et al., 2008; Andreatta et al., 2010; Delgado et al., 2011; Li et al., 2011).

For Study 1, we expected in line with our previous betweensubjects studies that the BACKWARDCS+ would acquire an implicit positive valence because of its coincidence with the experience of relief, but an explicit negative valence because it is the only stimulus which was contiguous to the US. These hypotheses were confirmed for rating data, but not for startle data (for a discussion of the discrepancy in startle data between Study 1 and previous studies see Discussion of Study 1). In contrast and most important, we expected for Study 2 that the BACKWARDCS+ would acquire an implicit positive valence because of its coincidence with the experience of relief, and we also expected explicit positive properties because this stimulus should be experienced as independent from the US on a cognitive level. These hypotheses were confirmed. Thus, startle response was attenuated in response to the BACKWARDCS+ and participants reported positive valence as well as low arousal ratings for the BACKWARDCS+.

Learning the relationship between a neutral stimulus (CS) and an aversive event (US) implicates two kinds of memories (Williams et al., 2001; Hamm and Weike, 2005; Riebe et al., 2012). On the one hand, organisms form implicit fear memories which activate subcortical structures of the fear matrix like the amygdala and initiate defensive responses in an automatic manner, i.e., without cognitive appraisal. On the other hand, organisms form explicit fear memories which involve cortical structures like prefrontal cortex (PFC) and initiate fear responses requiring cognitive appraisal. Intuitively, the explicit cognitive knowledge about CS−US association may strongly influence participants' verbal reports, but to a lesser extend implicit memories. Hence, it is plausible that a stimulus presented upon the moment of the relief may acquire either aversive or appetitive explicit properties dependent on declarative encoding of the CS−US relation. In line, in Study 1 as well as in our previous studies (Andreatta et al., 2010, 2012) participants received an unpredictable aversive stimulus which was shortly followed by the BACKWARDCS+. We think that after an aversive event an appetitive reaction is always started, but the explicit encoding of such appetitive reaction is determined by the declarative processing of the temporal relationship between the stimuli. Hence, the impossibility to reliably foresee the aversive event entailed a negative valuation of all stimuli which were temporally nearby the event. On the contrary in Study 2, participants were able to reliably predict the aversive event by a preceding stimulus. Consequently, participants may have experienced the BACKWARDCS+ as "purely" associated with the relief because there was no need for an association between the painful event and any following stimuli. This interpretation is supported by the contingency ratings. In fact, if the US was unpredictable as in Study 1 and in our previous studies (Andreatta et al., 2010, 2012), participants reported an association between the US and the BACKWARDCS+. If the US was predicted by a preceding CS as in Study 2, participants report no contingency between the US and the BACKWARDCS+. As results, the synergic information from the implicit and the explicit level allows the participants to rate the BACKWARDCS+ as appetitive (i.e., positive valence) and reassuring (i.e., low arousal).

As already mentioned, pain relief entails reward-like properties (Seymour et al., 2005; Leknes et al., 2008, 2011) and promotes appetitive learning (Tanimoto et al., 2004; Yarali et al., 2008; Andreatta et al., 2010, 2012; Navratilova et al., 2012). That is, brain areas involved in the processing of rewarding events (Tobler et al., 2003) are also activated by pain relief (Seymour et al., 2005; Leknes et al., 2011), and organisms react with appetitive conditioned responses to a stimulus presented upon the relief (Tanimoto et al., 2004; Yarali et al., 2008; Andreatta et al., 2010, 2012; Navratilova et al., 2012). Confirming these studies, we found discriminative conditioned responses to the CSs in Study 2 (i.e., potentiation of the startle response to the FORWARDCS+ and attenuation of the startle response to the BACKWARDCS+), but not in Study 1. Why? First, the number of the shocks in Study 1 was doubled compared to Study 2 (32 vs. 16) and to our previous studies with between designs (Andreatta et al., 2010, 2012). Second, in Study 1 half of the shocks were reliably predictable, whereas the other half was delivered unpredictably before the BACKWARDCS+, while in Study 2 all USs were fully predictable. In the laboratory, fear responses can be induced by increasing the frequency of the shocks (shock density) together with their imminence (i.e., how reliably the danger is foreseen; Fanselow and Lester, 1988). Presumably, the high number of shocks in Study 1 together with their relative unpredictability might have induced an enhanced state of sustained fear (Davis et al., 2010) which caused the lack of discriminative startle responses to the CSs. In line, unpredictability, defined as "*the absence of a signal for an aversive event*" (Fonteyne et al., 2010), induces stronger fear responses to the aversive event than when it is predictable (Carlsson et al., 2006; Baratta et al., 2007; Herry et al., 2007; Fonteyne et al., 2010), a reduced capacity to identify safety periods (Lohr et al., 2007) and a sustained state of apprehension (sustained fear or anxiety; Fanselow and Lester, 1988; Davis et al., 2010). Moreover,the shock density seems also to play a role. In fact, we found

<sup>1</sup>Explorative we compared the subjective ratings, the startle response, and the SCR to the forward CS+ after conditioning of Study 1 vs. Study 2. We found no significant differences for SCR, startle response, valence and contingency ratings (*p*s > 0.07). However, forward CS+ in Study 2 was significantly more arousing compared to forward CS+ in Study 1 [*t*(55) = 3.001, *p* = 0.004]. Supposedly, when a stimulus' reliability regarding a threat is partial, the arousal is spread over the other stimuli too as possible informers about the threat.

relief-conditioned responses to a stimulus in the between-subjects designed studies (Andreatta et al., 2010, 2012), but not in Study 1 despite in both studies the US was presented unpredictably. Possibly, doubling the number of the aversive US may also have increased the anxiety-related responses. Hence, the unpredictability together with the high frequency of an aversive event seems to impair the ability to distinguish between threatening and safety periods.

Notably, these findings broaden our knowledge about the role of (un-)predictability of an aversive event in determining fear vs. safety conditioned responses. In fact, we could demonstrate for the first time that the unpredictability of an aversive event not only implies a sustained feeling of fear and an incapacity to identify the absence of a threat (i.e., respite), but it may also erase the ability to identify the termination of the threat (i.e., relief). That is, the stronger a sustained feeling of fear (or anxiety) is, the less evident the appetitive feeling of relief becomes. Safety is functionally related to danger meaning that an individual can identify safety periods only if it has first located the danger (Lohr et al., 2007). Based on this safety/danger relation, Lohr et al. (2007) distinguished between two kinds of safety, namely the absence of threat (i.e., respite) and the termination of threat (i.e., relief). Interestingly, a fruitless search for safety has been implicated in the etiology of anxiety disorders (Seligman, 1968; Mineka and Zinbarg, 2006; Lohr et al., 2007; Grillon et al., 2008, 2009; Davis et al., 2010), and anxious individuals have been found to be particularly sensitive to unpredictable threats (Grillon et al., 2008, 2009; Davis et al., 2010; Glotzbach-Schoon et al., 2013). Hence, individuals who show exaggerated fear responses to threatening contexts (Grillon et al., 2008, 2009; Davis et al., 2010) are less able to identify safety periods (Seligman, 1968; Mineka and Zinbarg, 2006; Lohr et al., 2007; Grillon et al., 2008, 2009; Davis et al., 2010), but they might also be less able to experience relief—as suggested by the present studies. However, further studies have to investigate the role of trait anxiety in the modulation of the relief-related responses in order to clarify whether and how the relief after an aversive event is implicated in the etiology of anxiety disorders.

Besides startle response, SCR is frequently used as physiological measure of conditioned fear (Büchel et al., 1998; Labar et al., 1998; Knight et al., 2003, 2006; Delgado et al., 2006; Weike et al., 2008). SCR reflects a phasic change in sweat gland activity induced by a re-orientation of the attentional resources toward novel and salient stimuli (Williams et al., 2000; Bradley et al., 2001; Bradley, 2009). In line with previous studies, both studies

#### **REFERENCES**


events establish distinct memories requiring fear- and reward network. *Learn. Mem.* 19, 518–526. doi: 10.1101/lm. 026864.112

Andreatta, M., Mühlberger, A., Yarali, A., Gerber, B., and Pauli, P. (2010). A rift between implicit and explicit conditioned valence after painrelief learning in humans. *Proc. Biol. Sci.* 277, 2411–2416. doi: 10.1098/rspb.2010.0103

presented here found increased autonomic arousal to the danger signal (FORWARDCS+) compared to the safety signal (the CS−; Büchel et al., 1998; Labar et al., 1998; Knight et al., 2003, 2005, 2006; Weike et al., 2008; Alvarez et al., 2011), and these fear conditioned SCRs extinguished throughout the test phase. In parallel, the SCRs to the stimulus associated with the relief "signal" (BACKWARDCS+) were significantly reduced compared to the danger signal and even significantly reduced compared to the safety signal (CS−). The latter difference suggests that the processes triggered by a relief-associated stimulus differ from those underlying the processes of a stimulus signaling safety. To our knowledge, there is no evidence in the literature investigating the effects of conditioned pain relief on autonomic arousal, which makes the interpretation of our results quite difficult. Nevertheless, Leknes et al. (2008) showed that the electrodermal responses following a painful stimulus linearly decreased by the increase of painfulness. Furthermore, another possible explanation of the decreased SCR to the relief-associated stimulus might be linked to the imminence of the threat. Namely, the defensive pattern is determined by three stages defined on the imminence of threat (Fanselow, 1994). Thus, flight/fight responses are initiated by the physical contact with the threat, while the level of fear gradually decreases by danger detachment. Considering SCR as an index of physiological arousal, the low SCR in response to the BACKWARDCS+ during conditioning presumably relies on the evident termination or detachment of the painful stimulation (i.e., the US) and the no-need to initiate defensive responses.

In conclusion, our results concur with the growing evidences on the appetitive properties of pain relief and its conditionability. Importantly, the predictability and the cognitive appraisal of the association between two stimuli crucially affect the explicit aversiveness and pleasantness of the relief-associated stimulus. Thus, as soon as the danger (US) is reliably predicted by a stimulus (FORWARDCS+), another stimulus presented upon the termination of danger (BACKWARDCS+) can acquire not only implicit but also explicit appetitive properties linked to the experienced relief.

#### **ACKNOWLEDGMENTS**

We sincerely thank C. Ziegler und S. Meinecke for their help in collecting data. This work was supported by the Collaborative Research Center "Fear, Anxiety, and Anxiety Disorders," SFB-TRR 58, project B1, and the DFG-FOR605 "Emotion and Behaviour," and the University of Wuerzburg in the funding program Open Access Publishing.


perspective of emotion theory. *Am. Psychol.* 55, 1247–1263. doi: 10.1037/0003-066X.55.11.1247 Blumenthal, T. D., Cuthbert, B. N., Filion, D. L., Hackley, S., Lipp, O. V., and Vanboxtel, A. (2005). Committee report: guidelines for human startle eyeblink electromyographic studies. *Psychophysiology* 42, 1–15. doi: 10.1111/j.1469-8986.2005. 00271.x


striatum and amygdala in associative learning. *Nat. Neurosci.* 14, 1250–1252. doi: 10.1038/ nn.2904


from simultaneous fMRI and skin conductance recording. *Neuroimage* 14, 1070–1079. doi: 10.1006/nimg.2001.0904

Yarali, A., Niewalda, T., Chen, Y.- C., Tanimoto, H., Duerrnagel, S., and Gerber, B. (2008). Pain relief learning in fruit flies. *Anim. Behav.* 76, 1173–1185. doi: 10.1016/j.anbehav.2008.05.025

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 02 May 2013; accepted: 30 August 2013; published online: 24 September 2013.*

*Citation: Andreatta M, Mühlberger A, Glotzbach-Schoon E and Pauli P (2013) Pain predictability reverses valence ratings of a relief-associated stimulus. Front. Syst. Neurosci. 7:53. doi: 10.3389/fnsys. 2013.00053*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2013 Andreatta, Mühlberger, Glotzbach-Schoon and Pauli. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## *Apis mellifera* octopamine receptor 1 (AmOA1) expression in antennal lobe networks of the honey bee (*Apis mellifera*) and fruit fly (*Drosophila melanogaster*)

#### *Irina T. Sinakevitch1 \*, Adrian N. Smith1,2, Fernando Locatelli 3, Ramon Huerta4, Maxim Bazhenov5 and Brian H. Smith1 \**

*<sup>1</sup> School of Life Sciences, Arizona State University, Tempe, AZ, USA*

*<sup>2</sup> Mathematical, Computational and Modeling Sciences Center, Arizona State University, Tempe, AZ, USA*

*<sup>3</sup> Laboratorio de Neurobiología de la Memoria, Departamento de Fisiología, Biología Molecular y Celular, Facultad de Ciencias Exactas y Naturales, IFIByNE CONICET, Universidad de Buenos Aires, Buenos Aires, Argentina*

*<sup>4</sup> BioCircuits Institute, University of California San Diego, La Jolla CA, USA*

*<sup>5</sup> Department of Cell Biology and Neuroscience, University of California, Riverside, CA, USA*

#### *Edited by:*

*Dave J. Hayes, University of Toronto, Canada*

#### *Reviewed by:*

*Jürgen Rybak, Max Planck Institute for Chemical Ecology, Germany Kyung-An Han, University of Texas at El Paso, USA Martina Wicklein, Imperial College London, UK*

#### *\*Correspondence:*

*Irina T. Sinakevitch and Brian H. Smith, School of Life Sciences, Arizona State University, ISTB-1 bldg, 427 Est Tyler Mall, Tempe, AZ 85287-4501, USA e-mail: isinakev@asu.edu; brianhsmith@asu.edu*

Octopamine (OA) underlies reinforcement during appetitive conditioning in the honey bee and fruit fly, acting via different subtypes of receptors. Recently, antibodies raised against a peptide sequence of one honey bee OA receptor, AmOA1, were used to study the distribution of these receptors in the honey bee brain (Sinakevitch et al., 2011). These antibodies also recognize an isoform of the AmOA1 ortholog in the fruit fly (OAMB, mushroom body OA receptor). Here we describe in detail the distribution of AmOA1 receptors in different types of neurons in the honey bee and fruit fly antennal lobes. We integrate this information into a detailed anatomical analysis of olfactory receptor neurons (ORNs), uni- and multi-glomerular projection neurons (uPNs, and mPNs) and local interneurons (LNs) in glomeruli of the antennal lobe. These neurons were revealed by dye injection into the antennal nerve, antennal lobe, medial and lateral antenno-protocerbral tracts (m-APT and l-APT), and lateral protocerebral lobe (LPL) by use of labeled cell lines in the fruit fly or by staining with anti-GABA. We found that ORN receptor terminals and uPNs largely do not show immunostaining for AmOA1. About seventeen GABAergic mPNs leave the antennal lobe through the ml-APT and branch into the LPL. Many, but not all, mPNs show staining for AmOA1. AmOA1 receptors are also in glomeruli on GABAergic processes associated with LNs. The data suggest that in both species one important action of OA in the antennal lobe involves modulation of different types of inhibitory neurons via AmOA1 receptors. We integrated this new information into a model of circuitry within glomeruli of the antennal lobes of these species.

#### **Keywords: biogenic amine receptors, G-protein receptors, octopamine, learning and plasticity, olfactory pathways**

## **INTRODUCTION**

Many studies have demonstrated that honey bees (*Apis mellifera*) and fruit flies (*Drosophila melanogaster*) can associate odors with food reinforcement (Menzel and Muller, 1996; Page et al., 1998; Scheiner et al., 2001; Keene and Waddell, 2007). These studies used sucrose reinforcement as a means of conditioning animals to respond to and discriminate among odors (Duerr and Quinn, 1982; Menzel and Muller, 1996; Menzel et al., 1999; Scheiner, 2004; Scheiner et al., 2004). Based on these learning capabilities, sensory information about sucrose reinforcement should be represented in some way by neural circuitry in honey bee and fruit fly brains (Acevespina et al., 1983; Wright et al., 2007; Engel and Wu, 2009; Cevik and Erden, 2012). Many different areas of these brains receive input from a set of ventral unpaired median (VUM) neurons with cell bodies located on the ventral midline of maxillary and mandibular neuromeres in the subesophageal ganglion (Kreissl et al., 1994; Sinakevitch et al., 2005; Sinakevitch and Strausfeld, 2006; Schröter et al., 2007; Busch et al., 2009). When stimulated, VUM neurons release the biogenic amine octopamine (OA) broadly throughout areas of the brain that are important for learning associations between many different types of stimuli, including odors and food rewards (Hammer, 1993; Hammer and Menzel, 1998).

The broadly projecting morphological structure of VUM neurons, in conjunction with several physiological and molecular studies (Hammer, 1993; Schröter et al., 2007), makes VUM neurons likely candidates for representing sucrose via OA release. In the honey bee, at least two of these neurons—VUMmx1 and VUMmd1—each have a primary neurite that projects through the midline tract and gives rise to two symmetrical secondary axons that send collaterals to the antennal lobes, lateral horn (LH), lateral protocerebral lobe (LPL), and to the mushroom body (MB) calyces (**Figure 1**). In the fruit fly, the OA-VUMa2 neuron is similar to the honey bee VUMmx1 (Sinakevitch et al., 2005; Sinakevitch and Strausfeld, 2006; Busch et al., 2009). OA-VUMa2 sends secondary neurites to the posterior margins of

**FIGURE 1 | Schematic view of the main olfactory pathways in the honey bee brain (based on Fonta et al., 1993; Abel et al., 2001; Strausfeld, 2002; Sinakevitch et al., 2005, 2011; Kelber et al., 2006; Kirschner et al., 2006; Schröter et al., 2007; Girardin et al., 2013).** Olfactory Receptor Neuron (ORN) axons from the antenna enter through the antennal nerve (AN) into the antennal lobe (ant lobe) and converge onto the outer cortex of glomeruli. Each glomerulus is innervated by processes of several types of neurons. Uniglomerular projection neurons (uPNs magenta and green) have dendritic branches in a single glomerulus and send axons to higher-order brain centers such as the MB calyces (ca), lateral protocerbral lobe (LPL), and lateral horn (LH). There are two uniglomerular antenno-protocerebral tracts, the l-APT (green) and m-APT (magenta), which reflect the segregation of glomeruli into rostral (l-APT) and caudal (m-APT) hemispherical clusters. Kenyon cells (not shown in the figure) are the intrinsic cells that make up the MB. Kenyon cell dendrites form pairs of calyces (ca) with specific zones that receive different

the antennal lobes, where it projects fine ramifications into each glomerulus. OA-VUMa2 axons also follow the medial antenno-protocerbral tracts (m-APT) and connect the antennal lobe with the calyx and LH, while in the honey bee VUM axons follow the lateral antenno-protocerbral tracts (l-APT) and project to the LPL and MB calyx (Sinakevitch and Strausfeld, 2006; Busch et al., 2009).

In spite of the evidence that VUM neurons and OA are critical for appetitive learning in neural networks of the insect brain, very little is known about how OA receptors are integrated into those neural networks to drive these associations. We focus on the actions of VUM and OA on associative plasticity reported in the networks of the antennal lobe of honey bees and fruit flies (Fernandez et al., 2009; Kim et al., 2013). The honey bee antennal lobe is made up of approximately 160 glomeruli (Galizia et al., 1999; Robertson and Wanner, 2006), where axons from on average 400 olfactory receptor neurons (ORNs) converge onto 5–6 uniglomerular projection neurons (uPNs), assuming ∼65,000 ORNs reported by Esslen and Kaissling (1976) and ∼920 uPNs reported by Rybak (2012) (**Figure 1**) (Kelber et al., 2006; Kirschner et al., 2006; Nishino et al., 2009). In comparison, there are 56 glomeruli in the antennal lobe of adult fruit flies, which give rise to 150 uPNs (approximately 3/glomerulus) (Stocker, 2001; Laissue and Vosshall, 2008; Tanaka et al., 2012a). Glomeruli are functional coding units. In the fruit fly, as is also types of inputs, the lip, collar (co), and basal ring (br). The lip and br are innervated by uPN axons. The Kenyon cell axons project ventrally and split to form medal (M), vertical (V), and γ lobes, which are the main output regions of the MB. Protocerebral tract neurons (PCTN) receive inputs in the lobes and provide GABAergic feedback to the calyces. Multiglomerular PNs project axons via the medio-lateral protocerebral tracts (ml-APT 1,2) to the LPL and LH. At least two types of local interneurons (LN) interconnect glomeruli within the AL, homogeneous LNhomo (yellow), and heterogeneous LNhet (red). Ventral unpaired median neurons (VUM) have cell bodies in the maxillary (VUMmx1 is shown) and mandibullar neuromeres of the subesophageal (SEG) ganglion and connect gustatory processing in the SEG to all antennal lobe glomeruli, the LPL, LH, and MB calyces. CB, central body; M, medial lobe; V, vertical lobe; γ, gamma lobe; lo, lobula; me, medulla; m, median; l, lateral; r, rostral; c, caudal. Cell types in each side of the brain are bilaterally symmetric, but for clarity different cells are shown in each half. Scale bar: 250 μm.

likely but not yet demonstrated in the honey bee, ORNs that converge to a glomerulus express the same receptors, which defines the range of odor ligands that activate the glomerulus (Laissue and Vosshall, 2008).

The structure of glomeruli is similar in both species. Each glomerulus has two distinct areas: the cortex, which contains ORN axon terminals, and the core, which lacks ORN arborizations (Fonta et al., 1993; Hummel and Zipursky, 2004; Tanaka et al., 2012a). In both species, PNs leave the antennal lobe through three main output pathways called the antennoprotocerebral tracts (APTs), named from Galizia and Rössler (2010) (Stocker et al., 1990; Fonta et al., 1993; Abel et al., 2001; Kirschner et al., 2006; Tanaka et al., 2008, 2012a). The l-APT connects the antennal lobe with LH, LPL and calyx of the MB. The m-APT connects the antennal lobe with the MB calyx, LPL and LH. And finally the medio-lateral APT (ml-APT) connects the antennal lobe with the LPL and LH. Three major subtracts of the ml-APT have been described in the honey bee (ml-APT1, ml-APT2, ml-APT3; Kirschner et al., 2006). Different types of local interneurons (LNs) interconnect glomeruli (Schafer and Bicker, 1986; Fonta et al., 1993; Olsen et al., 2007b; Shang et al., 2007; Seki et al., 2010; Meyer and Galizia, 2012; Girardin et al., 2013). GABAergic LNs are the largest group in both species (Schafer and Bicker, 1986; Ng et al., 2002; Okada et al., 2009; Seki et al., 2010). Two types of GABAergic LNs have been described based on different branching patterns in fruit fly antennal lobe glomeruli: LN1 (arborizations only in the core of the glomeruli) and LN2 (arborization in core and cortex regions)(Okada et al., 2009). In the honey bee, heterogeneous LNs are distinguished from homogeneous LNs by dense branching processes in one of the invaded glomeruli (**Figure 1**) (Fonta et al., 1993; Meyer and Galizia, 2012; Girardin et al., 2013). In addition, multiglomerular projection neurons (mPNs) connect the antennal lobe with the LPL and LH through the ml-APTs (Fonta et al., 1993; Abel et al., 2001; Tanaka et al., 2008; Okada et al., 2009; Seki et al., 2010).

In our previous study of the honey bee, we characterized antibodies against one type of OA receptor—AmOA1—and used them to demonstrate expression of AmOA1 in the inhibitory neurons of the antennal lobe and MB neuropil (Sinakevitch et al., 2011). In Sinakevitch et al. (2011), we also showed that anti-AmOA1 antibodies recognize the orthologous fruit fly OAMB receptor which is an important part of the reinforcement pathway for appetitive learning in the fruit fly (Han et al., 1998; Kim et al., 2013).

Here we extend our earlier study to describe in detail the morphology and neural circuitry of the antennal lobe. We show specifically how GABAergic processing in these networks is targeted by OA via AmOA1. The GABAergic system in the fruit fly has recently been extensively studied and described in detail elsewhere (Okada et al., 2009; Seki et al., 2010). We show strong similarities in expression of AmOA1 across the GABAergic targets in the antennal lobes of both species, which most likely reflects a conserved phylogenetic modulatory mechanism in olfactory networks. Finally, we use this information to propose a model for modulation of information processing in networks of the fruit fly and honey bee antennal lobes.

## **MATERIALS AND METHODS**

#### **ANIMALS**

Honey bees (*Apis mellifera*) were adult New World Carniolan pollen foragers from colonies maintained at Arizona State University. Fruit fly *(Drosophila melanogaster)* stocks and crosses were maintained at 22◦C on a standard corn meal-yeast-agar medium supplemented with methyl-4-hydroxy-benzoate as a mold protector. The following strains were used: wild-type Oregon R; UASmcd8::GFP, used to express cell surface-associated GFP (Lee et al., 1999); GH146-GAL4, used as a marker of the projection neurons in antennal lobes of *Drosophila* (Stocker et al., 1997; Marin et al., 2002; Jefferis et al., 2007) and the APL neuron in the MB (Liu and Davis, 2009); and Or83b-GAL4, used as a marker of ORNs (Larsson et al., 2004). These strains were kindly provided by Dr. A. Fiala and Dr. T. Riemensperger (University of Wurzburg).

#### **DYE INJECTION**

Honey bee pollen foragers were collected at the entrance of the hive, briefly cooled, and restrained in individual harnesses. After recovering from cooling, honey bees were fed with 1 M sucrose solution and left undisturbed for 1–6 h before injection. Heads of the bees were fixed to the stage with soft dental wax (Kerr, Sybron Dental Specialties, Orange, CA, USA) in a way that allowed free movement of antennae and proboscis. A dissection knife was used to cut a window in the head capsule, dorsal to the joints of the antennae and rostral to the medial ocellus. The large pharyngeal glands were carefully moved until the MB vertical lobes [Strausfeld (2002) or alpha lobes in Rybak and Menzel (1993)] were visible, which are easily recognizable and serve as spatial reference for staining (Sachse and Galizia, 2002). The tip of a glass electrode coated with Rhodamine-dextran (Invitrogen, Grand Island, NY, USA) or with neurobiotin (Vector Laboratories, Burlingame, CA, USA), both prepared in 3% bovine serum albumin (BSA) solution (Sigma-Aldrich, St. Louis, MO, USA), was inserted into both sides of the protocerebrum rostrolateral to the vertical lobes, aiming for both l-APT and m-APT, which contain the axons of uniglomerular (u)PNs (Abel et al., 2001). In order to reveal all APTs, the dye was deposited directly into the coarse area of the antennal lobe. The glass tip was held in this position until the dye bolus dissolved in the tissue (∼3– 5 s). The window was subsequently closed using the same piece of cuticle that was previously removed. Eicosane was used to glue and seal the cuticle. Immediately afterward, one of the antennae was cut transversally at approximately the middle of the scapus. A glass electrode coated with Rhodamine-dextran or neurobiotin (the respective tracer that was not used for the PNs in the same animal) was inserted into the antenna through the opened cavity and the electrode was rotated and moved until the coating was completely dissolved in the lumen of the antennae. The electrode was removed and the antenna was sealed with eicosane.

The next day, the piece of cuticle covering the brain was removed. Glands and trachea covering the brain were removed and the brain was rinsed with Ringer solution (130 mM NaCl, 6 mM KCl, 4 mM MgCl2, 5 mM CaCl2, 160 mM sucrose, 25 mM glucose, 10 mM HEPES, pH 6.7, 500 mOsmol; all chemicals from Sigma-Aldrich). For simultaneous staining with anti-synapsin or goat anti-AmOA1 antibodies, the brain was dissected and immediately fixed in 4% paraformaldehyde (Sigma-Aldrich) in phosphate buffer saline (PBS, pH 7.4) made from tablets (Sigma-Aldrich). For simultaneous staining with anti-GABA and goat anti-AmOA1 antibodies, the fixative was a mixture containing 1.5% glutaraldehyde [Electron Microscopy Sciences (EMS), Hatfield, PA, USA] and 2.5% paraformaldehyde (EMS) in 0.1 M sodium cacodylate buffer (EMS, pH 7.0), with 1% sodium metabisulfite (SMB, Sigma-Aldrich).

#### **INTRACELLULAR STAINING**

For intracellular staining, the bee was mounted in the harness as described above and was alive during injections. Thin-walled borosilicate electrodes (resistance of 75–95 M-) with internal filament were used to stain one of the uPNs that was visualized by injection of Alexa-488-dextran 3000 (Invitrogen) into the m-APT as described above. Electrode tips were filled with a mixture of 7% neurobiotin and lysine fixable Rhodamine-dextran 3000 (Invitrogen) in 2 M potassium acetate (Vonhoff and Duch, 2010). To prevent dye dilution, an air bubble was left between the tip and the shaft. After intracellular penetration of the PN, the dye was injected iontophoretically by applying constant depolarizing current of 0.5 nA amplitude for 10–12 min. Subsequently, the electrode was removed and the brain was dissected out from the head capsule for fixation in 4% paraformaldehyde in PBS. Preparations were washed 6 × 30 min in PBS with 0.5% Triton X-100 (PBST), pH 7.4, then incubated with Streptavidin-Cy3 (Jackson ImmunoResearch Laboratories, West Grove, USA) to reveal neurobiotin in the cell.

#### **IMMUNOCYTOCHEMISTRY**

Mouse monoclonal anti-synapsin antibodies (SYNORF1; clone 3C11) were raised against bacterially expressed fruit fly synapsin. The anti-synapsin antibodies were kindly provided by E. Buchner, University of Wurzburg, Germany. These antibodies, which recognize presynaptic sites of neurons, were used here as a marker for synaptic neuropil in the antennal lobe of the honey bee. We employed the protocol originally used for studies of the honey bee brain with these antibodies (Brandt et al., 2005).

GABA antiserum (GEMAC, Talence, France) was raised in rabbits using GABA conjugated by glutaraldehyde to BSA, bovine hemoglobin, or poly-L-lysine. Antiserum specificity has been described elsewhere (Seguela et al., 1984; Sinakevitch et al., 1996, 2003, 2011; Strausfeld et al., 2003; Sinakevitch and Strausfeld, 2004). We used affinity purified goat anti-AmOA1 antibodies to describe the AmOA1 receptor distribution in the honey bee brain. Antibody specificity and staining controls in the bee brain were described in detail previously (Sinakevitch et al., 2011).

To immunostain the AmOA1 receptor ortholog in *Drosophila*, we used an AmOA1 antiserum from rabbit, which was previously used to study the distribution of AmOA1 receptors in the honey bee brain (Sinakevitch et al., 2011). This antiserum recognizes at least one isoform of the OA1 receptor in fruit fly: DmOA1A (CG3856-PB), the alternatively spliced isoform of the *Dmoa1* gene, which is identical to the OAMB (CG3856) receptor (Han et al., 1998; Balfanz et al., 2005; Maqueira et al., 2005). The specificity of these antibodies was demonstrated on oamb96 mutant flies, which lack part of the genomic region for oamb alleles (Lee et al., 2003), and is described in Sinakevitch et al. (2011, Figure S1).

## *Double-staining anti-synapsin and Rhodamine-dextran (or neurobiotin) in the honey bee antennal lobe (n* **=** *6, Table 1)*

Following injection with Rhodamine-dextran (or neurobiotin) into the antenna, brains were fixed overnight at 4◦C in 4% paraformaldehyde in PBS, then washed 6 × 20 min in PBST. After washing in PBST, brains were embedded in 8% agarose and 60μm agarose sections were made using a Leica vibrating blade microtome VT1000S (Leica Biosystems, Germany). The sections were pre-incubated with 5% normal donkey serum (Jackson ImmunoResearch Laboratories) and the anti-synapsin antibodies were added to the sections in the dilution 1:10 and incubated overnight at room temperature. The sections were then incubated with PBST 6 × 20 min and secondary antibodies F(ab )2 fragments of donkey anti-mouse IgG conjugated to Cy2 (Jackson ImmunoResearch Laboratories) were applied to reveal the synapsin staining (dilution 1:250).

## **Table 1 | Summary of all preparations used in this study.**


*aInjection site was not precise; occasionally the l-APT tract was included.*

*bThese tracts were the primary focus for our study.*

*cTotal number of preparations used for observations and conclusions in our study.*

*dDefined as getting fills in the targeted tracts.*

*eIn these preparations, we made injection in the antenna to label ORNs.*

In preparations, whenever neurobiotin tracer was used in the antenna, a 1:250 dilution of Streptavidin-Cy3 was added during the incubation of secondary antibodies. Then preparations were thoroughly washed in PBS and embedded in 80% glycerol.

## *Anti-GABA staining (n* **=** *10)*

Honey bee brains were removed in fixative containing 1.5% glutaraldehyde and 2.5% paraformaldehyde in 0.1 M sodium cacodylate buffer with 1% SMB. After fixation overnight at 4◦C, whole brains were incubated for 15 min in 0.05 M Tris-HCl-SMB buffer pH 7.5 containing 0.5% NaBH4. After washing in 0.05 M Tris-HCl-SMB buffer, brains were embedded in 8% agarose and separate brains were cut into sections 35, 40, or 60 μm thick. After washing in 0.05 M Tris-HCl-SMB buffer with 0.5% of Triton X100 (TX), pH 7.5, sections were pre-incubated with 5% normal donkey serum for 1 h. Then anti-GABA antibodies were added to brain sections in a dilution 1:1000 in 0.05 M Tris-HCl-SMB-TX, for overnight incubation at room temperature. After washing in 0.05 M Tris-HCl-TX, pH 7.5, F(ab )2 fragments of donkey anti-rabbit antibodies conjugated to either Cy3 or Cy5 (Jackson ImmunoResearch Laboratories, diluted 1:250 in 0.05 M Tris-HCl-TX) were used as secondary antibodies overnight at room temperature. After a final wash in 0.05 M Tris-HCl buffer pH 7.5, the sections were mounted on slides in 80% glycerol.

## *Anti-GABA staining after neurobiotin injection into the honey bee brain (n* **=** *65, Table 1)*

Following dye injections, brains were treated as described in the previous section. To reveal neurobiotin, after incubation with anti-GABA primary antibodies, Streptavidin-Cy2 (dilution 1:250, Jackson ImmunoResearch Laboratories) was added to the solution together with secondary antibodies conjugated to Cy3 or Cy5 as described above.

In control preparations, where anti-GABA was omitted, the secondary antibodies did not show any detectable staining. Sections of the honey bee brain without neurobiotin also did not show any detectable staining after incubation with both secondary antibodies and Streptavidin-Cy2 (data not shown).

## *Triple-staining with anti-AmOA1, anti-GABA and neurobiotin injected brains (n* **=** *26, Table 1)*

For simultaneous staining of AmOA1 and GABA in neurobiotin injected brains, brains were fixed and processed as described for anti-GABA staining. The brain sections were preincubated with 5% normal donkey serum for 1 h and, then simultaneously with both primary antibodies: goat anti-AmOA1 (1:1000) and rabbit anti-GABA (1:1000) overnight at room temperature. After a thorough wash in 0.05 M Tris-HCl-TX, secondary antibodies, F(ab )2 fragments of donkey anti-goat IgG-Cy2 and F(ab )2 fragments of donkey anti-rabbit IgG-Cy5 together with Streptavidin-Cy3 (all from Jackson ImmunoResearch Laboratories), were added overnight in dilution 1:200. The appropriate controls for goat anti-AmOA1 and anti-GABA stainings were described in detail previously (Sinakevitch et al., 2011). As controls for the specificity of the secondary antibodies, all secondaries were incubated with sections that had only one of the primary antibodies. The staining did not show any cross-reaction between the secondary antibodies and Streptavidin-Cy3. Streptavidin-Cy3 did not interact with any structure in the absence of the neurobiotin in the bee brain.

## *Triple staining with anti-AmOA1, neurobiotin and Rhodamine-dextran injected brains (n* **=** *6, Table 1)*

After Rhodamine-dextran injection, brains were fixed in 4% paraformaldehyde in PBS overnight, then washed 6 × 20 min in PBST. After washing in PBST, brains were embedded in the 8% agarose and 60μm brain sections were made. The sections were pre-incubated with 5% normal donkey serum for 1 h and then goat anti-AmOA1 antiserum (1:200) was added for incubation overnight at room temperature. The secondary antibodies were F(ab )2 fragments of donkey anti-goat IgG conjugated to Cy2. Streptavidin-Cy5 (Jackson ImmunoResearch Laboratories) was used to reveal neurobiotin (dilution 1:250). Preparations were then thoroughly washed in PBS and embedded in 80% glycerol.

## *Anti-synapsin and anti-GFP staining in fruit flies (n* **=** *5)*

Whole heads of Or83b-GAL4; UAS-msd8-GFP were fixed in 4% paraformaldehyde in PBS. The eye and edges of the head capsules were cut off for rapid fixation of the brain. Preparations were embedded in 8% agarose and cut into 50μm thick sections. Brain sections were washed 6 × 20 min in PBST, and preincubated for 1 h in 5% normal goat serum in PBST. The anti-synapsin (1:10) and anti-GFP chicken polyclonal antibodies (1:1000) (abcam, Cambridge, UK) were applied simultaneously to sections for overnight incubation at room temperature. The primary antibodies were diluted in PBST. After washing in PBST 6 × 1 h, secondary antibodies F(ab )2 fragments of goat anti-mouse IgG conjugated to Cy5 (Jackson ImmunoResearch Laboratories) and Alexa 488 goat anti-chicken IgG (Invitrogen), diluted in PBST (1:250) were applied for 6 h at room temperature. After final washing in PBST 6 × 1 h, sections were mounted in 80% glycerol.

## *Anti-GABA and anti-AmOA1 in fruit flies (n* **=** *10)*

Whole heads were placed in fixative containing 2% glutaraldehyde in 0.1 M sodium cacodylate buffer with 1% SMB, pH 7.0. After fixation, semi-opened brains were incubated for 15 min in 0.05 M Tris-HCl-SMB buffer pH 7.5 containing 0.5% NaBH4 to saturate double bonds. After washing in 0.05 M Tris-HCl-SMB buffer, heads were embedded in 8% agarose and cut into 50μm sections. To do the co-localization study of anti-AmOA1 with anti-GABA, both of which were raised in rabbit, we first did staining on consecutive sections: one section was labeled with anti-GABA and anti-GFP and the section next to it with anti-AmOA1 and anti-GFP. In order to obtain anti-GABA staining in one section and anti-AmOA1 in an adjacent section, we separated the sections of one brain in two wells of a 24-well nunc plate: odd sections in one well and even sections in a second well. After washing in 0.05 M Tris-HCl-SMB-TX, sections were preincubated with 5% normal swine serum (Dakopatts a/s, Glostrup, Denmark) for 1 h. "Odd" agarose brain sections were then incubated with anti-GABA antiserum (1:1000) and chicken polyclonal to GFP (1:1000), and "even" agarose brain sections were incubated with anti-AmOA1 and chicken polyclonal to GFP (1:1000) overnight. Primary antibodies were diluted in 0.05 M Tris-HCl-SMB-TX. After washing in 0.05 M Tris-HCl-TX, Alexa 488 goat anti-chicken IgG and goat anti-rabbit antibodies conjugated to Alexa 555 (Invitrogen) or Cy5 (Jackson ImmunoResearch Laboratories) diluted 1:250 in 0.05 M Tris-HCl-TX were used as the secondary antibody overnight. After a final wash in 0.05 M Tris-HCl, sections were embedded in 80% glycerol. The staining sections were compared and then AmOA1 staining sections were processed for anti-GABA staining. Sections labeled with anti-AmOA1 (and anti-GFP) were detached from slides, washed in PBS, and postfixed in 4% paraformaldehyde in PBS with 1% SMB for 20 min in order to deactivate antibodies of the first sequence of staining, and then processed for anti-GABA staining. We monitored two consecutive brain sections and used them as controls for double staining, where 0.05 M Tris-HCl buffer with 5% swine serum replaced GABA antiserum or AmOA1 antiserum. In these controls, we did not observe any interactions between antisera.

#### **CONFOCAL MICROSCOPY**

Digital images were captured with a Leica TCS SP2 or TCS SP5 confocal laser scanning microscope (Leica, Bensheim, Germany) using a Leica HCX PLAPO CS 40\_ oil-immersion objective (numerical aperture: 1.25) or a Leica HCX PLAPO CS 100\_ oil-immersion objective (numerical aperture 1.40) with appropriate laser and filter combinations. Stacks of optical sections at 1μm spacing were processed using Leica software. Size, resolution, contrast, and brightness of final images were adjusted with Adobe Photoshop software. Images are either a single slice or flattened confocal stacks (maximum intensity projections). Images of *Drosophila* were collected on a Zeiss LSM 510 confocal microscope (Carl Zeiss, Oberkochen, Germany). Groups of four to ten serial 0.5-μm optical sections (1, 024 × 1024 at 8-bit color depth) were scanned using 40 × 1.0 or 63 × 1.4 oil iris Plan-Apochromat objectives. Images were stored as TIFF files and edited in Adobe Photoshop CS2.

#### *Notes on immunostaining and injections (see Table 1 for a summary of the preparations used in this study)*

Immunostaining for GABA and AmOA1 was best in the honey bee when animals were sacrificed immediately after collection. During optimization of the co-staining procedures by dye/or tracer with immunostaining for GABA and AmOA1, we found that it was crucial for animals to be alive and able to respond to stimuli (e.g., extension of the proboscis to food by honey bees) before being sacrificed. Nearly dead or dead animals nevertheless gave us good results for neuroanatomy of injected cells and processes, but immunostainings for GABA and AmOA1 were difficult to interpret, especially in glomeruli, for multiple reasons: background staining was too high (no differences in staining between cells and neuropil structures), staining was patchy and inconsistent, staining was absent, or the antennal lobe was obviously deformed. In **Table 1**, we summarized only preparations that had good quality immunostaining for neurobiotin injections.

In addition, staining for AmOA1 in particular was variable from animal to animal. This may reflect biologically meaningful variation in variables we did not control for in this study (e.g., age, caste, genotype, experience). However, we focus only on general patterns here.

Other difficulties that we found in our technique concern the way we deposited dye, especially in LPL, and LPL and LH; this gave us a very low yield of successful preparations even when the same group of neurons appeared in a sample. Different methods of injection should be used in the future to characterize mPNs in ml-APTs. For example, precise placement of injections with dye-filled microelectrodes in ml-APTs only, or use of mPNs markers (when available) would improve reliability of future results. Nevertheless, our preparations have been adequate for drawing conservative conclusions concerning GABAergic mPNs.

#### **RESULTS**

#### **ANATOMY OF GLOMERULI IN THE HONEY BEE ANTENNAL LOBE** *Olfactory receptor neuron (ORN) terminals define the structure of the glomerular cortex*

Antibodies against synapsin label all presynaptic sites (Klagges et al., 1996), and here they revealed synaptic connectivity in all glomeruli of the antennal lobe (**Figures 2A1,A3**). In order to reveal ORN endings in the glomeruli, Rhodamine-dextran was deposited into the antennal nerve, as illustrated schematically in **Figure 2B** (see methods for details). The dye was taken up by ORN axons, which enter the antennal nerve through four tracts (T1–T4; Abel et al., 2001; Kirschner et al., 2006). Staining revealed axon endings in the cortex of each glomerulus (cortical layer; or glomeruli rind **Figure 2A2**) but not in the core. This pattern is characteristic of glomeruli in the T1-T3 tracts. In contrast, glomeruli innervated by T4 tracts receive ORNs in the entire glomerulus (not shown here; Arnold et al., 1985; Galizia et al., 1999; Nishino et al., 2009; Kreissl et al., 2010). We limit the description of glomerulus anatomy below to the T1 to T3 tracts.

ORNs are highly enriched with synapsin and they have targets in the cortex (**Figures 2A2,A3**). Glomeruli are roughly "eggshaped" with the narrower end oriented toward the aglomerular area of the antennal lobe. uPNs enter the glomerulus through the aglomerular neuropil and have large fibers in the glomerulus core. In addition, they have fine branches extending into the cortex (**Figure 2A2** insert) where they overlap and presumably make synaptic contact with ORN axons. The ORNs homogeneously innervate the glomerulus making up the lateral walls (approximately 5–10μm thick) and a cap-like top (approximately 10–20μm thick, the portion of the cortex indicated with b in **Figure 2A2** insert). Sections where the glomerulus was cut through the midline (the absence of overlaying ORN fibers and presence of the large fibers of PNs in the core) were used to measure the length and width of core and cortex at the midline of the glomerulus. We estimate that, regardless of the size of the glomerulus, the ratio between the core and cortex areas measured at the midline of length was as follows: **b** (length of cortex in the midline)/**a** (length of glomerulus in the midline (core + cortex)) = 0.29 ± 0.05μm [*SD*, *n* = 16 glomeruli that have ORN fibers in the cortex (**Figure 2B** insert)].

**FIGURE 2 |** *Apis mellifera***.** General morphology of the antennal lobe and glomerulus. GABAergic neurons in the honeybee antennal lobe are local interneurons and multiglomerular PNs that branch into LPL and LP. **(A)** Frontal section of the honeybee antennal lobe immunostained with anti-synapsin antibodies (green) and Rhodamine-dextran labeled olfactory receptors neurons (magenta). **(A1)** Anti-synapsin (green) shapes the synaptic neuropil of the antennal lobe in all glomeruli, highlighting the potential synaptic connections between different types of neurons. **(A2)** Rhodamine-dextran injection into the antennal nerve revealed olfactory receptor neuron (ORN) endings (magenta) surrounding each glomerulus to form the cortex layer. The core area of glomerulus is free of ORNs. **(A3)** The merged images of the cortex area of the ORN endings in the antennal lobe overlapped with the area marked by anti-synapsin (white) indicate that ORNs synapse and receive synapses from other antennal lobe neurons in the cortex rind of glomerulus. Insert in **(A2)**: The schematic of the glomerulus overlaid on the middle section indicated by the image of the projection neuron dendrite (blue, PNs)

and ORNs (magenta). The distribution of PN dendrites in the core and cortex of the glomerulus in the section made through the midline of the glomerulus demonstrates that large axons of PNs are in the core area and fine dendrites of PNs in the cortex where they overlap with ORNs: b-the length of the cortex area in the center, a-the length of the glomerulus. **(B)** Schematic view of the olfactory pathways in the brain of the honey bee where arrows show the site of Rhodamine-dextran or/and neurobiotin injections. The octopaminergic neuron VUMmx1 has a cell body in the subesophageal ganglion (SEG) and carries information along the olfactory pathway from the antennal lobe to the lateral horn (LH), lateral protocerebral lobe (LPL), and the MB calyx (ca). The tracts that carry the uniglomerular PNs are l-APT and m-APT, while the two tracts for multiglomerular mPNs are ml-APT 1,2. **(C)** Double stainings of uPNs (magenta) and anti-GABA (green) in the antennal lobe demonstrate that uniglomerular PNs are not GABAergic. **(C1)** Injection into the l-APT and m-APT as indicated in **(B)** revealed uPNs with dendrites in *(Continued)*

#### **FIGURE 2 | Continued**

both core and cortex areas of glomeruli. **(C2)** Merged images of GABA (green) and uPNs (magenta) indicate that cell bodies of the uPNs are not GABAergic. White labeling in the glomerulus is due to overloaded dye in PNs and not co-localization, as illustrated in the image of the glomerulus with uPN dendrites and anti-GABA staining in the insert of **(C2)**. **(D)** Double labeling of the l-APT **(D1** single image) and anti-GABA (**D2** green single image) on the frontal section of the honey bee protocerebrum. The l-APT stained by injection in the antennal lobe **(D3)** The merged image illustrates that the two axons framing l-APT are GABAergic (white) and connect to the lateral protocerebrum (LH), another l-APT GABAergic fiber originating from antennal lobe branches in the lateral protocerbral lobe (LPL). Double arrows indicate the absence of anti-GABA in the uPN l-APT axons before their entry to the MB calyx. **(E)** Double staining of the m-APT and ml-APT revealed by injection of dye into LPL (single image **E1**, magenta in **E3**) and anti-GABA (green in **E1**and **E2**) **(E1)** uPNs in m-APT

#### *Structure of uniglomerular (u)PNs in the antennoprotocerebral tracts (APT)*

In order to reveal uniglomerular PNs, neurobiotin tracer was injected into an area between the MB calyx and the vertical lobe (**Figure 2B**) where both tracts (l-APT and m-APT) meet to send their axons in the MB calyx. Each glomerulus is innervated by an estimated 5–6 uPNs (Rybak, 2012), and, in contrast to the ORNs, the dendrites of these uPNs cover the core and cortical layer of each glomerulus homogeneously (**Figure 2C1**). The uPNs enter glomeruli from the aglomerular neuropil and form large branches (2–4μm) at the beginning of the core area. Fine arborizations of the uPN dendrites (thickness ranges between 0.2–0.7μm) are tightly packed in the cortex. Because they overlap with ORNs in the cortex, the fine arborizations of uPNs might be postsynaptic to the ORNs; however, direct synapses between ORNs and uPNs have not been conclusively shown.

#### *uPNs of the m- and l-APT are not GABAergic*

Co-staining of neurobiotin injected uPNs with anti-GABA antibodies revealed that uPNs do not exhibit GABA immunoreactivity in their cell bodies, dendrites, (**Figure 2C2**), or axons in l-APT (**Figures 2D1–D3**) and m-APT (**Figures 2E1–E3**, **4C–E**). The white color in a few glomeruli in **Figure 2C2** is due to close proximity of tightly packed uPN dendrites and GABAergic arborizations in low magnification. Higher magnification of a glomerulus, co-labeled with anti-GABA and neurobiotin injected uPNs, illustrates clearly that there is no co-localization of GABA within the dendrites of the uPNs in the glomerulus (insert in **Figure 2C2**). GABA immunoreactivity is distributed throughout the whole glomerulus in both cortex and core area, clearly originating from different GABAergic cells.

#### *Approximately 17 GABAergic mPNs from the ml-APT branch in the LPL*

In other preparations, neurobiotin was injected into the LPL and LH, which fills neurons that belong to the l-APT, m-APT, and ml-APT (Kirschner et al., 2006). The m-APT and ml-APT leave the antennal lobe together; **Figure 2E1** shows a horizontal section at the beginning of both tracts as they exit the antennal lobe. Most uPN axons in the tracts from both m-APT and ml-APT neurons are labeled by neurobiotin (**Figure 2E1**). tract **(E2)** Anti-GABA staining manifests only in the section that we identify as the beginning of the ml-APT **(E3)** The m-APT is not stained with GABA, however a few GABAergic fibers are in the m-APT. These fibers are in the lateral part of the m-APT exiting from the antennal lobe that we identified as ml-APT 1,2(white merge image). **(F)** GABA staining in the m-APT and the two groups of GABAergic neurons in the frontal section of the antennal lobe are made in the area of the m-APT tract. The axons from the ml-APT are indicated by arrows. **(G)** GABA staining is in the ml-APT-1 that branches to the ventral part of the LH. **(H)** Schematic presentation of GABAergic cell clusters made after ten frontal brain sections (35 μm each) stained with GABA. The three groups of neurons identified are MVG, MG, and LG. Four neurons are identified as Giant MVG GABAergic neurons: they have a defined location and large somata (arrows). All figures show the right part of the brain: the middle of the brain is on the left and the lateral on the right. V, vertical lobe; γ, gamma lobe of MB. Scale bar: **A**, **C–H** = 35μm; insert in **C2** = 15μm.

Anti-GABA antibodies stained groups of axons in the ml-APT tract (short arrow in **Figure 2E2**). Merging the images of neurobiotin injected neurons (magenta) and anti-GABA stained neurons (green) reveals that this group of GABAergic axons runs laterally at the beginning of the m-APT before turning into ml-APT tracts [matching arrow positions in **Figures 2E2,E3**, (white)]. The same GABAergic axons leaving the antennal lobe are illustrated in the frontal section of the antennal lobe (**Figure 2F)**. These GABA stained axons leave the antennal lobe through the ml-APT1,2 tracts, and they then enter and branch into different areas of the LPL (**Figure 2G**). When we counted the number of these fibers in the frontal and horizontal sections of eight bee brains, we found that approximately 17 ± 3 GABA-positive fibers were in the ml-APT 1, 2.12 ± 2 in the ml-APT 1 and 5 ± 2 in the ml-APT-2 (ml-APT-2 is not illustrated here). We identified these neurons as mPNs according to Kirschner et al. (2006).

#### *Most GABAergic neurons in the antennal lobe are local interneurons*

**Figure 2H** shows the schematic reconstruction of the antennal lobe from the frontal sections stained with GABA antiserum. We identified three distinct soma groups of GABAergic neurons in the antennal lobe: the medio-ventral group MVG (**Figures 2E,H**), the medial group MG (**Figure 2H**) and the lateral group (LG). Most of the GABAergic neurons in these groups are LNs that interconnect glomeruli and do not exit the antennal lobe neuropil. The exceptions include approximately 17 GABAergic mPNs that leave the antennal lobe through the ml-APT, and at least four unidentified neurons that leave through the l-APT. In addition, two GABAergic neurons located in the most dorsal part of the medial group (MG in **Figure 2H**) leave the antennal lobe via midline bundles (these neurons are not illustrated here).

#### *There are approximately 375 GABAergic local interneurons in the antennal lobe*

We counted the cells in the frontal sections of three honey bee brains stained with anti-GABA antibodies. In total there are 402 ± 30 cells distributed as follows: MG (*n* = 30 ± 5) and MVG (*n* = 18 ± 5) and LG (*n* = 350 ± 25). In addition, there are four giant medio-ventral GABAergic neurons with large somata, in the medial ventral part of the antennal lobe (arrows in **Figure 2H**). Because the injection of neurobiotin into the antennal lobe revealed both neurobiotin tracer and GABA within the same cell population, we can associate all of these GABAergic neurons with the antennal lobe (not illustrated here). The GABAergic cells located dorsally to the antennal lobe are not counted in our preparations because they are not co-localized with staining of the tracer injected into aglomerular area of the antennal lobe. The LG is the largest group of GABAergic neurons and covers the lateral area of the antennal lobe. Most of the LG neurons have a cell body size ranging from 8–10μm. The exceptions are the larger GABA neurons (15μm) that cluster on the top of the antennal lobe and the rostral part of the LG (yellow arrowheads in **Figure 2H**). Furthermore, as described by Kreissl et al. (2010), these neurons exhibit both alatostatin and GABA (*n* = 20, size = 15μm) in their cell bodies, and they branch within the core and inner cortex area of each glomerulus.

#### *Anti-AmOA1 antibodies label gabaergic processes in the glomeruli*

To examine AmOA1 staining in glomeruli, we co-labeled frontal sections of the antennal lobe with anti-AmOA1 antibodies and GABA antiserum and analyzed the distribution of staining within glomeruli (**Figure 3A**). AmOA1 immunostaining was variable across glomeruli by amount and intensity in the core and cortex area (compare **Figures 3A1,C2,H1**, from three different honey bee brains). An example of an antennal lobe glomerulus that exhibits a high level of anti-AmOA1 staining in the core and cortex area is shown in **Figure 3A1**. Furthermore, AmOA1 staining is expressed in both GABAergic (arrow) and non-GABAergic processes (arrowhead) but not in uPNs (double arrowheads in **Figures 3A1–A3**). GABA and AmOA1 immunoreactivities are also distributed throughout the whole area of the glomerulus (**Figures 3A1,A2**).

#### *Anti-AmOA1 is not in the ORN endings and uPN dendrites*

In order to study AmOA1 distribution on ORNs and uPNs, we injected Rodamine-dextran into the antennal nerve and neurobiotin in the m- and l-APTs into the area between the MB lobes (**Figure 2B**, **Table 1**). Then we applied anti-AmOA1 antibodies to frontal sections of the antennal lobe. All three stainings are illustrated in **Figure 3C1** where the anti-AmOA1 staining is labeled in

**FIGURE 3 |** *Apis mellifera***: The GABAergic processes in glomeruli express AmOA1 (A) Triple labeling of the anti-AmOA1 (green A1) and anti–GABA (magenta A2) in a glomerulus with neurobiotin-injected uPNs (blue A3). (A3)** The fibers that enter into the glomerulus are GABAergic co-stained with AmOA1 (white, arrow), non-GABAergic processes co-stained with AmOA1 (green, arrowhead) and uPNs that do not co-localize with AmOA1 (blue, double arrowheads). **(B)** GABA and AmOA1 staining at higher magnification **(C)** Triple labeling of a glomerulus **(C1)** in which anti-AmOA1 (green, **C1** and **C2** single image) and ORNs were labeled by anterograde staining with Rhodamine-dextran (magenta **C1**, single image **C3**), and uPNs are shown by retrograde staining (blue **C1**, and black and white in a single image **C4**). **(D)** The same glomerulus as in **(C1)** but only ORNs (magenta) and AmOA1(green) are shown **(E)** illustrates a high

magnification of the area shown in the square designating in **(D)**. Arrows show the close proximity of the AmOA1 stained profiles and ORNs. **(F)** The same glomerulus as in **(C1)** but only staining of AmOA1 and uPNs are shown. **(G)** A high magnification of the area shown by the square designated in **(F)**. Arrowheads show AmOA1 in the area that surrounds the uPNs fibers, which might be presynaptic to uPNs. **(H)** Detail of the antennal lobe labeled with anti-AmOA1 (green, **H1**) and anterogradely labeled ORNs (magenta **H2** in merge image). ORNs do not show the co-expression with AmOA1. **(H3)** PNs are labeled by injection into the LPL where three glomeruli from the lateral part of the antennal lobe are shown with cell bodies surrounding the antennal lobe. **(H4)** Merged image where anti-AmOA1 (green) and PNs (magenta) illustrate that there are possible co-localizations of AmOA1 in the cell bodies of subsets of PNs. Scale bars: **A,C,D,F** = 15μm; **B,E,G** = 5μm; **H** = 35μm.

green (**Figure 3C2**), the ORN endings in magenta (**Figure 3C3**) and uPN dendrites in blue (**Figure 3C4**). Most colors are not mixed, which indicates that most of the AmOA1 staining is not in uPNs or ORNs (**Figures 3C1,D–G**). There are only a few mixed colors: yellow for ORNs and anti-AmOA1 (arrows); light green showing possible co-staining of uPNs and anti-AmOA1 arrowheads in **Figure 3C1**. Higher magnification revealed that white colors could indicate a possible co-staining of anti-AmOA1 and ORNs (arrows in **Figure 3E**) and uPNs (arrowheads in **Figure 3G**), or more likely it could indicate close proximity of ORN and anti-AmOA1 stained processes.

Data from our light microscope analysis showed clear differences between the distribution of AmOA1 receptor in the GABAergic processes and in the uPNs and ORNs. We found that anti-AmOA1 antibodies are also expressed on the cell body of the PNs in the medio-ventral group (**Figures 3H1–H3**), and may co-stain their axons in the glomeruli (**Figure 3H4**). These neurons might belong to mPNs that leave the antennal lobe via ml-APT1,2.

#### *Anti-AmOA1 labels gabaergic processes and some Kenyon cells but not axons or endings of uPNs in calyces and LPL*

Anti-AmOA1 did not label axons of uPNs in the m-APT and l-APT (**Figures 4A,B2,C2**). However, it labeled GABAergic processes that frame the l-APT (**Figures 4B1**,**B3**), and it also labeled the subsets of GABAergic mPN axons of the ml-APT 1,2

**FIGURE 4 |** *Apis mellifera***: The AmOA1 immunostainings in the central brain neuropils (LPL, PL, and MB calyx) are co-localized in the GABAergic neurons but not in uPNs. (A)** An injection of a neurobiotin tracer into the antennal lobe revealed all antenna-protocerebral tracts (APTs). **(B)** The origin of l-APT tracts did not show GABA (magenta, **B1**) and AmOA1 (green, **B2**) staining inside of the tract. **(B3)** The merged image shows white fibers indicating the co-localization of anti-GABA and Anti-AmOA1 in the area surrounded the l-APT. **(C)** The beginning of the m-APT in a frontal section of the brain in which the anti-GABA (**C1**, magenta) and anti-AmOA1 (**C2**, green) are co-localized in the lateral part of the tract (white image merge in **C3**). The medial part of the m-APT containing axons from uPNs is not GABAergic. **(D)** The calyx of the MB on the frontal section of the brain with injected subsets of uPNs from l-APT and m-APT (**D1**, single image) stained with anti-GABA (**D2**, single image), and anti-AmOA1 (**D3**, single image). **(D1)** uPNs ending in the lip (lip) and basal ring (br) of the MB calyx. uPNs enter to the calyx via APT. **(D2)** Anti-GABA profiles originating from PCT (feedback) neurons that enter to the calyx via PCT (protocerebral tract) can be found in all calyxes. **(D3)** Anti-AmOA1 immunostained the Kenyon cell (K) as well as a subset of the PCT neuron GABAergic endings as demonstrated in the merged image **(D4)** (anti-GABA magenta, and anti-AmOA1 green). **(D4)** The white area in the merged image clearly indicates the distribution of anti-AmOA1 staining within the subset of the GABAergic endings. **(D5)** The merged image of the uPNs (magenta) and anti-AmOA1 do not show co-localization of AmOA1 within uPNs ending in the calyx. **(E)** The Lateral Protocerebral Lobe (LPL) with triple staining injected m-APT uPNs dendrites **(E1)**, anti-GABA **(E2),** and anti-AmOA1 **(E3)**. **(E4)** Merged images of anti-GABA (magenta) and anti-AmOA1 (green) revealed co-staining in the fibers from ml-APT 1, 2 (arrows) and in the GABAergic processes of the LPL. **(E5)** There is no evidence for co-localization in merged image of uPNs (magenta) and anti-AmOA1 (green). **(F)** The anti-AmOA1 stainings only the subset of the GABAergic mPNs in the ml-APT1. **(F1)** The neurobiotin deposits in the antennal lobe revealed an ml-APT tract that we associated with mPNs. **(F2)** GABA immunoreactivity in the ml-APT-1. **(F3)** Anti-AmOA1staining in the ml-APT1. **(F4)** As shown in merged staining (white) only a few GABAergic fibers (magenta) are co-stained with anti-AmOA1 (green). **(F5)** The same for the merged image showing injected mPNs fibers (magenta) and anti-AmOA1 (green). Scale bar: **A,D–F** = 100μm; **B,C** = 20μm.

(**Figure 4C1**). In both tracts, only the GABA-positive processes contain AmOA1 receptors, meaning that the axons of the uPNs do not express AmOA1. **Figure 4D1** illustrates terminals of the uPNs in the basal ring and lip of the calyx of the MB. This same area receives GABAergic innervation from GABAergic protocerbral tract neurons (PCT; **Figures 1**, **4D2**) (Sinakevitch et al., 2011). The anti-AmOA1 is mostly co-localized with the GABAergic endings of the PCT neurons in the calyx of the MB [**Figure 4D4**, white processes on merged image of anti-GABA (magenta) and anti-AmOA1 (green)]. Anti-AmOA1 also labeled many cell bodies of the Kenyon cells (**Figures 4D3,D4**). Anti-AmOA1 is not exhibited on the uPNs endings in the MB calyx (**Figure 4D5**). Further, the GABA processes co-localize with anti-AmOA1 but not with uPN outputs in the basal ring and lip of the calyx.

The uPNs also branch in the LH, which is in the latero-ventral area of the protocerebrum (Abel et al., 2001; Kirschner et al., 2006). The uPNs from the m-APT and l-APT tracts branch in different parts of the LPL. The m-APT uPNs end in the ventral area of the LH, while the l-APT ends in the dorsal part of LH (Kirschner et al., 2006).

**Figure 4E1** shows the endings of the m-APT in the LH, and these endings are not co-labeled with anti-AmOA1 or with anti-GABA. The AmOA1 immunoreactivity is localized in GABAergic terminals. Both the ml-APT-1 and ml-APT-2 tracts ended in the LPL in different areas; ml-APT-1 ended in the most ventral section of the LPL and ml-APT-2 in the dorsal section. Anti-AmOA1 is stained in the axons and endings of the GABAergic ml-APT-1,2 mPNs (**Figures 4E,F**).

#### *Distribution of GABAergic mPNs in antennal lobe glomeruli*

To identify how mPNs branch in the antennal lobe glomeruli, we injected neurobiotin into multiple sites close to the LH and rostral to the MB vertical lobe with subsequent anti-GABA and anti-AmOA1 staining (**Figure 5A**, LPL and LH sections in **Table 1**). We acknowledge that this method has its limitations. Not all the mPNs in the ml-APT could be revealed and sometimes a few uPNs were also stained (especially from the injection site in the LPL). However, by examining the neurobiotin tracer staining in six preparations with similar groups of neurons filled from 15 total preparations (**Table 1**) through the LPL as indicated in **Figure 5A**, we found three groups of cell bodies that co-labeled with GABA. The largest group is Dorso-Caudal Lateral mPNs (DCLmPNs, **Figures 5A,B**). We also identified at least two GABAergic neurons in the Dorso-Rostral Lateral mPNs group (DRLmPNs, **Figures 5A,C**). Finally, we identified at least three GABAergic neurons in the Ventro-Medial mPN group (VMmPNs, **Figure 5D**). The somata described here belong to the ml-APT because it was possible to follow their axons from the beginning of ml-APT to their cell body in these preparations. The methods that we used here gave us the approximate position and branching patterns of some of the GABAergic mPNs, however the numbers of the GABAergic fibers in the ml-APT tracts suggest that there might be many more with such patterns. It is important to note that ml-APT-1,2 also had neurobiotin labeled fibers that were not co-localized with anti-GABA staining.

The largest group of neurons that were filled with neurobiotin after LPL injection is located caudally at the most lateral part of the antennal lobe near the mechanosensory and motor center neuropil, where the antennal lobe connects to subesophageal ganglion (DCLmPNs, **Figures 5B,C**). In LPL-injected preparations, the number of neurons in this group varied from 30 to 40 due to the injection site. Not all the neurons from these groups exhibit GABA-like immunoreactivity (**Figures 5B1,B2**). We found that 15 ± 4 neurons (*n* = 6) in this group co-localized with GABA (**Figure 5B2**). Moreover, among those GABAergic neurons only two were co-stained by anti-AmOA1 in the cell bodies and primary axons (**Figure 5B3**). This is illustrated in higher magnification of the merged images of anti-GABA (green) and anti-AmOA1 (magenta) in **Figure 5B3**. Significantly, this is consistent with the data shown in **Figures 4E,F**, where only a few fibers from ml-APT1 that enter into the LPL are labeled with AmOA1. We could not identify the glomerular distribution of the dendrites from these neurons; however some of them might have arborizations in many glomeruli as shown in **Figure 5D1**. Note that axons from the antennal nerve to mechanosensory and motor center neuropils are stained with anti-AmOA1 (**Figure 5B3**). The following group, which we identify as DRLmPNs, is illustrated in **Figure 5C**, which demonstrates that the cell bodies lay laterally and rostrally in the relationship to the DCLmPNs group. All three neurobiotin filled cells shown in close up [**Figure 5C2** (magenta)] co-localized with anti-GABA antibodies [**Figure 5C3** (green, single image of GABA staining) and **Figure 5C4** (merged image of neurobiotin and anti-GABA)]. The VMmPNs group reveals scattered cell bodies in the medial part of the antennal lobe; two cell bodies on the ventro-medial part of the glomeruli are illustrated in **Figure 5D1**. In higher magnification, both cell bodies are co-localized with anti-GABA staining [white merged images in **Figures 5D4,D3** (anti-GABA, green), **5D2** (neurobiotin, magenta)].

We noted that a few thick fibers run through the coarse area of the antennal lobe extending branches to all glomeruli, however, due to an extensive branching pattern, it was difficult to identify how many fibers belong to one cell in the frontal sections of the brain. In the frontal section through the top of the antennal lobe, the branching in the glomeruli are seen mostly in the cortex area (**Figure 5E**). A higher magnification of a glomerulus revealed different structural patterns (i.e., bleblike and spine-like endings) in the cortex area of the glomerulus (**Figure 5F**).

#### *Dendrites of the uPNs and mPNs reveal differences in their connections to the core and cortex area of the glomerulus*

**Figure 5G** illustrates the projected view of an image stack of a uPN labeled by intracellular neurobiotin injections. The cell body (15μm) is on the surface of the antennal lobe between glomeruli, as shown in the schematic in **Figure 5H**. The primary neurites plunge vertically between glomeruli and run approximately 60μm to bifurcating points toward either the MB calyces and to the glomerulus. A neurite (20μm length from bifurcation point) runs through the coarse area of the antennal lobe close to enter a glomerulus through the core (**Figures 5G,H**). There the neurite gives rise to thick fibers that branch mostly in the core of glomerulus as well as fine, densely packed fibers that cover the entire glomerulus (see **Figures 2C1**, **5G,H**). The area of the

**FIGURE 5 |** *Apis mellifera.* The mPNs branching in the glomeruli as revealed by neurobiotin injection into the lateral protocerebral lobe. **(A)** Schematic of the honey bee brain that illustrates octopaminergic neurons that branch in the antennal lobe, lateral protocerebral lobe (LPL) and calyx of the MB. Neurobiotin injected into the LPL revealed three groups of neurons named as follow DCLmPNs **(B1)**, DRLmPNs **(C1)**, and VMmPNs **(D1)**. A subpopulation of the neurobiotin injected neurons from DCLmPNs (magenta **B1** and **B2**) were co-labeled with GABA (green). **(B3)** A higher magnification of the GABA neurons co-stained with anti-AmOA1: arrowheads indicate the GABAergic DCLmPNs neurons that co-labeled with anti-AmOA1. **(C1)** There are three neurons in the DRLmPNs group that are located dorsally in a lateral cluster of the most caudal part of the antennal lobe. These neurobiotin labeled neurons (**C2**, magenta) are co-stained with anti-GABA (**C3**, green single image). **(C4)** The merged image shows the anti-GABA (green) and neurobiotin (magenta) labeled neurons. **(D)** An image from the brain section of the antennal lobe where

cortex of the glomerulus is connected with fine fibers of the uPN (**Figures 5G,H**).

The glomerular branching of the uPNs and mPNs is schematically drawn in the sagittal view of the antennal lobe (**Figure 5H**). In contrast to the uPNs that have branches all over the glomerulus, with thick fibers in the core and fine dendrites in the cortex, the mPNs branch only in the cortex. According to Nishino et al. (2009), the fibers of the ORNs are topographically organized within the glomerulus, where few axons from different cells connect different glomeruli. One neuron we identify from VMmPNs group, it was possible to follow the neurite from cell body and its branching into glomeruli. This neuron (**D2**, magenta single image) co-localized with GABA (**D3**, green single image) is shown in the merged image in **(D4)**. **(E)** The section from the same preparation through most ventral part of the antennal lobe. The branching pattern of the neurons in the glomeruli is in the cortex area. **(F)** The mPNs ending in the glomeruli cortex revealed spine like (arrows) and bleb-like structures (arrowheads). **(G)** The anatomy of single uPN revealed by intracellular injection into the cell body on the left, collapse frontal view. **(H)** The right schematic demonstrates the branching pattern of the uPN and mPN in the antennal glomeruli. uPN has the thick fibers in the core and fine arborization in the cortex (red), mPNs fibers are in outer area of the core and in the inner cortex. Asterisks in **(B2,B3)** indicated the axons from antennal nerve traveled to mechanosensory and motor center neuropil. Scale bar: **B** = 20μm, **C1,D2,E** = 50μm, **F,G** = 15μm.

the ORNs from the most distal part of the antennal segment enter close to the core or inner area of the cortex and the ORNs from the proximal part of the antennal segment enter to the peripheral area of the cortex. There each glomerulus has two areas: the cortex, which receives ORNs, and the core, where ORN endings are not present. Both uPNs and mPNs are present in the cortex of the glomerulus and might receive excitatory input from the organized ORNs in this region (**Figure 5H**).

#### **ANATOMY OF GLOMERULI IN THE FRUIT FLY ANTENNAL LOBE**

#### *Olfactory receptor neuron (ORN) terminals also define the structure of the glomerular cortex in the fruit fly*

We used the enhancer trap lines OR83b-GAL4× UASmcd8GFP, which express the GFP in ORNs, to reveal receptor neuron axon endings in the antennal lobe (Larsson et al., 2004). In this line, ORNs from the antenna and the maxillary palp (MP) have axons that terminate in the glomeruli (**Figure 6A**). ORNs from the antenna enter the antennal lobe from lateral and anterior positions, and ORNs from the MP travel dorsally through the suboesophageal ganglion (SEG) to enter the antennal lobe from a ventral posterior position (**Figure 6A**; Jefferis et al., 2004; Sweeney et al., 2007).

In the fruit fly, glomeruli have an outer cortex innervated by ORN terminals and a core area that lacks ORN terminals (**Figure 6B**). The glomerular structure in the fruit fly is similar to that of the honey bee. In horizontal sections (**Figure 6A**, section panel **B**), anti-synapsin antibodies labeled the entire glomerulus (**Figure 6B1**) and anti-GFP labeled the ORNs glomerulus (**Figure 6B2**). Like in the honey bee, ORNs terminate in the cortex (**Figure 6B2**). However, glomeruli in fruit flies do not exhibit strictly defined borders between the core and cortex, which makes estimation of the relative areas of core and cortex difficult. Finally, as in honey bees, the fruit fly antennal lobe contains glomerular and aglomerular neuropils (**Figures 6C,D,F**).

#### *Most of the ORNs in fruit flies do not label with AmOA1*

In general, anti-AmOA1 antibodies label the clusters of cells that surround the antennal lobe and are located dorso-laterally, laterally, and ventrally to the antennal lobe (**Figures 6C1,D1,F1**). We did not find co-localization of AmOA1 within the most ORNs terminals (**Figures 6C1–C3**).

#### *Most uPNs in fruit flies do not label with anti-AmOA1*

To study the distribution of the OA1 receptors in a uPNs we used the GAL4-GH146 line crossed with UAS –mCD8-GFP, which drives GFP expression in a large subset of uPNs and a few mPNs (**Figures 6D,E**) (Marin et al., 2002). We could then visualize via GFP the cell bodies and dendrites of uPNs. The uPNs are cholinergic (Stocker et al., 1997; Jefferis et al., 2002; Python and Stocker, 2002; Olsen et al., 2007a). Thick fibers from uPNs are in areas that might correspond to the core (**Figure 6D**, arrowhead), while fine ramifications of the PNs are all over the glomerulus. Most of the GFP labeled cell bodies and axons, as well as their endings in the MB calyx and LH, do not label with anti-AmOA1 antibodies (**Figures 6D,E**). Anti-OA1 staining is also absent in m-APT projection neurons and their outputs in the calyx (**Figure 6E1**), with the exception of one or two axons that co-localize with anti-OA1 in the medio-lateral part of the m-APT (**Figure 6E2**, two arrowheads in m-APT).

Most of the AmOA1 stained axons that project from the antennal lobe belong to mPNs in the ml-APT (**Figures 6E1–E3**). Those axons are not labeled with anti-GFP, but they co-label with anti-GABA (**Figures 6E1** inserts). Some GFP stained mPNs do not show labeling with AmOA1 (**Figures 6E2,E3**), but they nevertheless co-stained with anti-GABA (**Figure 6E2** insert).

#### *A subset of local GABAergic interneurons in the Drosophila antennal lobes label with anti-OA1 in the cell bodies, axons, and endings in the glomerulus*

Anti-OA1 staining indicated cell bodies of neurons located laterally (**Figures 6A,B,D**). Scattered, stained processes are exhibited in all glomeruli and coarse areas of the antennal lobe. In the antennal lobe, GABA-like immunoreactivity is found in the LNs with cell bodies in lateral and dorso-lateral clusters. Furthermore, in addition to the two classes of GABAergic LNs, LN1, and LN2, there are GABAergic mPNs in the ml-APT tract. The LNs supply GABAergic processes in glomeruli (**Figure 6F**). Comparisons of anti-OA1 staining (magenta) with anti-GABA staining (green) in the antennal lobe suggest that OA1 is expressed in the subpopulation of local GABAergic interneurons (white) and in the GABAergic ml-APT multiglomerular PNs. The glomeruli are homogeneously stained with the anti-OA1 suggesting that all three types of GABAergic neurons express OA1. In addition, in the lateral cluster there are neurons marked by anti-OA1 but not by anti-GABA. These neurons, which are not GABAergic, might be excitatory LNs or non-GABAergic ml-APT PNs that connect antennal lobe to the LH (Tanaka et al., 2012a,b).

## **DISCUSSION**

Several studies have documented the role of OA in driving plasticity linked to associative conditioning in fruit flies (Schwaerzel et al., 2003; Kim et al., 2013) and honey bees (Hammer, 1993; Farooqui et al., 2003). Our current study was motivated by an effort to understand the downstream components of OA signaling. That is, how OA induces neural plasticity by acting via different types of receptors. We identified neurons that express the AmOA1 receptor in neural networks of the antennal lobe (AL) of the honey bee and fruit fly. We also focused on the AmOA1 receptor because of its established role in behavioral plasticity in fruit flies (Kim et al., 2013) and because of the availability of a characterized antibody against this receptor (Sinakevitch et al., 2011). Our data show similar expression patterns in both species. Anti-AmOA1 receptor antibodies label subsets of GABAergic local neurons and mPNs. In addition, we show that AmOA1 receptors are expressed on non-GABAergic neurons in the antennal lobe. In contrast, we could not find such clear evidence that AmOA1 is expressed in ORN receptor terminals or uPNs.

How could this expression pattern account for changes in calcium responses of uPNs after associative conditioning (Fernandez et al., 2009)? Changes in calcium dynamics in uPNs are subtle and consist of many different patterns of shifts from excitation to inhibition and vice versa. OA binding to AmOA1/DmOA1 results in the release of calcium from cytosolic stores (Han et al., 1998; Grohmann et al., 2003; Beggs et al., 2011; Hoff et al., 2011), which would likely make GABAergic LNs and mPNs more excitable. Therefore, the plasticity reported in honey bee uPNs (Fernandez et al., 2009; Locatelli et al., 2013), in so far as it is induced by AmOA1, must have arisen indirectly by changes in the action of inhibition in the antennal lobe network. This conclusion is consistent with a recent study employing OA application to the honey bee antennal lobe (Rein et al., 2013) which produced both excitatory and inhibitory shifts in calcium dynamics of uPNs.

**FIGURE 6 |** *Drosophila melanogaster***: a subpopulation of GABAergic neurons are co-stained with anti-OA1 antibodies. (A)** Schematic of the organization of the *Drosophila* olfactory system in sagittal view. MP, maxillary palp; ORN, olfactory receptor neuron; SEG, subesophageal ganglion; ant lob, antennal lobe; ca, calyx, ped, pedunculus; v, vertical lobe of MB; A, anterior; P, posterior; Dorsal, Ve- ventral. Broken lines indicate the approximate orientation of sections through antennal lobe (in **C,F,D**) and lateral protocerbrum **(E)**. **(B)** Section through the center of one glomerulus labeled with anti-synapsin (green, **B1**) and anti-GFP in ORNs (magenta, **B2**) in OR83b-GAL4;UAS-mcd8-GFP flies. The glomerulus has a core area where the ORNs do not branch **(B3)**. **(C1)** Anti-OA1 stained groups of cells and processes in the glomeruli and the aglomerular neuropile area of the antennal lobe in OR83b-GAL4;UAS-mcd8-GFP (**C1**, magenta) on the oblique frontal brain cross-section of the antennal lobe. **(C2)** In the same section, GFP (green) takes up a large percentage, or perhaps all, of the olfactory receptor endings in the antennal lobe glomeruli. **(C3)** The majority of the sensory neurons terminals do not label with the OA1 antiserum (magenta) with a few exceptions. **(D,E)** Anti-OA1 antibodies do not label the majority of uniglomerular projection neurons (uPNs). **(D1)** Here, anti-OA1 labeled clusters of cells surrounding the antennal lobes. These neurons are not projection neurons. GH146-GAL4; UAS-mcd8-GFP projection neurons expressed GFP (green) **(D2)**. In GFP expressing neurons there is no OA1 immunoreactivity as shown in our merged image **(D3)**, co-localization would show as white. **(E)** Anti-OA1 staining is absent in most uPN axons that leave the antennal lobe via the m-APT and branch in the calyx (ca) of the MB and lateral horn (LH); one exception is the axon shown by two arrowheads **(E1)**. mPNs leave the antennal lobe via the ml-APT, a large portion of the ml-APT fibers are OA1 positive (arrow) in **(E1,E3)**. These fibers also exhibit anti-GABA staining (inserts in **E1**). Three neurons that have their axons in ml-APT are also labeled with anti-GFP and GABA in GH146-GAL4; UAS-mcd8-GFP (**E2** and insert in **E2**). These neurons are not labeled with anti-OA1 in this brain preparation **(E3)**. **(F1)** In the same section as shown in **(C)** anti-OA1 (magenta) labels the cell bodies of laterally located neurons. The scattered, stained processes are in all glomeruli. **(F2)** The same sections labeled with anti-GABA antibodies; and GABA-like immunoreactivity is found in neurons with cell bodies in lateral and dorso-lateral clusters. These neurons supply GABAergic processes to the glomeruli. The merged image **(F3)** of the same sections shows the group of GABAergic neurons co-stained with anti-OA1 (white). Anti-OA1 staining is found in the cell bodies and processes in the glomeruli and in the aglomerular area of the antennal lobe. Arrows in **(C1)** and **(D1,F1)** indicate lateral neurons cluster (LN) and ventral cluster (VN) of anti-AmOA1 positives neurons. In **(D2)**, the arrowhead indicates thick fibers of the PNs entering the glomerulus that might correspond to the core area of the glomerulus. Scale bar: **A** = 10μm, **B–F** = 20μm.

This study also concluded that OA likely targets inhibition in the antennal lobe.

Clearly OA has multiple targets via AmOA1 in the antennal lobe. However, AmOA1 might not be the only modulatory pathway for OA. A second type of OA receptor (OA2) has been identified in fruit flies and honey bees (Evans and Robb, 1993; Roeder, 1999; Hauser et al., 2006). OA2 stimulates adenylate cyclase and thereby increases the concentration of adenosine 3 ,5 -cyclic monophosphate (cAMP). Very little is known about the distribution of OA2 in the nervous system of either species.

**Figure 7** summarizes our findings and presents a model incorporating our information about AmOA1 in antennal lobe glomeruli exclusively for the honey bee. In order to construct the model, we present the distribution of AmOA1 receptor in a more detailed discussion of important neuroanatomical features revealed in our current study.

#### **OLFACTORY RECEPTOR NEURONS (ORNs)**

In the fruit fly, ORNs that express the same receptors project to the same glomerulus (Laissue and Vosshall, 2008; Tanaka et al., 2012a). Although it has not been directly shown, we assume that this is also true for the honey bee. However, there are some differences in the structure of the glomeruli between fruit flies and honey bees. In the honey bee the endings of ORNs are clearly functionally and topologically organized in the outer layer (cortex) of the glomerulus (Nishino et al., 2009). This was not the case in fruit flies in which the distribution of ORNs has no precise topological organization within the cortex. This difference might be due to a somewhat different organization of the antenna combined with inputs from the MP. The core area of glomerulus is free from ORNs in both insects (Hummel and Zipursky, 2004). We found no indication of AmOA1 expression in ORN terminals.

#### **UNIGLOMERULAR PROJECTION NEURONS (uPNs)**

uPNs reveal dendritic branching limited to a single glomerulus, and they project axons to higher order neuropils (MB and LPL) (Abel et al., 2001; Kirschner et al., 2006; Galizia and Sachse, 2010). In both insects, the dendrites of uPNs branch throughout the entire glomerulus (both cortex and core). In fruit flies, the ORNs synapse on uPNs in each glomerulus (Wilson, 2011). In the honey bee, there has to date been no conclusive study to show that ORNs synapse onto uPNs. Nevertheless, for our model in **Figure 7** we assume that uPNs are postsynaptic to ORNs in the cortex area of the glomeruli, and they can be inhibited by LNs in the core and send information to the MB and LPL.

In the honey bee, approximately five uPNs branch in each glomerulus (Rybak, 2012). Thick fibers are located at the entrance of the core area and fine fibers in the cortex, which could be related to the possibility that uPNs receive different types of inputs to these two areas. Consistent with other reports, we also observed one fine branch that extends from the uPN to the neighboring glomerulus cortex (not shown in **Figure 7**).

In both insects, the uPNs form two pathways that connect the antennal lobe with the calyx and LH via the m-APT and l-APT tracts (Okada et al., 2009; Galizia and Rössler, 2010; Tanaka et al., 2012a), respectively. Furthermore, the axon terminals from each pathway are topographically organized in the calyx and LPL

**FIGURE 7 | Schematic representation of the neural network proposed for a glomerulus in the honey bee (model architecture based on Fonta et al., 1993; Abel et al., 2001; Nishino et al., 2009; Meyer and Galizia, 2012; Girardin et al., 2013).** Octopamine is released into the whole glomerulus in both core and cortex and acts on the GABArgic local interneurons (LNs) and on GABAergic multiglomerular projection neurons (mPNs). Each glomerulus has a uniglomerular PN (uPNs) that branches into both core and cortex areas of the glomerulus. uPNs receive excitatory synapses from ORNs in the cortex area and inhibitory synapses from hetero-multiglomerular local neurons from neighboring glomeruli in the core area. The uPNs also receive inhibitory synapses in the cortex that come from the mPNs. In the glomerulus there are GABAergic and non-GABAergic multiglomerular LNs. For simplicity, only one type of the GABAergic LNs is shown in the glomeruli; LNg (LN1 or heteroLN in **Figure 1**) that branches in all areas of the glomerulus (cortex and core) where it receives excitatory synapses from ORN in the cortex and inhibitory synapses in the core and sends the inhibitory output into the neighboring glomeruli. Both types of GABAergic neurons (LNs and mPNs) express AmOA1 receptors in the glomerulus. Based on our indirect evidence of AmOA1 and GABA stainings in glomeruli, we hypothesize two possible mPNs connections, one that receives excitatory synapses from ORNs in the glomerulus cortex and makes synapses in the LH as well as another that receives synaptic input from LPL and LH and makes synapses onto the processes in the cortex. Future physiological and anatomical studies will clarify the branching patterns that we proposed for mPNs.

for both insects. uPNs do not express anti-AmOA1 receptors in their branches within glomeruli or in their axons in the calyx, LPL, and LH.

#### **MULTIGLOMERULAR PNs**

Both fruit flies and honey bees have multiglomerular PNs. mPNs are GABAergic in fruit flies and have postsynaptic sites in the glomerulus with presynaptic sites in the LH (Okada et al., 2009). In the honey bee, there are at least 17 GABAergic ml-ACT mPNs that branch in different areas of the LH, LPL, and LH. We cannot, at this point rule, out the possibility that there could be two kinds of mPNs in honey bees, which we represent in the schematic in **Figure 7**. Some mPNs (dark blue) might receive input from ORNs or one or more types of antennal lobe LNs in the cortex of multiple glomeruli and synapse in the LPL and LH, as in fruit flies (Okada et al., 2009). Other mPNs (light blue) might receive input in the LPL and LH and synapse in the glomerular cortex. The latter subtype of mPN, if they exist, would constitute a feedback pathway to the antennal lobe. A more detailed analysis of the GABAergic mPNs will be needed to test this hypothesis. We can, however, conclude that subsets of GABAergic mPNs express AmOA1 in processes in the LPL and LH.

#### **LOCAL INTERNEURONS (LNs)**

Large numbers of LNs have been reported in the honey bee (Galizia and Sachse, 2010). Similarly, a high diversity of LNs has been reported in the antennal lobe of fruit flies (Chou et al., 2010): ∼100 ipsilaterally projecting and ∼100 billaterally projecting LNs. Most of the LNs are GABAergic (Ng et al., 2002; Wilson and Laurent, 2005; Okada et al., 2009; Seki et al., 2010), but some LNs are excitatory (Olsen et al., 2007b; Shang et al., 2007; Seki et al., 2010) and cholinergic (Shang et al., 2007; Das et al., 2008). The most ventral bilaterally projecting LNs are glutamatergic LNs (Chou et al., 2010). These LNs are also diverse in their glomerular innervation pattern, fine dendritic structures, densities and distribution of presynaptic terminals, and odor response properties (Okada et al., 2009; Chou et al., 2010; Seki et al., 2010; Tanaka et al., 2012a). In the honey bee, the LNs are also reported to have a high diversity in their dendritic distribution and response properties (Schafer and Bicker, 1986; Fonta et al., 1993; Meyer and Galizia, 2012; Girardin et al., 2013). In addition to GABAergic LNs, the honey bee has a large number of histaminergic LNs (Dacks et al., 2010), which have not been reported in fruit flies.

In the honey bee antennal lobe, we found approximately 375 GABAergic LNs. A previous study of GABA in the honey bee antennal lobe reported 750 GABAergic neurons (Schafer and Bicker, 1986). The difference between our study and that one might be due to counting GABAergic neurons in clusters that are dorsal to the antennal lobe. In our study, injection of dye into the antennal lobe combined with subsequent anti-GABA staining did not co-label the clusters of cells that are located rostrally.

Our results show that in both insects GABAergic and non-GABAergic neurons express AmOA1. The majority of the GABAergic LNs express the receptor in their branches throughout the glomeruli. In the model (**Figure 7**) we only consider heteroLNs, which are a subpopulation of LNs that have extensive arborizations in the core and cortex of one glomerulus and then less dense arborizations in the core of many glomeruli (Fonta et al., 1993). Our hypothesis is that heteroLNs receive excitatory input from ORNs in the cortex and inhibit the neural circuitry located in the core of neighboring glomeruli (**Figure 7**). We consider another subpopulation of LNs, the homoLNs (GABA and non-GABAergic), which make intraglomerular connections. These LNs receive inhibitory input from heteroLNs in the core and relieve the inhibition in other glomeruli. Among the homoLNs in the honey bee, there are 20 alatostatin GABAergic neurons that branch in the core and inner area of the cortex of the glomerulus (Kreissl et al., 2010). HeteroLNs might inhibit homoLNs in the core of neighboring glomeruli and thus provide additional excitation for uPNs in the given glomerulus.

#### **THE GLOMERULUS AS A FUNCTIONAL UNIT OF CODING AND PLASTICITY**

We found that glomeruli in the antennal lobe of the honey bee, independent of their size, all have a similar ratio of cortex to core. The neurons that are exclusive to each glomerulus are uPNs and ORNs. Both of these types of neurons are excitatory and their properties are not directly modified by OA via AmOA1. On the other hand, GABAergic LNs interconnect glomeruli and thus shape signals from uPNs and possibly ORNs. Each of the mPNs provides feed-forward (to the LPL) or possibly feedback (from the LPL) inhibition involving a large number of glomeruli. According to our model (**Figure 7**), the core of the glomerulus might be a computational unit that processes information from the ORNs and from other glomeruli while influencing the response profile of uPNs. This affects the information flow to the calyx of the MB and to the LPL and LH. Reinforcement learning through activation of AmOA1 receptors would shape this processing by targeting GABAergic processing.

#### **HYPOTHESIS FOR REINFORCEMENT-BASED PLASTICITY VIA AmOA1**

We propose a model for antennal lobe plasticity based on OA release onto AmOA1 receptors. In the antennal lobe, the fibers of the VUMs branch mostly in the cortex of the glomerulus with varicosity-like distributions (personal observation; Sinakevitch et al., 2005, 2011; Sinakevitch and Strausfeld, 2006). These wide branching fibers in the cortex could modulate responses of the ORNs, uPNs, and LNs during olfactory conditioning. We hypothesize that OA release from VUM is contingent upon gustatory stimulation of sucrose receptors on the mouthparts coincident with cholinergic input from ORNs to prime OA release. This hypothesis is based on pharmacological evidence of different types of acetylcholine receptors expressed in DUM neurons (Lapied et al., 1990; Grolleau et al., 1996; Sinakevitch et al., 1996; Courjaret and Lapied, 2001), which are homologs of VUM neurons (Sinakevitch et al., 2005; Sinakevitch and Strausfeld, 2006). Second, OA release from VUM acts on AmOA1 receptors expressed in GABAergic and non-GABAergic LNs, which leads to the prediction that OA will increase the excitability (Riffell et al., 2013), leading to increased inhibition in the cortex of the glomerulus. OA will not have that effect on other neurons, for example PNs, which may or may not express another type of OA receptor AmOA2 (Hauser et al., 2006) that does not induce immediate excitability. Future empirical and modeling studies need to test this hypothesis.

#### **ACKNOWLEDGMENTS**

We thank Dr. Carlos Castillo-Chavez for supporting Adrian N. Smith by providing Research Assistant positions and for general research supervising. We thank Dr. Birgit Elhmer for reading the first draft of the manuscript and Dr. Carsten Duch for use of his lab facilities. We thank Nadezhda Zolotova (undergraduate student, UCLA) for her help in characterization of mPNs. Irina T. Sinakevitch is grateful for Christopher Geldis, Elisabeth Cash, and Melanie Sturm, ASU graduate students, for discussions on the manuscript.

#### **REFERENCES**


## **GRANTS AND SUPPORT**

Fondation Recherche Medicale to Irina T. Sinakevitch UFM20060306548; NIH-NCRR RR014166 to Brian H. Smith; NIH-NIDCD 1R01DC011422 to Ramon Huerta, Maxim Bazhenov, Brian H. Smith, and 1R01DC012943 to Maxim Bazhenov; NSF Bridge-to-the-Doctorate Fellowship and the Navajo Nation Graduate Trust Fund to Adrian N. Smith and CONICET-Argentina to Fernando Locatelli.

processing by an octopaminergic reinforcement pathway in the honeybee. *J. Neurosci.* 23, 5370–5380. Available online at: http://www.jneurosci.org/content/ 23/12/5370.full.pdf


receptor with 'mixed' nicotinicmuscarinic pharmacology. *Neurosci. Lett.* 220, 142–146. doi: 10.1016/S0304-3940(96)13243-2


neurons in the antennal lobe of the fly. *Neuron* 36, 463–474. doi: 10.1016/S0896-6273(02)00975-3


533–561. doi: 10.1016/S0301-0082 (99)00016-7


specificity studies and immunocytochemical results. *Proc. Natl. Acad. Sci. U.S.A.* 81, 3888–3892. doi: 10.1073/pnas.81.12.3888


Wright, G. A., Mustard, J. A., Kottcamp, S. M., and Smith, B. H. (2007). Olfactory memory formation and the influence of reward pathway during appetitive learning by honey bees. *J. Exp. Biol.* 210, 4024–4033. doi: 10.1242/jeb. 006585

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 13 March 2013; accepted: 07 October 2013; published online: October 2013. 25*

*Citation: Sinakevitch IT, Smith AN, Locatelli F, Huerta R, Bazhenov M and Smith BH (2013) Apis mellifera octopamine receptor 1 (AmOA1) expression in antennal lobe networks of the honey bee (Apis mellifera) and fruit fly (Drosophila melanogaster). Front. Syst. Neurosci. 7:70. doi: 10.3389/fnsys. 2013.00070*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2013 Sinakevitch, Smith, Locatelli, Huerta, Bazhenov and Smith. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Opposite modulation of brain stimulation reward by NMDA and AMPA receptors in the ventral tegmental area

### *Charles Ducrot 1, Emmanuel Fortier 2, Claude Bouchard2 and Pierre-Paul Rompré2,3\**

*<sup>1</sup> Département de Physiologie, Université de Montréal, Montréal, QC, Canada*

*<sup>2</sup> Département de Psychiatrie, Université de Montréal, Montréal, QC, Canada*

*<sup>3</sup> FRSQ Research Group in Behavioral Neurobiology, Department of Psychology, Concordia University, Montréal, QC, Canada*

#### *Edited by:*

*Dave J. Hayes, University of Ottawa, Canada*

#### *Reviewed by:*

*George Panagis, University of Crete, Greece Nobuyoshi Suto, The Scripps Research Institute, USA*

#### *\*Correspondence:*

*Pierre-Paul Rompré, Département de Psychiatrie, Université de Montréal, Montréal, QC H3C 3J7, Canada e-mail: pierre-paul.rompre@ umontreal.ca*

Previous studies have shown that blockade of ventral tegmental area (VTA) glutamate N-Methyl-D-Aspartate (NMDA) receptors induces reward, stimulates forward locomotion and enhances brain stimulation reward. Glutamate induces two types of excitatory response on VTA neurons, a fast and short lasting depolarization mediated by α-amino-3-hydroxy-5-methyl-4-isoxazole propionate (AMPA) receptors and a longer lasting depolarization mediated by NMDA receptors. A role for the two glutamate receptors in modulation of VTA neuronal activity is evidenced by the functional change in AMPA and NMDA synaptic responses that result from repeated exposure to reward. Since both receptors contribute to the action of glutamate on VTA neuronal activity, we studied the effects of VTA AMPA and NMDA receptor blockade on reward induced by electrical brain stimulation. Experiments were performed on rats trained to self-administer electrical pulses in the medial posterior mesencephalon. Reward thresholds were measured with the curve-shift paradigm before and for 2 h after bilateral VTA microinjections of the AMPA antagonist, NBQX (2,3,-Dioxo-6-nitro-1,2,3,4-tetrahydrobenzo(f)quinoxaline-7-sulfonamide, 0, 80, and 800 pmol/0.5μl/side) and of a single dose (0.825 nmol/0.5 μl/side) of the NMDA antagonist, PPPA (2*R*,4*S*)-4-(3-Phosphonopropyl)-2-piperidinecarboxylic acid). NBQX produced a dose-dependent increase in reward threshold with no significant change in maximum rate of responding. Whereas PPPA injected at the same VTA sites produced a significant time dependent decrease in reward threshold and increase in maximum rate of responding. We found a negative correlation between the magnitude of the attenuation effect of NBQX and the enhancement effect of PPPA; moreover, NBQX and PPPA were most effective when injected, respectively, into the anterior and posterior VTA. These results suggest that glutamate acts on different receptor sub-types, most likely located on different VTA neurons, to modulate reward.

**Keywords: AMPA, glutamate, NMDA, reward, ventral midbrain**

### **INTRODUCTION**

The ventral tegmental area (VTA) contains dopamine neurons that give rise to the ascending mesocorticolimbic pathway that plays a key role in motivation and reward. A large body of evidence shows that dopamine neurons are activated by stimuli that have a positive reinforcing property. Drugs of abuse, for instance, stimulate dopamine cell firing and/or increase synaptic dopamine levels in the nucleus accumbens (Wise and Rompré, 1989; Wise, 1996), a limbic region considered as a functional interface between motivation and action (Mogenson et al., 1980). Reward induced by electrical brain stimulation also increases dopamine cell firing and accumbens dopamine release (Moisan and Rompré, 1998; Hernandez and Shizgal, 2009). But dopamine neurons do not respond uniquely to appetitive stimuli. Brishoux et al. (2009) found a sub-group of dopamine neurons that was activated by footshock. Activation of different dopamine subpopulations by stimuli that have opposite motivational valence is likely mediated by different VTA afferent inputs. Ventral midbrain neurons are under the control of numerous glutamatergic afferents originating from cortical and subcortical limbic regions (Carr and Sesack, 2000; Geisler et al., 2007; Omelchenko et al., 2009) and from VTA interneurons (Dobi et al., 2010). Selective activation of VTA glutamatergic afferent inputs from the laterodorsal tegmentum nucleus induces conditioned-place preference while activation of the lateral habenula glutamatergic inputs produces conditioned-place aversion, opposite motivational effects that appear to be mediated by activation of the mesoaccumbens and mesoprefrontal dopamine pathway respectively (Lammel et al., 2012). The rewarding value of a stimulus is signaled when firing of dopamine neurons shifts from a tonic to a phasic mode, and this mode of firing is associated with enhanced dopamine release (Gonon, 1988; Schultz, 2007). The induction of burst firing pattern of dopamine neurons is under the control of glutamatergic afferents from the laterodorsal tegmental nucleus (Lodge and Grace, 2006). Consistently, Lammel et al. (2012) showed that activation of mesoaccumbens dopamine neurons by laterodorsal tegmental afferents was blocked by the AMPA antagonist, CNQX, and the conditioned-placed preference prevented by blockade of accumbens D1/D2- like dopamine receptors. But the modulatory role of glutamate is not limited to dopamine neurons. Glutamatergic terminals from efferent neurons of different limbic nuclei and from local interneurons establish synaptic connections with non-dopamine neurons (Carr and Sesack, 2000; Omelchenko et al., 2009; Dobi et al., 2010). Among these non-dopamine neurons are GABAergic neurons that exert a negative modulation on dopamine activity (Grace et al., 2007). Selective activation of lateral habenula glutamatergic pathway induces excitatory post-synaptic currents (EPSCs) in GABAergic neurons located in the rostromedial tegmental nucleus also named the tailed of VTA (RMtg/tVTA; Lammel et al., 2012). These neurons send an inhibitory input to more rostral VTA dopamine neurons and this signal is triggered by aversive stimuli (see Bourdy and Barrot, 2012). It appears then that glutamate can generate opposite motivational effects by acting either on different VTA dopamine and/or GABA neurons. It is not clear at this point whether these different motivational effects are mediated by different glutamatergic receptors. Both activation and blockade of VTA NMDA receptors increases accumbens dopamine release (French et al., 1993; Karreman et al., 1996; Mathe et al., 1998; Kretschmer, 1999) and stimulates forward locomotion (Kretschmer, 1999; Cornish et al., 2001). Mice (David et al., 1998) and rats (Webb et al., 2012) can be trained to self-administer the NMDA antagonists, AP-5 and AP-7, into the VTA, suggesting that a positive motivational valence predominates following blockade of the receptors. Consistently, Bergeron and Rompré (2013) found that blockade of VTA NMDA receptors enhances brain stimulation reward and operant responding, effects that are mostly likely mediated by NMDA receptors that are devoid of GluN2b subunits. Less is known about the role of VTA AMPA receptors in reward. Activation of VTA AMPA receptors stimulates dopamine impulse flow and accumbens dopamine release (Chergui et al., 1993; Karreman et al., 1996). But blockade of VTA AMPA receptors stimulates locomotor activity, increases operant responding for conditioned reinforcement and induces conditioned-place preference, behaviors that reflect of a positive motivational effect (Harris and Aston-Jones, 2003; Harris et al., 2004; Nolan et al., 2010).

To further clarify the role of VTA AMPA receptors in reward, we investigated the effects of VTA microinjections of the AMPA antagonist, NBQX, on reward and operant responding induced by electrical brain stimulation. We also tested the effects of PPPA, a NMDA antagonist, to determine whether blockade of each receptor sub-type independently at the same VTA sites produces similar effects.

## **EXPERIMENTAL PROCEDURES**

#### **ANIMALS**

Male Long-Evans rats (Charles River Canada) weighting between 300 and 350 g at the time of surgery were used. They were housed 2 per cage (1 per cage after surgery) in a temperature (22◦C) and humidity (40%) controlled room with a 12 h light/dark cycle (lights on at 06:00). They were allowed to habituate for 7 days to the housing environment before surgery and had free access to food and water. All procedures were in accordance with the guidelines of the Canadian Council on Animal Care and all efforts were made to minimize suffering and the number of animals used.

#### **SURGERY**

Rats were anaesthetized with isoflurane (2.5–3.5%, O2 0.7 L/min) and placed on a stereotaxic apparatus. The surface of the skull was exposed between lambda and bregma and burr holes were made into the cranium at the point of insertion of the stimulation electrode and the guide cannulae. A moveable stimulation electrode (Miliaressis, 1981) made from a 0.27 mm stainless steel wire insulated with epoxy (except for the round tip) was implanted into the postero-medial mesencephalon using the following flat skull coordinates: 7.6 mm posterior to bregma, 0.0 mm lateral to the midline and 6.8 mm below the surface of the skull (Paxinos and Watson, 1986). Because electrical stimulation of this area activates reward-relevant axons that travel bilaterally through the VTA (Boye and Rompré, 1996), a guide cannula (HRS Scientific, Montreal, Canada, model C315G) was implanted in each hemisphere; coordinates were 5.4–5.6 mm posterior to bregma, 3.2 mm lateral to the midline (18◦ medio-lateral angle) and 6.4 mm below the surface of the skull; each cannula was closed with an obturator of the same length. A bare wire connected to a male Amphenol connector was wrapped around four stainless steel screws that were threaded into the skull; it served as the inactive electrode. The cannula/electrode assembly was anchored to the skull with dental acrylic. A 0.05 mL injection of Duplocillin LA containing 15,000 I.U. of penicillin was administered (im) to prevent infections. The analgesic Anaphen (5 mg/kg, sc) was administered at the end of surgery.

#### **BEHAVIORAL TRAINING**

Five to seven days after surgery, rats were placed in a test cage (25 × 25 cm) made from polymer walls and one front Plexiglas wall that allowed observation. To reduce disturbance from external noise, test cages were encased in ventilated melamine boxes. Each test cage was equipped with an infrared photocell inside a hole (3 cm diameter and 3 cm deep) located 2 cm above the wiremesh floor. Interruption of the photocell triggered a constantcurrent pulse generator (Mundl, 1980) that delivered a single 400 ms train of 0.1 ms cathodal rectangular pulses. Each train was followed by a period of 600 ms during which the pulse generator could not be triggered (see Bergeron and Rompré, 2013). Using the standard shaping procedure, rats were trained to produce a nose poke to receive a train of stimulation; the current and the frequency were varied until the rat learned the task. If a rat did not learn to respond, the electrode was lowered by 0.2 or 0.4 mm and a new site was tested; the electrode was lowered this way until the rat learned to respond regularly. Following this period of shaping, rats were trained to respond during discrete 55-s trials, each being followed by an interval of 15-s during which stimulation was not available. The beginning of each trial was signaled by 5 trains of noncontingent priming stimulation delivered at a rate of 1 per second. With the current intensity held constant, the frequency (number of pulses per train \* 2.5) was varied from 100 to 30 Hz in 0.06–0.09 log unit steps; this generated a curve relating the number of nose-pokes per trial to the stimulation frequency (rate/frequency or R/F curve). An index of reward threshold was inferred from each R/F curve and it was defined as the pulse frequency sustaining a half-maximal rate of responding (M50). The current intensity was set for each rat to generate a M50 value between 60 and 70 Hz. During the training phase, four R/F curves were determined during consecutive daily test sessions. Testing began when the lower and higher M50 values derived from the last three R/F curves varied by less than 0.1 log unit for 3 days.

#### **DRUG AND VEHICLE TESTS**

A first saline test was carried out to habituate the animals to the injection procedure. This test was similar to the subsequent drug and vehicle tests and consisted of determining four R/F curves before and seven curves after the injections. Bilateral injections were made by inserting into each guide cannula an injection cannula (model C315I) that extended 2 mm beyond the tip of the guide. Each injection cannula was connected with polyethylene tubing to a 2-μl microsyringe and a 0.5μl volume of sterile 0.9% saline was injected into each hemisphere simultaneously with a micro-infusion pump over a period of 60 s; the injection cannulae were left in place for an additional 60 s to allow diffusion into the tissue. Results from this test were not included in the analysis. Drug and vehicle tests began at least 5 days after this first habituation test. On a test day, four baseline R/F curves were first determined over a period of 70 min; the first one was considered as a warmup and discarded. Then each rat was centrally injected with the drug, or its vehicle, using the procedure described above and seven additional R/F curves were determined over a test session that lasted approximately 125 min, starting immediately after the injection. Rats were initially tested with one of two doses of NBQX [80 (30 ng) and 800 (300 ng) pmol/0.5μl/side], or the vehicle; they received all doses (including vehicle) in a counterbalanced order. These doses were based on previous studies (Nolan et al., 2010; Waraczynski et al., 2012) and on preliminary data from our lab. Following completion of the NBQX tests, we tested the effect of a single dose of the NMDA antagonist, PPPA, (0.825 nmol/0.5μl/side), that was previously reported to alter reward (Bergeron and Rompré, 2013). There were at least 7 days between two consecutive drug or vehicle tests.

#### **DATA ANALYSIS**

The mean changes in M50 (reward threshold) and maximal rate of responding (maximal rate of nose poke from each R/F curve) were expressed as the percentage of pre-injection value (baseline) and group means were calculated for each dose and vehicle result. Mean percentage change of both M50 and maximum response were analyzed with a Two-Way (dose × time) analysis of variance (ANOVA) for repeated measures. Homogeneity of variance was tested and square root or log data transformations were performed or Greenhouse-Geiser correction of degrees of freedom used when necessary. Comparisons among means were made with the Holm-Sidak test with the level of significance set at 0.05 (SigmaStat, V11.0, Systat Software Inc.; IBM SPSS Version 20).

## **HISTOLOGY**

At the end of the experiment, the animals were deeply anesthetized with urethane (1.4–2.0 g/kg, i-p.) and the stimulation and injection sites were marked by passing an anodal current of 0.1 mA during 60 s through the electrode and the two injection cannulae that were inserted into the guides. The animals were then perfused with 0.9% saline followed by a 10% formalin solution and the brains were extracted and soaked in a solution containing 3% potassium ferrocyanide, 3% potassium ferricyanide and 0.5% trichloroacetic acid for 24 h. The brains were then rinsed and stored in a 10% formalin solution for several days. They were subsequently frozen and sliced in 40 μm sections that were mounted on gelatin-coated glass slides. The location of the stimulation and of the injections sites were determined under light microscopy from freshly sliced sections and/or sections stained with Nissl's technique. Animals that had both injection sites within the VTA, including the rostral and caudal linear nuclei, the paranigral, parabrachial and the interfascicular nuclei, and the medial part of the substantia nigra between 4.8 and 6.0 mm behind bregma (Paxinos and Watson, 1986) were included in the analyses.

### **DRUGS**

NBQX disodium (2,3,-Dioxo-6-nitro-1,2,3,4-tetrahydrobenzo(f) quinoxaline-7-sulfonamide disodium), and PPPA [(2R, 4S)- 4-(3-Phosphopropyl)-2-piperidinecarboxylic acid], were purchased from *Tocris Bioscience* (Ellisville, MI, USA). They were dissolved in sterile 0.9% saline and stored frozen in 20μl aliquots. Drug solutions were thawed and diluted when necessary just prior to testing. Doses are expressed as salt.

### **RESULTS**

Of the 12 animals initially prepared for the study, 8 were successfully trained and completed the experiment. Histological analysis revealed that the stimulation sites were located within the postero-medial mesencephalon, within the ventral central gray, between the anterior-posterior regions corresponding to 7.3 and 7.8 mm posterior to bregma (**Figure 1**, right panels). The injection sites for the eight animals included in the analyses are shown in the left panels of **Figure 1**. Sites were located between 5.0 and 5.8 mm posterior to bregma, within the ventral part of the VTA, a region that contains neurons activated by rewarding electrical stimulation (Moisan and Rompré, 1998; Marcangione and Rompré, 2008).

### **NBQX ATTENUATED BRAIN STIMULATION REWARD**

Rate-frequency curves obtained from rat SB197 after bilateral VTA injections of vehicle, 80 and 800 pmol/0.5μl/side of NBQX are shown in **Figure 2**. NBQX produced a dose-dependent rightward shift of the R/F curve reflecting a reduction of the rewarding effectiveness of the stimulation; the larger shift was produced

by the high dose. This attenuation of reward was not evident following injections of the vehicle (top panel). **Figure 3** shows mean changes (expressed in % of baseline) in reward threshold (top panels) and maximal response (bottom panels) for each dose of NBQX. The magnitude of the increase in reward threshold produced by NBQX varied with the dose but not the time post-injection. Analysis of variance yielded a significant effect of treatment [*F(*2*,* <sup>21</sup>*)* = 4*.*92, *p <* 0*.*05] but no treatment by time interaction [*F(*12*,* <sup>126</sup>*)* = 1*.*32, *p >* 0*.*05]. *Post-hoc* test confirmed that the increase in threshold measured after injection of 800 pmol of NBQX is significantly different than that measured after vehicle; there is no significant difference between vehicle and 80 pmol. Maximum rate of responding was not significantly altered by NBQX (bottom panel). Analysis of variance yielded no significant effect of treatment [*F(*2*,* <sup>21</sup>*)* = 0*.*97, *p >* 0*.*05] and no treatment by time interaction [*F(*4*.*755*,* <sup>49</sup>*.*923*)* = 1*.*31, *p >* 0*.*05], suggesting that the increase in reward threshold results from a selective reduction in the rewarding effectiveness of the stimulation.

#### **PPPA ENHANCED BRAIN STIMULATION REWARD**

Since AMPA and NMDA receptors can be functionally linked and that both are involved in the modulation of VTA neurons by glutamate, we tested the effect of PPPA, a NMDA antagonist, injected at the same sites in every animal that was tested with NBQX. The dose of PPPA used was found previously to enhance reward when injected into the VTA (Bergeron and Rompré, 2013). **Figure 4** illustrates the R/F curves obtained from the same rat, SB197, after bilateral VTA injections of PPPA (bottom panel). PPPA produced initially a small rightward shift of R/F curve that was followed by a shift to the left; the magnitude of the leftward shifts induced by the vehicle was smaller, suggesting that in this rat PPPA slightly enhanced reward. Grouped

baseline curves, are shown. The *x*-axis represents the number of pulses per train on a log scale.

means from the eight animals tested confirmed the enhancement effect of PPPA (**Figure 5**, top panels). The ANOVA performed on mean changes in reward threshold yielded a significant effect of treatment [*F(*1*,* <sup>14</sup>*)* = 8*.*68, *P <* 0*.*05] and a significant treatment by time interaction [*F(*6*,* <sup>84</sup>*)* = 6*.*45, *p <* 0*.*001]. *Post-hoc* test showed that PPPA significantly reduced threshold compared to vehicle during the first 54 min post-injection (top, left panel). Unlike NBQX, PPPA produced a significant change in maximum response (bottom panels). The ANOVA yielded a significant effect of treatment [*F(*1*,* <sup>14</sup>*)* = 9*.*5, *p <* 0*.*002] and a significant treatment by time interaction [*F(*6*,* <sup>84</sup>*)* = 2*.*8, *p <* 0*.*02]. *Post-hoc* test showed that the maximum response was enhanced compared to vehicle between 54 and 90 min post-injection. It is interesting to notice the discrepancy between the time course of the effect of PPPA on reward threshold and on maximum response.

#### **INVERSE RELATIONSHIP BETWEEN THE ATTENUATION EFFECT OF NBQX AND THE ENHANCEMENT EFFECT OF PPPA ON REWARD: A SITE DEPENDENT EFFECT**

In order to determine whether the opposite effects of NBQX and PPPA on reward was related to the site of injection within the

**maximum response (bottom panels) as a function of time (left panels) after injection of vehicle (VEH) and each of the two doses of NBQX.** Data represent the mean (±sem) from eight animals and are expressed as percentage of baseline on a log scale. Changes (mean ± sem) measured over the entire test session are shown in the right panels. Asterisk indicates a statistically significant difference (*p <* 0*.*05) with VEH.

VTA, we first correlate the changes in reward threshold produced by each drug. The correlation was calculated from the average percent change in reward threshold of the first three measures (between 20 and 54 min), a time at which PPPA and NBQX produced significant behavioral effects; they were transformed in absolute log unit deviation from baseline. Scatter plot presented in **Figure 6** (top panel) shows that the magnitude of the attenuation effect of NBQX on reward was inversely related to the magnitude of the reward enhancement produced by PPPA. This inverse relationship seems to be due to the site of injection over the antero-posterior axis. In effect, the effectiveness of NBQX at attenuating reward is inversely related to the VTA anteroposterior level, being most effective in the anterior VTA (middle panel). The opposite was found with PPPA, and the more posterior was the injection the larger the reward enhancement (bottom panel).

#### **DISCUSSION**

A major finding of this study is that blockade of VTA AMPA receptors following *in situ* microinjections of NBQX, a selective AMPA receptor antagonist (Sheardown et al., 1990), attenuated in a dose-dependent manner reward induced by electrical stimulation of the medial posterior mesencephalon. The reward attenuation reflected by a rightward-shift of the R/F curve was not accompanied by a significant change in maximum rate of

**FIGURE 4 | Rate-frequency curves obtained from rat SB197 prior to (baseline) and injection of vehicle (top panel, 0 nmol) and PPPA (0.825 nmol/0.5 µl/side).** For clarity only four curves, including averaged baseline curves, are shown. The *x*-axis represents the number of pulses per train on a log scale.

represent mean (±sem) from eight animals and are expressed as percentage of baseline on a log scale. Changes (mean ± sem) measured over the entire test session are shown in the right panels. Asterisk indicates a significant difference (∗*p <* 0*.*05, ∗∗*p <* 0*.*01, ∗∗∗*p <* 0*.*001) with VEH.

responding. This selective displacement of the R/F curve on the stimulation frequency axis demonstrates that NBQX produced a specific attenuation of the reward signal initiated by the electrical stimulation (Miliaressis et al., 1986).

**percent change (from baseline) in reward threshold measured for each rat after injection of 0.825 nmol of PPPA (***x***-axis) and 800 pmol of NBQX (***y***-axis).** Scatter plots illustrating the log percent change (from baseline) in reward threshold measured for each rat after injection of 0.825 nmol of PPPA and 800 pmol of NBQX as a function of the location of the injections sites along the antero-posterior axis of the brain are shown in the middle and bottom panels respectively. Correlation index (R) and level of significance are shown along with the linear regression line (plain line) and the 95% confidence interval (dotted lines) in each plot. See text for details.

The most likely hypothesis to account for the attenuation effect of NBQX is a reduction in glutamatergic excitatory inputs to dopamine neurons. Rewarding electrical stimulation increases dopamine cell firing and accumbens dopamine release (Moisan and Rompré, 1998; Hernandez and Shizgal, 2009). Specific reward attenuations are observed following systemic injection of low doses of the dopamine receptor antagonist, haloperidol (Benaliouad et al., 2007). Ventral midbrain microinjections of drugs known to increase and block dopamine impulse flow enhance and attenuate brain stimulation reward respectively (Rompré and Wise, 1989a,b; Wise and Rompré, 1989). Accumbens dopamine release induced by a cue associated with brain stimulation reward is blocked by a VTA microinjection of lidocaine, suggesting that it is mediated by synaptic activation of VTA neurons (Sombers et al., 2009). Glutamate containing terminals establish synaptic contacts with dopamine neurons (Carr and Sesack, 2000; Omelchenko et al., 2009) and brain stimulation reward is associated with an increase in VTA glutamate release an effect that is prevented by infusion of tetrodotoxin (You et al., 2001). Activation of glutamatergic efferents to dopamine neurons produces AMPA mediated EPSCs, increases dopamine cell firing and induces conditioned-place preference (Lammel et al., 2012). It thus appears likely that NBQX attenuated the reward signal initiated by posterior mesencephalic electrical stimulation, a signal carried by glutamatergic inputs to dopamine neurons. This hypothesis is rather inconsistent with previous findings showing that blockade of VTA AMPA receptors stimulates spontaneous motor activity and induces a conditionedplace preference (Harris and Aston-Jones, 2003; Harris et al., 2004; Nolan et al., 2010). It was also shown, however, that VTA microinjection of CNQX, an AMPA antagonist, also prevents morphine-induced increase in dopamine cell firing and morphine-induced conditioned-place preference (Harris et al., 2004; Jalabert et al., 2011). The valence of the motivational effect of AMPA receptor blockade may depend upon the level of activity of the dopamine system. Morphine, like brain stimulation reward, stimulates dopamine impulse flow and release and under such a condition of high dopamine activity blockade of AMPA receptors may reduce the positive reinforcing effect.

Electrophysiological studies have shown that activation of dopamine neurons by glutamate is also mediated by NMDA receptors. Electrical stimulation of VTA afferents to dopamine neurons, for instance, produces NMDA mediated EPSCs (see Ungless et al., 2001) and NMDA produces a dose-dependent increase dopamine cell firing (Seutin et al., 1990; Wang and French, 1993). Deletion of NMDA receptor from dopamine neurons abolishes burst firing, a mode of activity that is associated with enhanced dopamine release and reward (Gonon, 1988; Schultz, 2007; Zweifel et al., 2009). On the basis of these findings, microinjections of the selective NMDA antagonist, PPPA, was expected to attenuate reward. However, PPPA produced an enhancement rather than an attenuation of the reward signal initiated by posterior mesencephalic electrical stimulation. The reward enhancement was unrelated to the increase in maximum rate of responding since these PPPA effects occurred over a different time course. The effects of PPPA are very similar to those reported previously with the same dose by Bergeron and Rompré (2013). To explain this unexpected enhancement of reward and operant responding, Bergeron and Rompré proposed that PPPA acted on NMDA receptors that were mainly composed of GluN2A sub-units and that these NMDA receptors were expressed on non-dopamine neurons. There is no direct evidence that dopamine neurons expressed NMDA receptors that are devoid of GluN2A receptors. However, GluN2A subunits are present in the VTA and some electrophysiological data suggest that they are expressed by non-dopamine neurons (Jones and Gibb, 2005; Suarez et al., 2010).

It is well established that exposure to drug reward induces synaptic plasticity at glutamatergic synaptic inputs to dopamine neurons; this plasticity is characterized by enhanced AMPA EPSCs, due to replacement of GluR2 sub-units for the more conductive GluR1 sub-units, and by reduced NMDA EPSCs (Ungless et al., 2001). Following these synaptic changes, the contribution of AMPA receptor to activation of dopamine neurons is stronger than that of NMDA and under this condition it is expected that blockade of AMPA receptors has a larger impact than blockade of NMDA receptors on the excitatory signal to dopamine neurons. That could explain why NBQX but not PPPA attenuated reward. For this hypothesis to be right, however, one has to postulate that brain stimulation reward induces an increase in AMPA/NMDA EPSC ratio at glutamatergic synapses on dopamine neurons. This might be the case as Boye et al. (2007) have shown that training for brain stimulation reward sensitizes to amphetamine induced forward locomotion, a sensitization effect that is associated with an enhanced AMPA/NMDA ratio (Saal et al., 2003). Carlezon et al. (2001), however, reported a reduction in VTA AMPA receptor sub-unit, GluR1, in animals that were trained to respond for brain stimulation reward, a finding inconsistent with an increase in AMPA/NMDA EPSC ratio.

A large amount of non-dopamine neurons are GABAergic neurons that provide a tonic inhibitory input to dopamine (Grace et al., 2007). Glutamatergic terminals establish synaptic contacts with non-dopamine neurons (Omelchenko et al., 2009) and these are stimulated by activation of NMDA receptors (Wang and French, 1995). It could be then that PPPA enhanced reward by removing the tonic GABAergic inhibition to dopamine neurons, as proposed by Bergeron and Rompré (2013). Since the activity of VTA non-dopamine neurons is also stimulated by activation of AMPA receptors (Wang and French, 1995), it is not clear why NBQX did not produce effects that were similar to those of PPPA. A hypothesis that may reconcile the opposite effects is that NBQX and PPPA acted on different sub-populations of VTA neurons. Our correlation analyses tend to support this hypothesis. First we found that the magnitude of the reward attenuation effect of NBQX was inversely related to the magnitude of the reward enhancing effect of PPPA. This means that at sites where NBQX produced the larger reward attenuations, PPPA produced the smallest reward enhancements. A second analysis revealed opposite correlations between the anterior-posterior VTA level and the absolute change in reward produced by each antagonist. NBQX was most effective when injected in the anterior VTA while PPPA was most effective when injected in the posterior VTA. GABAergic neurons in the RMtg/tVTA receive a dense glutamatergic innervation from the lateral habenula and these provide an inhibitory input to VTA dopamine neurons (Brinschwitz et al., 2010). Selective activation of lateral habenula glutamatergic pathway induces EPSCs in RMtg/tVTA GABAergic neurons and induced conditioned-place aversion (Lammel et al., 2012). Dopamine cell firing is, respectively, reduced and enhanced by activation and inhibition of RMtg/tVTA neurons (Bourdy and Barrot, 2012). Rats self-administer NMDA antagonists into the VTA and the most effective sites are within or near the RMtg/tVTA (David et al., 1998; Webb et al., 2012). It is possible that PPPA enhanced reward by blocking some glutamatergic inputs to RMtg/tVTA neurons. Again, the reason why NBQX was not as effective as PPPA is unclear, particularly because EPSCs induced in RMtg/tVTA neurons by stimulation of glutamatergic inputs are mediated, at least in part, by AMPA receptors (Lecca et al., 2011). It could be that PPPA enhanced reward by acting on RMtg/tVTA and on other VTA neurons since it is still effective when injected in more VTA anterior sites; this conclusion was proposed by Bergeron and Rompré (2013) on the bases of the results that they obtained with another NMDA antagonist, R-CPP.

Shabat-Simon et al. (2008) have shown that the rewarding effect of opiates as measured with the conditioned-place preference and self-administration paradigms is prevented by blockade of VTA AMPA receptors. Interestingly, they showed that CNQX was most and least effective when injected, respectively, into the anterior and posterior VTA. These findings parallel the results that we obtained in the present study with NBQX.

Another observation suggesting that NBQX and PPPA acted on different sub-population of VTA neurons to alter brain stimulation reward is that PPPA, but not NBQX, produced a significant increase in maximum rate of responding. The fact that the enhancement of reward and operant responding occurred over a different time course suggests that they are mediated by the action of PPPA on different substrates. A similar dissociation between changes in reward and operant responding has been reported previously with different pharmacological treatments (Rompré, 1995; Boye and Rompré, 2000; Benaliouad et al., 2009; Gallo et al., 2010). That reinforces the hypothesis that some VTA neurons are more sensitive to NMDA than AMPA receptor blockade and vice versa. Investigation of NMDA and AMPA synaptic responses on different VTA neuron sub-populations in animals previously trained for brain stimulation reward may provide important insights into the identification of the neural substrates of reward.

#### **AUTHOR CONTRIBUTIONS**

Pierre-Paul Rompré, Charles Ducrot and Emmanuel Fortier participated to the design of the experiment. Charles Ducrot, Emmanuel Fortier and Claude Bouchard performed the surgery, the behavioral tests and the analysis the data. All contributed to the final version of the manuscript.

#### **ACKNOWLEDGMENTS**

This work was supported by the National Sciences and Engineering Research Council of Canada (Pierre-Paul Rompré, grant #119057). Authors would like to thank Dr. Giovanni Hernandez for his helpful comments.

### **REFERENCES**


234–241. doi: 10.1016/S0893-133X (01)00232-9


studied by *in vivo* electrochemistry. *Neuroscience* 24, 19–28. doi: 10.1016/0306-4522(88)90307-7


putative rostromedial tegmental neurons, inhibitory afferents to midbrain dopamine cells. *Neuropsychopharmacology* 36, 589–602. doi: 10.1038/npp. 2010.190


ventral tegmental dopamine neurons. *Synapse* 13, 270–277. doi: 10.1002/syn.890130310


J. P., et al. (2009). Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. *Proc. Natl. Acad. Sci. U.S.A.* 106, 7281–7288. doi: 10.1073/pnas. 0813415106

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 09 August 2013; paper pending published: 28 August 2013; accepted: 11 September 2013; published online: 03 October 2013.*

*Citation: Ducrot C, Fortier E, Bouchard C and Rompré P-P (2013) Opposite modulation of brain stimulation reward by NMDA and AMPA receptors in the ventral tegmental area. Front. Syst. Neurosci. 7:57. doi: 10.3389/fnsys. 2013.00057*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2013 Ducrot, Fortier, Bouchard and Rompré. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# GABAergic circuits underpin valuative processing

Dave J. Hayes \*

Division of Brain, Imaging and Behaviour–Systems Neuroscience, Toronto Western Research Institute, Toronto Western Hospital, University Health Network, Toronto, ON, Canada

Keywords: affect, appetitive, aversive, emotion, limbic, punishment, reward, valence

Affect is the fundamental neuropsychological state combining value- and arousal-related processes underpinning emotion and mood. A major goal of the emerging field of affective science is to explain the mechanisms of valuation within the brain. A core network of brain activity is seen across mammals in response to appetitive or aversive stimuli, and appears to be largely independent of stimulus modality (Bissonette et al., 2014; Hayes et al., 2014a). However, the underlying mechanisms of valuation (i.e., appetitive- and aversive-related brain activity) are unclear, and there is particularly little information about how these two valuative networks interact. One candidate which is likely central to the activity of both networks is the neurotransmitter γ-aminobutyric acid (GABA). Here, I briefly discuss some of the evidence pointing to GABA as a central player in mediating appetitive and aversive activity throughout the brain. The broader implication is that the role of GABA in valuative processing may be at the heart of affective regulation, and thus also important for a wide variety of psychological phenomena, from emotion (Stan et al., 2014) and impulsivity (Hayes et al., 2014b) to sense of self (Wiebking et al., 2014a,b).

#### Edited by:

Mikhail Lebedev, Duke University, USA

#### Reviewed by:

Vinay V. Parikh, Temple University, USA Jose Bargas, Universidad Nacional Autónoma de México, Mexico Gregory B. Bissonette, University of Maryland, USA

#### \*Correspondence:

Dave J. Hayes, dave.hayes@utoronto.ca

Received: 11 March 2015 Accepted: 26 April 2015 Published: 12 May 2015

#### Citation:

Hayes DJ (2015) GABAergic circuits underpin valuative processing. Front. Syst. Neurosci. 9:76. doi: 10.3389/fnsys.2015.00076

## Keys to Understanding GABA Circuitry

An exploration of GABA in appetitive and aversive behavior began following its identification in the mammalian brain (Roberts and Frankel, 1950). Although central dopamine was discovered seven years later, many barriers to GABA-related research—including its relative ubiquity throughout the brain, the robust effects of GABAergic drugs administered systemically (e.g., which can easily lead to seizures or immobility), and little knowledge about GABAergic neurocircuitry—has led to a much greater understanding of dopamine in this context (Iversen and Iversen, 2007). Early advances in rodents were nonetheless made delineating key roles for GABA in consummatory behavior, stress, and anxiety (Kelly et al., 1977; Biggio et al., 1990). Studies such as these revealed that beyond dopamine, GABA was involved in mediating motivated behaviors in widespread, but regionally-selective ways (Kelly et al., 1977), and that dynamic cortical and subcortical changes to GABAergic microcircuits were involved (Biggio et al., 1990).

Improved mapping of the extensive brain GABAergic circuits (illustrated in **Figure 1**), coupled with technological advances in detection and manipulation, have partly driven the recent focus of the role of GABA, and its sister excitatory neurotransmitter glutamate, in value-related processing. Moreover, structural advances have continued steadily, from the identification of key GABAergic hubs, such as the basal ganglia and nucleus accumbens (Groenewegen and Russchen, 1984), to more

**Abbreviations:** Ctx, cortex; DA, dopamine; Amyg, amygdala; GABA, gamma-aminobutyric acid; GP, globus pallidus; LHb, Lateral habenula; Motor ctx, motor cortices including premotor, supplementary motor, and frontal eye fields; NAcbC, nucleus accumbens septi core; NAcbS, nucleus accumbens septi shell; PFC, prefrontal cortex; SN substantia nigra; Thal, thalamus; VP, ventral pallidum; VTA, ventral tegmental area.

recent elaborations on the nature of inter- and intra-regional short and long-range GABAergic connections (Caputi et al., 2013). GABA circuits are uniquely situated as both local communicators and whole-brain integrators, given their dynamic control over excitatory and inhibitory signal conduction through axo-dendritic and astrocytic synapses (Frola et al., 2013) and their role in broader oscillatory and synchronistic activities (Melzer et al., 2012).

GABAergic interneurons, particularly parvalbumincontaining, are fundamental drivers of cortical oscillations, which emerging research suggests may be a fundamental context-dependent mechanism for intra- and inter-regional communication (Sohal, 2012; Jadi and Sejnowski, 2015). For instance, beyond the hippocampus and select regions of the neocortex, where these oscillations are better studied, there is also evidence for GABA-driven oscillatory synchronization within the striatum (Sharott et al., 2009)—a hub region connecting the cortex to the basal ganglia and heavily involved in motivation and valuative processing. Moreover, there is evidence that GABA cells are necessary for sustained reward-related signaling, as noted in a study of reversal learning in mice with decreased levels of prefrontal cortical GABAergic interneurons (Bissonette et al., 2015). Going forward, I briefly discuss a sample of recent studies which support the fundamental role of GABA in valuative processing and highlight future directions which will likely contribute to advances in this area.

## GABAergic Microcircuits Regulate Valuative Networks

The present focus on GABA is not intended to ignore the important role played by other neurotransmitters. GABAglutamatergic dynamics are fundamentally important for a complete understanding of circuit dynamics, as has been underscored by findings of neuronal co-release in value-related regions (Root et al., 2014) and multimodal neuroimaging studies in humans (Duncan et al., 2014). Nonetheless, GABA likely plays a unique role in neural control as, for instance, local and long-range GABAergic projections are highly diffuse and GABAergic synapses can precisely target the dendritic shafts of pyramidal cells, allowing for earlier neuronal signal gating in comparison to glutamatergic synapses (Chiu et al., 2013). Moreover, some GABAergic projections are noted to bypass the typical brainstem-thalamo-cortical and cortico-basal ganglia-thalamo-cortical connections, including for instance the direct meso-cortico, meso-limbic and pallidal-cortical pathways (Cohen et al., 2012; Nieh et al.,

2013; Saunders et al., 2015)—see **Figure 1** for illustrative purposes.

Studies investigating neurotransmitter involvement in combined appetitive and aversive processing are limited in general and sparser for GABA (Hayes et al., 2014a). Existing studies in rodents using combined electrophysiological and optogenetic techniques have shown that the majority of ventral tegmental area (VTA) cells respond to value-related stimuli. This is interesting as one-third to half of VTA cells are GABAergic (Swanson, 1982) and may also include cells which co-release dopamine and GABA (Tritsch et al., 2012). While dopaminergic cells typically increase activity to appetitive stimuli as might be expected, some GABAergic cells can increase activity in response to aversive stimuli, be modulated by reward cues (Cohen et al., 2012), and also appear to help signal expected rewards (Lammel et al., 2012). Many of these cells are flexibly responsive to both appetitive and aversive cues, suggesting that they respond to the learned value of the stimulus and not its static properties (Kim et al., 2010). Networks of intra-VTA GABAergic cells, however, show increased correlations to theta band power in response to appetitive, but not aversive, cues (Kim et al., 2012) and so might be involved in the integration of appetitive networks needed to learn about, and maintain, reward-related behaviors such as electrical brain self-stimulation (Steffensen et al., 2001; Lassen et al., 2007).

Though the nucleus accumbens is comprised almost entirely of short- and long-range GABAergic projections and is a high-density region tasked with integrating motor, sensory, and valuative/motivational signals (Mogenson et al., 1980), it has been underexplored in this context (Carlezon and Thomas, 2009; Hayes et al., 2014a,b). Feeding studies have shown that presumably inhibiting the GABAergic cells of the accumbens shell corresponds to increases in appetitive feeding behaviors (Stratford and Kelley, 1997). Others have identified a rostrocaudal gradient in the shell, whereby GABA<sup>A</sup> receptor activation in the rostral shell leads to increases in appetitive feeding, conditioned place preference and sucrose responding, and caudal activation results in aversive, defensive, behaviors (Reynolds and Berridge, 2002). Although our group found similar orexigenic effects of GABAergic drugs in the rostral shell, we noted a clear increase and decrease in electrical brain self-stimulation following intra-accumbens GABA<sup>A</sup> receptor blockade and stimulation, respectively, at the same injection sites (Hayes et al., 2011). These seemingly opposing results suggest that although GABAergic accumbens microcircuits are fundamental to valuative processing, subtle differences in their activation are likely involved in differentiating responses to different kinds of appetitive and aversive stimuli. One clear possibility is that valuative GABAergic signaling is contextdependent on the temporal interplay with other biochemicals, such as glutamate (Clements and Greenshaw, 2005; Richard and Berridge, 2011).

The evidence for GABA as a central regulator in valuative processing is not simply limited to the VTA or accumbens. For instance, the GABAergic lateral habenula inhibits VTA-related medial prefrontal cortex dopaminergic projections and local GABAergic cells which are both tied to aversive processing and behavioral output (Lammel et al., 2012; Shabel et al., 2012). The activation of nucleus accumbens GABAergic cells are also known to inhibit ventral pallidal activity (Wang et al., 2014), which is an area that contributes to changes in valuative responding (Tindell et al., 2004). For example, inactivation of pallidum by stimulating inhibitory GABA<sup>A</sup> receptors results in the elimination of rewardrelated saccade responding in rhesus monkeys (Tachibana and Okihide, 2013). Interestingly, a recent study suggested that increases in feeding following intra-pallidal, but not intraaccumbens, GABA<sup>A</sup> receptor antagonism corresponded to a specific "fat craving" signal instead of general increases in appetitive activity (Covelo et al., 2014). Aversion-related activity may also involve GABAergic inhibition of infralimbic cortex in rats, a homolog to the primate medial prefrontal cortex, as has been demonstrated by showing that pain-related GABAergic inhibitions in the prefrontal cortex are reversed following the local injection of GABA<sup>A</sup> receptor antagonists (Ji and Neugebauer, 2011)—though infralimbic activation in the absence of pain may also be anxiogenic (Bi et al., 2013). Moreover, GABA<sup>A</sup> receptor activation in the infralimbic cortex increases impulsive responding (Murphy et al., 2012), while activation in the prelimbic cortex reduces aversive behaviors such as fearpotentiated startle and freezing (Almada et al., 2015).

Beyond the regions of the extended amygdala noted above (e.g., nucleus accumbens shell, habenula), the amygdala itself is one region that deserves a brief note. Although its role in processing aversive stimuli is well-established, our understanding of its role in appetitive encoding is relatively recent —it is also considered mainly as a singular structure in human studies, though it is known to be made up of a number of uniquelyconnected nuclei (e.g., central, basolateral, and lateral) comprised of differing cell types (Phelps and LeDoux, 2005; Janak and Tye, 2015). Few studies have looked at how this region processes both appetitive and aversive stimuli and none have focused on GABAergic cells. Studies in primates and rodents using single cell basolateral and central area recordings showed that at least half of the cells sampled were value-responsive, sensory modalityindependent, consisted of general responders and those with preferences for either appetitive or aversive stimuli, and that there was no clear anatomical distribution for such cells across each subregion (Paton et al., 2006; Belova et al., 2007, 2008; Shabel and Janak, 2009). At least one study has also shown the importance of safety signaling for cells throughout the basal amygdala, identifying subpopulations of cells that respond to combinations of value and safety cues (Sangha et al., 2013). Of further interest, recent reports have described long-range GABAergic hippocampal- and intra-amygdalar projections likely involved in valuative processing (Bienvenu et al., 2015; McDonald and Zaric, 2015).

Taken together, and as underscored by **Figure 1**, these studies support the hypothesis that interconnected GABAergic microcircuits are a binding feature of valuative processing. Indeed, these circuits may be at the heart of whole-brain reinforcement or valuative networks (English et al., 2011; Vickery et al., 2011). Moreover, they likely also contribute to the intraregional integration and differentiation of appetitive and aversive signals (Hayes et al., 2014a).

## Future Considerations

A corollary of the hypothesis above is the emphasis on undiscovered intra- and inter-regional value-related GABAergic signaling throughout the brain. In this vein, our group showed in a human multimodal neuroimaging study that GABA<sup>A</sup> receptors in medial prefrontal cortex are negatively correlated to aversive signals in both the medial prefrontal region itself as well as distant sensorimotor clusters, but that GABA<sup>A</sup> receptors within different clusters of sensorimotor cortex respond differentially to the context of aversive stimuli (Hayes et al., 2013). Moreover, we reviewed the human and rodent literature on impulsivity and GABA, and concluded that GABAergic networks throughout at least cortico-basal ganglia regions acted as a common substrate of impulsive behaviors (Hayes et al., 2014b).

We believe that affective/valuative networks are at the core of many psychological phenomena, from emotion (Stan et al., 2014) and impulsivity to sense of self (Wiebking et al., 2014a,b), and may even be fundamental to the initial recruitment of cognitive control (Inzlicht et al., 2015). Indeed, some neuroimaging studies have identified a common wide-spread network of regions which may be common to these processes (Northoff and Hayes, 2011; Amft et al., 2015). Future studies using complex multimodal neuroimaging approaches will be necessary to elaborate and connect the present neural and biochemical findings in humans (Duncan et al., 2014). These should also include in vivo neuroanatomical explorations of affective circuitry white matter, by for instance using recent advancements in multitensor tractography (Chen et al., 2014).

Conceptually, it is important to note that when discussing a "system" of any sort in neuroscience (e.g., GABAergic, valuative) one is often taking a broad sum-of-parts operational definition. Because the entirety, and mechanistic underpinnings, of such systems are incomplete, and cannot be fully understood in isolation, these terms become placeholders for our dynamic knowledge (see LeDoux, 2012; Gross and Barrett, 2013; Hayes et al., 2014a for related discussions). For example, future research will have to continue to identify clusters of GABAergic cells which make up value-processing microcircuits as well as their connections to other value- and non-value related clusters, including other cell types, such that a better understanding of their true function becomes clearer (and probably resulting in clearer delineations between multiple "systems"). Analogous advances in network neuroscience have been made to identify

## References


many major nodes/hubs (i.e., clusters), edges (i.e., connections), and the interactions within and between such brain networks (Behrens and Sporns, 2012)—while most of this work is being done in humans, progress on the vast animal literature has also been made (Ikemoto, 2010). At this point, the greatest advances at the molecular-cellular level of understanding are likely being made through the identification and spatiotemporal electrochemical characterization of value-related microcircuits, for instance in the traditional mesocorticolimbic circuit (e.g., Nieh et al., 2013; Lammel et al., 2014). Indeed, the bulk of information connecting behavior to underlying mechanisms is confined to this value-related circuit, with disparate results for other areas. Moreover, behavioral-cellular/neurochemical connections related to GABA have been mainly restricted to single regions, such as the VTA, nucleus accumbens, and areas of the prefrontal cortex, although recent experiments have focused increasingly on inter-regional interactions (Lammel et al., 2012; Shabel et al., 2012; Hayes et al., 2013; Wang et al., 2014).

Going forward, future studies will also need to clearly answer the question of how brain areas within similar affective networks parse aversion- and appetitive-related neural signals, while also providing mechanisms for fast intraregional communication. Recent reviews of the human and animal literature have provided some insight to this question (Bissonette et al., 2014; Hayes et al., 2014a; Lindquist et al., 2015), but more work is needed. For instance, the significance, if any, of structures which show asymmetrical activity is unclear, e.g., appetitive and aversive stimuli may result in more dopamine in the right and left accumbens, respectively (Besson and Louilot, 1995). Moreover, how these neural processes are translated at the behavioral or "cognitive" level is of equal importance. Can appetitive and aversive stimuli be subjectively, consciously, experienced simultaneously (Barrett et al., 2007)? Why do some people experience painful experiences as pleasurable, and is there any connection to those who prefer to self-administer aversive stimuli rather than be alone with their own thoughts (Wilson et al., 2014)?

## Acknowledgments

This work was partially supported by a Canadian Institutes of Health Research Postdoctoral Fellowship. Thanks to Dr Jeffrey Dalley for related discussions.


fMRI-PET investigation in healthy humans. Neuropsychopharmacology 38, 1438–1450. doi: 10.1038/npp.2013.40


Root, D. H., Mejias-Aponte, C. A., Zhang, S., Wang, H. L., Hoffman, A. F., Lupica, C. R., et al. (2014). Single rodent mesohabenular axons release glutamate and GABA. Nat. Neurosci. 17, 1543–1551. doi: 10.1038/nn.3823 Root, D.H., Mejias-Aponte, C.A., Zhang, S., Wang, H.L., Hoffman, A.F., Lupica, C.R., Morales, M.


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Hayes. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

## A novel V1a receptor antagonist blocks vasopressin-induced changes in the CNS response to emotional stimuli: an fMRI study

*Royce J. Lee1 \*, Emil F. Coccaro1, Henk Cremers 1, Rosemary McCarron1, Shi-Fang Lu2,3, Michael J. Brownstein2 and Neal G. Simon2,3*

*<sup>1</sup> Clinical Neurosciences and Psychopharmacology Research Unit, Department of Psychiatry and Behavioral Neurosciences, The University of Chicago, Chicago IL, USA*

*<sup>2</sup> Azevan Pharmaceuticals, Inc., Bethlehem, PA, USA*

*<sup>3</sup> Department of Biological Sciences, Lehigh University, Bethlehem, PA, USA*

#### *Edited by:*

*Dave J. Hayes, University of Toronto, Canada*

#### *Reviewed by:*

*Thomas F. Münte, University of Magdeburg, Germany Jitendra Sharma, Massachusetts Institute of technology, USA Liliana R. Ramona Demenescu, RWTH Aachen, Germany Izelle Labuschagne, Monash University, Australia*

#### *\*Correspondence:*

*Royce J. Lee, Clinical Neurosciences and Psychopharmacology Research Unit, Department of Psychiatry and Behavioral Neurosciences, MC 3077, The University of Chicago, 5841 S. Maryland Ave., Chicago, IL 60637, USA*

*e-mail: rlee@yoda.bsd.uchicago.edu*

**Background**: We hypothesized that SRX246, a vasopressin V1a receptor antagonist, blocks the effect of intranasally administered vasopressin on brain processing of angry Ekman faces. An interaction of intranasal and oral drug was predicted in the amygdala.

**Methods**: Twenty-nine healthy male subjects received a baseline fMRI scan while they viewed angry faces and then were randomized to receive oral SRX246 (120 mg PO twice a day) or placebo. After an average of 7 days of treatment, they were given an acute dose of intranasal vasopressin (40 IU) or placebo and underwent a second scan. The primary outcome was BOLD activity in the amygdala in response to angry faces. Secondary analyses were focused on ROIs in a brain regions previously linked to vasopressin signaling.

**Results**: In subjects randomized to oral placebo-intranasal vasopressin, there was a significantly diminished amygdala BOLD response from the baseline to post-drug scan compared with oral placebo-intranasal placebo subjects. RM-ANOVA of the BOLD signal changes in the amygdala revealed a significant oral drug × intranasal drug × session interaction (*F(*1*,* <sup>25</sup>*)* = 4*.*353, *p <* 0*.*05). Follow-up tests showed that antagonism of AVPR1a with SRX246 blocked the effect of intranasal vasopressin on the neural response to angry faces. Secondary analyses revealed that SRX246 treatment was associated with significantly attenuated BOLD responses to angry faces in the right temporoparietal junction, precuneus, anterior cingulate, and putamen. Exploratory analyses revealed that the interactive and main effects of intranasal vasopressin and SRX246 were not seen for happy or neutral faces, but were detected for aversive faces (fear + anger) and at a trend level for fear faces.

**Conclusion**: We found confirmatory evidence that SRX246 has effects on the amygdala that counter the effects of intranasal vasopressin. These effects were strongest for angry faces, but may generalize to other emotions with an aversive quality.

**Keywords: vasopressin, depression, fMRI, anger, neuropeptides, stress, amygdala, parietal**

#### **INTRODUCTION**

Vasopressin (AVP) is a mediator of social and emotional behavior in many species (Garrison et al., 2012) including humans, and it has been suggested that AVP receptor antagonists might be useful for treating stress-related neuropsychiatric problems including inappropriate aggression, post-traumatic stress disorder, and major depression (Meyer-Lindenberg and Tost, 2012). The hypothesis that these disorders might respond to AVP antagonists is supported by preclinical and clinical studies showing that CNS vasopressinergic signaling is deregulated in patients with these indications (reviewed in Simon et al., 2008).

There are two AVP receptor subtypes in the brain that are potential therapeutic targets: V1a and V1b. Small-molecule V1b receptor antagonists have thus far not proven effective for the treatment of depression (Griebel et al., 2012), which may be due to the relatively low expression and restricted, hypothalamic distribution of the receptor. In contrast, the V1a receptor is the dominant CNS subtype and is found throughout the limbic system and several cortical regions, providing a strong rationale for determining the potential role of this receptor in the regulation of emotion. To this end, SRX246 was developed as a novel, AVPR1a antagonist that penetrates the blood brain barrier and has CNS effects in multiple preclinical models (Ferris et al., 2008; Simon et al., 2008; Fabio et al., 2013). Because there is currently no PET ligand that can be used to establish AVPR1a target engagement, we felt that it was important to demonstrate that SRX246 produced CNS effects in humans after oral dosing before commencing clinical trials in patients. This step is in accord with the growing consensus that evidence of brain penetration and pharmacological effect is a vital early component of drug development for CNS indications (reviewed in Griebel and Holsboer, 2012).

Due to the impermeability of the blood-brain barrier to orally or intravenously administered neuropeptides, human research regarding central vasopressin signaling has relied on intranasal administration of vasopressin. After intranasal administration, a small amount of vasopressin crosses the blood brain barrier (Riekkinen et al., 1987; Born et al., 2002). A series of human studies has found that intranasally administered vasopressin enhances attention to and memory of emotional facial expressions (Thompson et al., 2004, 2006), Several recent studies have examined the effects of intranasal vasopressin on regional brain activity as measured by the fMRI blood oxygen level dependent (BOLD) response to experimental emotional and social stimuli. In total, intranasal vasopressin appears to increase the neural response to socially relevant stimuli in circuits that mediate emotion regulation (subgenual cingulate: Zink et al., 2010 and Brunnlieb et al., 2013b), theory of mind (inferior parietal lobe: Zink et al., 2011; superior temporal sulcus: Brunnlieb et al., 2013a; posterior cingulate, Brunnlieb et al., 2013b), and social recognition (lateral septum; Rilling et al., 2012). Notably, only one of the five studies found evidence for direct vasopressin on amygdala BOLD (Brunnlieb et al., 2013b). The stimuli in this study consisted of line drawings of aversive social interactions, rather than face stimuli. The absence of a direct effect of intranasal vasopressin on amygdala reactivity to faces was initially unexpected (Zink et al., 2010), given the a priori hypothesis that vasopressin would increase amygdala BOLD response to aversive faces. With regards to vasopressin modulation of amygdala response to face stimuli, indirect effects on the amygdala were detected when examining functional connectivity of the subgenual to supragenual cingulate (Zink et al., 2010). Thus, the human vasopressin challenge literature has found that vasopressin has effects on behavior predicted by preclinical models, increasing reactivity to social stimuli. The brain imaging literature has found evidence of altered neural function following vasopressin challenge in a brain regions previously implicated in social cognition. Because vasopressin as a pharmacological probe in these studies lacks specificity for the V1a or V1b receptor, this work has been unable provide specific information regarding which subtype vasopressin receptor is involved. Progress in this area would require either vasopressin V1a or V1b specific agonists safe for human use, or specific antagonists.

A novel small-molecule vasopressin antagonist (SRX246; Azevan Pharmaceuticals) with a high degree of selectivity for the AVPR1a has undergone preclinical and early clinical testing. SRX246 crosses the blood brain barrier and binds to V1a receptors with a high degree of selectivity (Fabio et al., 2013). Its ability to selectively target AVPR1a is reflected in its ability to reverse AVPR1a-mediated stress reactivity and neural response to intruder threat (Ferris et al., 2008). Because it has undergone successful Phase I single-ascending-dose and 14-day multiple ascending dose clinical trials with a benign safety profile (detailed in Fabio et al., 2013), it became available for use as a pharmacological probe of the AVPR1a.

A double-blinded, placebo controlled experiment was conducted using challenge with intranasal vasopressin and treatment with SRX246, an oral AVPR1a antagonist. We asked normal male volunteers to look at emotional faces while their brains were scanned using fMRI. Half of the subjects were then given SRX246 for an average of 7 days, and half were given placebo-containing capsules. An hour before they were re-scanned, half of the subjects in each condition were given intranasal arginine vasopressin (AVP). AVP challenge was used to maximize the chance of finding an effect of AVPR1a blockade in a sample of healthy subjects, who presumably did not exhibit excessive central AVP signaling. We then looked for evidence that brain regions were activated when patients looked at angry faces vs. a fixation point, that vasopressin affected such activation, and that SRX246 blunted the responses seen in the presence or absence of AVP. Because the amygdala is known to express vasopressin AVPR1a receptors (Young et al., 1999; Huber et al., 2005; Stoop, 2012) and to be activated during explicit recognition of emotional (including angry) faces (Derntl et al., 2009), it served as the principal region of interest in the experiment. Secondary analyses examined effects in additional candidate regions found to be modulated by vasopressin in previous research: the temporoparietal junction, precuneus, anterior cingulate, subgenual cingulate, and putamen.

#### **METHODS**

#### **PARTICIPANTS**

All study procedures were approved by the Institutional Review Board of The University of Chicago. All subjects provided written, informed consent. Subjects were recruited from the Chicago region with IRB approved advertisements in local media. Because of previous literature indicating the possibility of sexually dimorphic effects of vasopressin (Thompson et al., 2006), only male subjects were studied to preserve statistical power. Twenty-nine healthy male subjects (ages 18–55) were studied after they were verified to meet inclusion/exclusion criteria by medical exam and psychiatric screening with semi-structured SCID and SID-P IV interviews. Inclusion criteria included being medically and psychiatrically healthy, no past history of an Axis I or II psychiatric disorder, no obstruction of either nostril to the olfactory epithelium, normal screening blood and urine tests, non-smoking, normal body weight, right handedness, and no current use of prescription medications or drugs.

#### **EXPERIMENTAL DESIGN**

The study was a double-blinded, between subjects design with two fMRI scanning sessions. The first session occurred before any drug administration (Session 1). The second session (Session 2) followed randomization, treatment for a minimum of 5 days with oral SRX246 or placebo, and 45 min after acute administration of intranasal vasopressin or matching placebo (see **Figure 1**). A between subjects design was chosen to eliminate possible carry over effects of SRX246 or IN vasopressin on the fMRI measures. To mitigate the effect of individual differences in brain reactivity to face stimuli, the design included the first baseline scan to evaluate patterns of change from the first to the second scan.

Two experimental drugs were administered in double-blinded fashion: chronic oral SRX246 and acute intranasal (IN) vasopressin, both with placebo counterparts. After the first scanning session (Session 1, described in the next section), subjects were randomized to receive 5 days of oral SRX246 (120 mg by mouth, twice a day; *n* = 15) or equivalent dosing of the matching pill placebo (*n* = 14). The mean of number of days between the first and second scanning session was 7.3 (*SD* = 1*.*3); the SRX246 and placebo treatments continued until the day of the second scanning session. The second session began with randomization to either IN AVP or IN placebo. Vasopressin was prepared by the research pharmacy of the University of Chicago General Clinical Research Center. Forty IU synthetic vasopressin (8-arginine-vasopressin, Pitressin, Monarch Pharmaceuticals) was dispensed using Good Clinical Practices into two intranasal atomizers (MAD 300; LMA North America Inc., San Diego CA). IN placebo was prepared from a commercial nasal saline solution to mask the mild scent of Pitressin solution. IN drug was administered in 4 puffs (0.2 mL) per nostril over 15 min by the research staff to subjects reclining on an exam table with their heads tilted back. Subjects rested on the examination table by themselves with the examination room door open until the beginning of the second fMRI scanning session, timed to begin 45 min after IN drug administration. This time point was chosen based on the time course of CSF levels of vasopressin after IN administration (Born et al., 2002) and to remain consistent with previous fMRI studies of IN vasopressin effects.

The first and second scanning sessions were identical. In the scanner, subjects viewed 4 blocks of each emotional facial expression, using stimuli from the Ekman Pictures of Facial Affect stimulus set (angry, neutral, happy, fear). The stimuli were presented over 4 runs. To preserve statistical power, analyses focused on the neural responses to angry faces, based on a priori hypotheses regarding the relevance of angry faces to vasopressin function. Each emotion block lasted 20 s and consisted of 5 faces displayed for 4 s in the center of the screen, with no interstimulus interval. The behavioral task was to identify the valence of the emotion (positive, negative, neutral) by button press. An explicit emotion paradigm was employed based on a large amount of high resolution fMRI data demonstrating that this type of task evokes a readily detectable amygdala response (Pessoa et al., 2002; Habel et al., 2007; Derntl et al., 2009; Dyck et al., 2011). Fixation cross was chosen as a contrast condition rather than neutral faces for this study in order to maximize statistical power by avoiding the variability associated with contrasts of emotional faces with neutral faces. Neutral faces also activate the amygdala (Fitzgerald et al., 2006; Derntl et al., 2009). The extent and variability of this activation reduces BOLD signal intensity when neutral faces are used as a contrast condition (Mattavelli et al., 2013) and reduces the reliability of amygdala response to emotional faces (Johnstone et al., 2005).

fMRI data were acquired using a Philips Achieva Quasar 3T MRI scanner at the Brain Research Imaging Center at The University of Chicago. For identification of landmarks and orientation of follow up scans, low-resolution structural MRI was obtained with a T1-weighted spin-echo sequence (*TR* = 600 ms, *TE* <sup>=</sup> 10 ms; *FA* <sup>=</sup> <sup>70</sup>◦, *FOV* <sup>=</sup> 23 cm2, slice thickness/gap <sup>=</sup> 4.0/0.5). fMRI images were obtained with high-field functional MRI utilizing T2∗-weighted echo planar imaging with BOLD (blood oxygenation level dependent) contrast (echo time/*TE* = 20 ms, repetition time/TR of 2000 ms, flip angle of 80◦, field of view of 230 mm2, 30 4 mm oblique axial slices approximately parallel to the AC-PC line, 0.5 mm slice gap). A modified high efficiency z-shim compensation was applied to the 4 slices covering the orbitofrontal cortex (Du et al., 2007) to minimize susceptibility artifacts. Acceptable signal to noise ratio was confirmed for the ventral brain and medial temporal lobes.

fMRI data were pre-processed using SPM8 software (Wellcome Department of Cognitive Neurology, London). Images were band pass filtered to remove very low frequency drift artifact and high frequency, non-physiologic noise. Images acquired during excessive movement (≥3 mm X, Y, or Z spatial displacement and/or 5◦ of rotation) were excluded from the analysis. Motion in the three planes was recorded and images were motion corrected relative to the first image of the first run, normalized to a Montreal Neurological Institute template, resampled to 2 mm3 voxels, and smoothed with an 8 mm<sup>3</sup> kernel. T2<sup>∗</sup> functional data of each subject were examined for susceptibility artifacts and/or signal loss near the principal regions of interest.

T-statistical images were generated for the first and second fMRI sessions separately to confirm that the task led to the expected pattern of regional brain activation. To control for Type I error, Family Wise Error (FWE) was utilized for the entire brain region in voxelwide analyses (*p <* 0*.*05, cluster size *>* 10 contiguous voxels). Analyses were focused on angry face contrasts given the limited power of the study. An exploratory analysis of other facial emotion conditions is provided in Supplementary Material.

To establish the effect of the IN vasopressin probe in the principal ROI of the left and right amygdala, data were compared between the IN vasopressin and IN placebo group within the sub-sample of subjects randomized to oral placebo (*n* = 14). One-Way ANOVA was conducted on whole-brain contrast images of angry faces vs. fixation point, with the factors of session and IN drug; the hypothesized IN drug effect was tested with the statistical interaction between the two factors by confirming significant clusters of activation within the anatomical amygdala. To balance concerns of Type I and Type II error, correction for multiple comparisons on significant clusters utilized familywise error correction (FWE) within the small volume of the anatomic amygdala as defined by Wake Forest University (WFU) Pickatlas (Maldjian et al., 2003; *p <* 0*.*05, one-tailed). Parameter estimates (β weights) of average activation were extracted from the anatomical amygdala ROI (WFU Piackatlas) and exported to SPSS 18 (IBM) for statistical analysis. Two separate paired *t*-tests were then conducted in the seven subjects receiving intranasal vasopressin and the seven subjects receiving intranasal placebo. To account for baseline differences, follow up tests were repeated using ANCOVA to compare IN vasopressin vs. IN placebo on the extracted Session 2 amygdala BOLD signal, covarying for Session 1 in the 14 subjects receiving oral placebo.

The primary hypothesis of the study, that SRX246 engages its target by blocking vasopressin effects on the amygdala, was tested with repeated measures (RM) ANOVA for the interaction of the factors of oral drug (SRX246 vs. oral placebo), intranasal drug (IN vasopressin vs. IN placebo), side (left vs. right), and session (Session 1 vs. Session 2) on the extracted parameter estimates of average amygdala BOLD response to angry faces. Repeated Measures (RM) ANOVA was conducted with the between-subjects factors of oral drug and intranasal drug, with the within-subjects factor session. The statistical significance threshold was set at *p* = 0*.*05, 2-tailed. Significant main effects and/or interactions were followed up with appropriate *post-hoc*, 2-tailed tests. Paired *t*-tests were conducted to compare Session 1 and Session 2 BOLD response in the four drug subgroups. Additionally ANCOVA of Session 2 BOLD response covarying for Session 1, comparing IN vasopressin to IN placebo was conducted in subjects randomized to oral SRX246 and oral placebo.

In exploratory analyses, the effects of SRX246 and vasopressin on BOLD reactivity were assessed in ROIs previously found to be modulated by vasopressin or expressing the AVPR1a: subgenual cingulate (Zink et al., 2010), anterior and posterior cingulate (Zink et al., 2010), precuneus (Brunnlieb et al., 2013a), temporoparietal junction (Rilling et al., 2012; Brunnlieb et al., 2013b), and caudate/putamen (Hammock and Young, 2006). Voxel-wide, whole brain analysis was performed on Session 2 data, comparing the response to angry faces vs. fixation point between the SRX246 to oral placebo treatments. ROI analyses were performed using the corresponding anatomical structure [Automated Anatomical Atlas (AAL) SPM], with the significance threshold set at *p <* 0*.*05, FWE corrected for the anatomical search volume. Individual differences in the tortuous gyral/sulcal morphology of the temporoparietal region make spatial definitions of it unreliable for group analyses (Tzourio-Mazoyer et al., 2002). Instead, the ROI was a 9 mm sphere centered on MNI coordinates reported in the most recent of a series of studies that have found functional activations in the right temporoparietal junction during theory of mind related tasks (54, −54, 22; Koster-Hale et al., 2013). Average activations within these ROIs at Session 2 were extracted as β weights into SPSS for ANCOVA, covarying for baseline differences. Because this is the first fMRI study of SRX246, exploratory, whole brain exploratory comparisons of SRX246 vs. placebo on BOLD response to angry faces (vs. fixation point) during Session were conducted. Uncorrected results are provided in **Table 3**.

#### **BEHAVIORAL DATA**

Accuracy and reaction time were recorded and analyzed by RM-ANOVA, with the within subjects factor of session and between subjects factors of oral and intranasal drug. Significant interactions were followed up with *post-hoc t*-tests (2-tailed). To relate the behavioral measures with brain response, change in reaction time and accuracy (Session 2 – Session 1) was correlated with change in amygdala BOLD (Session 2 – Session 1).

#### **SIDE EFFECTS**

The Adverse Events Questionnaire (AEQ) was used to measure a range of possible somatic and psychological side effects. Suicidal symptoms were assessed with the Columbia Suicide Severity Rating Scale (C-SSRS; validation in Posner et al., 2011). Depressive symptoms were measured with the Beck Depression Inventory II (BDI-II; Beck et al., 1996). Differences between drugand placebo-treated subjects in safety (vital signs and laboratory parameters), side effects (AEQ, BDI-II, CSSR-S), and EKG data (PR interval, QT, QTc) were assessed with a series of separate RM-ANOVAs for each measure.

## **RESULTS**

#### **TASK-RELATED ACTIVATIONS ON SESSION 1 AND SESSION 2**

BOLD responses to angry faces vs. fixation point were observed in the visual cortex, right and left amygdala, temporal pole, and ventral prefrontal cortex for Session 1 and 2 (**Table 1** for Session 1, **Table 2** and **Figure 2** for Session 2). Areas of activation were within regions expected to show task-related changes in brain activity. Attenuated intensity and cluster size of BOLD was observed from Session 1 to Session 2 in the entire sample. Contrasts of angry faces vs. neutral faces did not result in measurable amygdala BOLD signal suitable for analysis of drug related effects (see Supplementary Material 3.1∗).

#### **EFFECT OF IN VASOPRESSIN**

One-Way ANOVA in the subsample of 14 subjects who did not take SRX246 revealed a significant interaction of session × IN



*Clusters of significant activation (>10 contiguous voxels) that survive statistical threshold for multiple comparisons (FWE across the entire brain, p < 0.05).*



*Clusters of significant activation (>10 contiguous voxels) that survive statistical threshold for multiple comparisons (FWE across the entire brain, p < 0.05).*

drug within both the left and right amygdala (*p <* 0*.*05, FWE corrected for the volume of the region; depicted in **Figure 3**). Followup paired *t*-tests comparing amygdala BOLD intensity across the first and second session revealed that in subjects receiving IN vasopressin, BOLD signal intensity significantly decreased in the right amygdala [*t(*1*,*6*)* = −2*.*560, *p <* 0*.*05] and at a trend level in the average of both sides [*t(*1*,*6*)* = −2*.*058, *p* = 0*.*09], but no effect was seen in the left amygdala. No significant pairwise difference in amygdala BOLD between the two sessions was seen in the subjects receiving IN placebo. ANCOVA of the amygdala BOLD response in subjects receiving oral placebo produced similar results, with IN vasopressin associated with significantly decreased BOLD response at Session 2, controlling for Session 1 BOLD response, in the right amygdala [*F(*1*,* <sup>13</sup>*)* = 5*.*013, *p <* 0*.*05] and at a trend level for the combined left and right amygdala [*F(*1*,* <sup>13</sup>*)* = 4*.*538, *p* = 0*.*057]. No effect was found in the left amygdala. See Supplementary Material 3.2∗ for analyses of other emotion conditions.

#### **PRIMARY HYPOTHESIS: INTERACTION OF ORAL SRX246 AND INTRANASAL VASOPRESSIN ON AMYGDALA BOLD RESPONSE**

The primary hypothesis that SRX246 blocks effects of vasopressin on the amygdala was confirmed. RM ANOVA of extracted parameter estimates from the left and right amygdala in the entire sample revealed a significant 3 way interaction of session × oral drug × IN drug in contrasts of angry faces vs. fixation point [*F(*1*,* <sup>25</sup>*)* = 4*.*353, *p <* 0*.*05]. Follow up testing of the interaction in the four drug subgroups with paired *t*-tests comparing Session 1 to Session 2 BOLD response in the left and right amygdala confirmed that a significant Session 1 vs. Session 2 difference was found only in the subgroup of subjects randomized to oral placebo and IN vasopressin (as described in the Effect of IN Vasopressin). Subjects randomized to oral SRX246 and IN vasopressin did not show a Session 1 to Session 2 difference [*t(*1*,*7*)* = 0*.*819, *p* = 0*.*44 (**Figure 4**)]. ANCOVA of Session 2 amygdala BOLD, covarying for Session 1, confirmed that in subjects taking oral SRX246, no effect of IN vasopressin was seen in the combined amygdala [*F(*1*,* <sup>14</sup>*)* = 0*.*384, *p* = 0*.*55], right amygdala [*F(*1*,* <sup>14</sup>*)* = 0*.*067, *p* = 0*.*8], or left amygdala [*F(*1*,* <sup>14</sup>*)* = 1*.*063, *p* = 0*.*32]. There was no main effect of SRX or vasopressin on amygdala BOLD response. See Supplementary Material 3.3∗ for analyses of other emotion conditions.

#### **SECONDARY ANALYSES**

Cross-sectional, Session 2 comparison of SRX246 (*n* = 15) vs. oral placebo (*n* = 14) revealed clusters of activation that survived small volume correction in contrasts of angry faces

**FIGURE 2 | Regions of BOLD signal intensity for Session 2 angry faces** *>* **fixation point.** Results of the Session 2 voxel-wide whole brain analysis for all subjects (*n* = 29), angry faces *>* fixation point, thresholded at *p <* 0*.*001 FWE corrected for the entire brain, cluster size *>* 10 contiguous voxels. A similar pattern of BOLD activation is present albeit with attenuated intensity and size of activated clusters relative to Session 1.

**FIGURE 3 | Intranasal drug × session interaction in the region of the right amygdala.** Results of ROI analysis of the amygdala with One-Way ANOVA in subjects randomized to oral placebo (*n* = 14). A cluster in the right amygdala survived statistical thresholding (*p <* 0*.*05, FWE corrected for the small volume of the anatomical amygdala). Pictured are voxels surviving the statistical threshold an interaction of session (Session 1 *>* Session 2) × drug (IN placebo *>* IN vasopressin). Follow-up testing revealed that SRX246 blocked the effect of IN vasopressin in the right amygdala.

vs. fixation point within the ROIs of the right sided temporoparietal junction, precuneus, anterior cingulate gyrus, and putamen.

SRX246 compared with oral placebo was associated with significantly diminished BOLD signal intensity in the right temporoparietal junction [*p <* 0*.*005, FWE corrected for the functional ROI in the right temporoparietal junction; *t(*1*,*27*)* = 4*.*52; cluster size = 107 voxels; 50, −58, 20; **Figure 5**]. ANCOVA confirmed that the difference remained significant after controlling for Session 1 BOLD intensity [*F(*1*,* <sup>26</sup>*)* = 6*.*478, *p <* 0*.*05].

SRX246 was associated with significantly diminished BOLD activity in a cluster within the right precuneus (*p <* 0*.*05, FWE corrected for the small volume of the anatomical precuneus, cluster size = 33 voxels; MNI coordinates = 8, −56, 38; **Figure 6**). ANCOVA revealed that the difference remained significant after controlling for Session 1 BOLD intensity [*F(*1*,* <sup>26</sup>*)* = 6*.*208, *p* = 0*.*02].

**FIGURE 4 | Right Amygdala BOLD in the Four Drug Subgroups: Significant differences are seen in paired** *t***-tests between Session 1 and 2 only in the subgroup of subjects randomized to oral placebo and intranasal vasopressin (***p <* **0***.***05).**

**FIGURE 5 | SRX246 associated blunting of right TPJ to angry faces.** The cluster (circled in blue) of significantly blunted BOLD response within the right temporoparietal junction in the subjects randomized to SRX246 vs. placebo, angry faces *>* fixation point (*p <* 0*.*005, FWE corrected for the small volume of the function right TPJ, cluster size = 107 voxels; MNI coordinates = 50, −58, 20).

SRX246 was associated with reduced BOLD response in a cluster of the right anterior cingulate [*p <* 0*.*05 FWE for small volume of the right anterior cingulate, *t(*1*,*27*)* = 4*.*34, cluster size = 20 voxels; MNI coordinates = 6, 46, 22; **Figure 7**]. ANCOVA revealed that the difference remained significant after controlling for Session1 BOLD intensity [*F(*1*,* <sup>26</sup>*)* = 14*.*473, *p* = 0*.*001].

SRX246 was associated with reduced BOLD response in two clusters within the right putamen [*p <* 0*.*05, FWE corrected for small volume, *T(*1*,* <sup>27</sup>*)* = 4*.*7 and 4.55, cluster size = 18 and 17 voxels; MNI coordinates = 22, 2, 8 and 28, 12, −8, respectively; **Figure 8**]. ANCOVA revealed the difference remained significant after controlling for Session 1 BOLD intensity [*F(*1*,* <sup>26</sup>*)* = 19*.*724, *p <* 0*.*001].

Clusters of significant activation from exploratory voxel-wide whole brain analysis are presented in **Table 3**, at the uncorrected statistical threshold of *p <* 0*.*001, one-tailed. No main effects of IN vasopressin were detected in the above ROIs. See Supplementary Material 3.4∗ for exploratory analyses of other emotion conditions.

#### **BEHAVIORAL EFFECTS**

For accuracy, RM-ANOVA revealed no main effects or interactions for session, IN drug, or oral drug. For reaction time, RM-ANOVA revealed a significant effect of time [*F(*1*,* <sup>24</sup>*)* = 15*.*05, *p <* 0*.*05], with reaction time to Angry faces decreasing from Session 1 (423.41 ms, *SD* = 130*.*937) to Session 2 (342.414 ms, *SD* = 158*.*743). A trend level interaction was detected for session × oral drug [*F(*1*,* <sup>24</sup>*)* = 3*.*435, *p* = 0*.*08]. Follow up testing with paired *t*-tests comparing Session 1 to Session 2 revealed that subjects randomized to oral placebo showed a significant decrease in reaction time to angry faces from Session 1 to Session 2 [*t(*1*,*13*)* = 4*.*773, *p <* 0*.*001]; subjects randomized to SRX246 also showed a decrease but the difference was not significant [*t(*1*,*13*)* = 1*.*182, *p* = 0*.*26].

Change in amygdala BOLD response to angry faces (Session 2 – Session 1) was negatively correlated with the change in number of hits (# correct valence identifications for Session 2 – Session 1). This was true for the average of left and right amygdala together (*r* = −0*.*520, *p* = 0*.*005, *n* = 28), for the left amygdala (*r* = −0*.*462, *p* = 0*.*013) and the right amygdala (*r* = −0*.*527, *p* = 0*.*004). Thus, the decreased BOLD response from Session 1 to Session 2 seemed to predict improved performance. There was no relationship between change in amygdala BOLD response to angry faces and reaction time measures.

#### **SAFETY AND SIDE EFFECTS**

There were no serious or unexpected adverse events. SRX246 was not associated with change in AEQ subscores, BDI-II, vital sign parameters, laboratory parameters, CSSR-S score, or urine specific gravity.

### **DISCUSSION**

Chronic treatment with SRX246, a novel AVPR1a antagonist, blunts the effect of acute vasopressin administration on the functional response of the amygdala to angry faces in healthy males. An effect of intranasal vasopressin on subcortical processing of emotional facial expressions was confirmed: in healthy males, vasopressin enhanced accommodation, as reflected in a decrease from Session 1 to Session 2, of the amygdala to angry faces. This effect was effectively blocked by pretreatment with SRX246.

**angry faces.** SRX246 was associated with reduced BOLD response in a cluster of the right anterior cingulate [*p <* 0*.*05 FWE for small volume of the right anterior cingulate. *t(*1*,* <sup>27</sup>*)* = 4*.*34, cluster size = 20 voxels; MNI coordinates = 6, 46, 22].

**FIGURE 8 | SRX246 associated blunting of right putamen to angry faces.** SRX246 was associated with reduced BOLD response in two clusters within the right putamen [*p <* 0*.*05, FWE corrected for small volume, *T(*1*,* <sup>27</sup>*)* = 4*.*7 and 4.55, cluster size = 18 and 17 voxels; MNI coordinates = 22, 2, 8 and 28, 12, −8, respectively].



*Results of the exploratory whole-brain comparison of SRX246 vs. placebo. Listed are clusters of significant activation (>10 contiguous voxels) within the corresponding brain structures that survive statistical threshold (p < 0.001, uncorrected for multiple comparisons).*

Additional main effects of SRX246 were found on cortical processing of angry faces in the right side of the brain within the ROIs of the temporoparietal junction, precuneus, anterior cingulate, and putamen. These novel findings provide the first evidence for AVPR1a signaling in a neural circuit that mediates processing of social and emotional information (Saxe and Kanwisher, 2003; Zink and Meyer-Lindenberg, 2012).

The findings add to a growing literature regarding the role of vasopressinergic AVPR1a signaling in human social and emotional behavior. Genetic variation in the promoter region of AVPR1a has been associated with risk for autism, in which social deficits are the core symptom (Kim et al., 2002; Wassink et al., 2004). The same genetic polymorphisms have been linked to an altered functional response of the amygdala to fearful and angry faces in healthy adults (Meyer-Lindenberg et al., 2009).

Preclinical studies have demonstrated that vasopressin plays an important role in social recognition. We observed that IN vasopressin was associated with accommodation of the BOLD response of the amygdala to angry faces and that SRX246 pretreatment blocked this effect. These results are consistent with a role for vasopressin in social recognition in humans and provide the first evidence for the involvement of AVPR1a in this process. The direction of the effect of IN vasopressin, specifically decreased BOLD signal, differed with a previous animal model finding suggesting excitation as an expected outcome (Huber et al., 2005). Two potential explanations for this discrepancy can be put forward. One is found in rodent models, where AVPR1a signaling in the lateral septum is a necessary and sufficient condition for social recognition (Allaman-Exertier et al., 2007). Interestingly, the lateral septum tends to inhibit the amygdala, an action which likely facilitates social approach behaviors (Thomas et al., 2012). Whether the lateral septum, or its functional human equivalent, facilitates social recognition by suppression of the amygdala in humans is unknown. The second is that amygdala habituation to repeated presentation of emotional faces has been established in humans (Hariri et al., 2002). Our data indicate that vasopressin signaling through the AVPR1a plays a role in this process and that AVPR1a antagonism can modulate this effect.

Secondary analyses of the main effects of SRX246 showed that treatment with the AVPR1a antagonist reduced the response to angry faces in the right sided temporoparietal junction, precuneus, anterior cingulate, and putamen. The inhibitory effect of AVPR1a blockade on temporoparietal junction activation to angry faces is consistent with previous findings of vasopressinergic modulation of this brain region during processing of social and emotional information (Zink et al., 2011; Rilling et al., 2012; Brunnlieb et al., 2013b). The TPJ is involved in cognitive processes such as theory of mind and psychological perspective taking (Saxe and Kanwisher, 2003; Decety and Grezes, 2006) that play a fundamental role in human social interactive behavior. Hyperactivity has been noted in this region in anxious patients during negative social interactions (McClure-Tone et al., 2011) and altered function is considered a potential mechanism in disorders such as autism (Kana et al., 2012). Given that the TPJ is responsive to the emotional, social, and moral aspects (Kret et al., 2011; Koster-Hale et al., 2013) of stimuli involving social interaction, stress-related over-mentalizing during negative social interactions, mediated in part through the TPJ, may be a psychopathological mechanism amenable to treatment with AVPR1a antagonists. Our results raise the possibility that the TPJ may represent a novel treatment target in stress related disorders, although an important question in this context is whether the effects of SRX246 are mediated directly by AVPR1a in the region itself or indirectly via connected substructures such as the amygdala, lateral septum, posterior cingulate, or thalamus that, based on studies in rodents and non-human primates, are known to express vasopressin receptors (Young et al., 1999; Phelps and Young, 2003).

AVPR1a modulation of the precuneus is consistent with previous findings that IN vasopressin increased precuneus activity during a simulated aggressive social interaction (Brunnlieb et al., 2013a). Precuneus activity has been reported in fMRI studies of face processing (metaanalysis in Fusar-Poli et al., 2009), and has specifically been associated with social recognition (Lee et al., 2013). In general, the precuneus is thought to play a role in self-awareness and higher cognitive processes above and beyond sensory discrimination (reviewed in Cavanna and Trimble, 2006). The clinical potential of pharmacological modulation of the precuneus is suggested by a link to biological risk factors for anxiety and depression (Rogers et al., 2013).

That the AVPR1a antagonist treatment resulted in blunting of the anterior cingulate response to angry faces is consistent with previous research. IN vasopressin increases anterior cingulate activity during processing of social interactions (Brunnlieb et al., 2013a) and prevents supragenual cingulate deactivation during viewing of angry and fearful faces (Zink et al., 2010). The anterior cingulate is activated by viewing of emotional faces (Fusar-Poli et al., 2009). Like the precuneus, it is also implicated in social cognition (reviewed in Frith and Frith, 2003). Increased reactivity of the anterior cingulate to negatively valenced stimuli is a consistent finding in major depressive disorder (reviewed in Hamilton et al., 2013); thus modulation of the anterior cingulate via AVPR1a antagonism may well have clinical utility. The putamen expresses vasopressin receptors (Hammock and Young, 2006) and is activated by viewing of angry faces (Strauss et al., 2005). Our finding that SRX246 blunted the response in this region is in accord with these findings and indicates a prominent role for the V1a receptor subtype. A recent review of fMRI studies of face processing in major depression has found a pattern of putamen hyperreactivity to angry faces (Stuhrmann et al., 2011). Future studies should investigate the possibility that blocking AVPR1a in depressed patients reverses putamen hyperreactivity to aversive stimuli.

Given the design of the study and results from the exploratory analyses on the processing of emotions other than anger, the effects of vasopressin modulation and SRX246 may extend beyond only angry faces to the processing of aversive emotional expressions more generally. Such an interpretation would be consistent with the findings of Thompson and others (2006), who found that intranasal vasopressin increased tone of the corrugator supercilii muscle, a facial expression associated with response to threat common to anger and fear. This possibility is interesting in terms of potential treatments for stress-related disorders, but given the limitations of the current study, additional work is needed to fully characterize the effects of vasopressin and AVPR1a receptor antagonism on specific emotional responses.

Our study presents the first translational investigation of a novel, first-in-class AVPR1a antagonist in a vasopressin challenge, emotional processing paradigm. Some limitations of the study are worth mentioning regarding the interpretation of the results. To optimize reliability, the sample was made as homogenous as possible in terms of age range, gender, and psychiatric profile. To preserve statistical power for the primary comparisons, the analysis focused on the amygdala and contrasts involving angry facial expressions. For feasibility reasons, the study was not powered to detect emotion specific effects of SRX246. Secondary analyses regarding main effects of SRX246 were similarly restricted to a subset of brain regions previously identified to be affected by vasopressin signaling. The limited power of the study makes it likely that significant effects of vasopressin and SRX246 on other brain regions were not detected. Finally, interpretation of the fMRI BOLD signal as responsive to pharmacological manipulations requires the inference that the drug has a direct effect on regional brain activity. While the results, when combined with extensive *in vitro* studies that demonstrate exceptional selectivity and selectivity for the V1a receptor and *in vivo* results in preclinical models (e.g., Ferris et al., 2008; Fabio et al., 2013) strongly suggest target engagement, definitive evidence requires studies with a PET or SPECT ligand. Unfortunately, no such ligands are available.

In conclusion, the results provide the initial demonstration in humans that blockade of AVPR1a with SRX246 significantly reduces the effect of intranasally administered vasopressin on the response of the amygdala to angry face stimuli as measured by the fMRI BOLD response. Additional effects of SRX246 were observed on responses in the temporoparietal junction, putamen, precuneus, and anterior cingulate. These findings extend a growing body of evidence establishing the importance of vasopressin signaling in the processing of social and emotional stimuli. Because exaggerated responses to negatively valenced emotional stimuli in circuits that include these structures are characteristic of several stress-related psychiatric disorders, the ability of SRX246, a novel AVPR1a antagonist, to attenuate the response to angry faces supports the potential of AVPR1a antagonism as a new approach to the treatment of these indications.

## **AUTHOR CONTRIBUTIONS**

Royce J. Lee conceptualized the experiment, designed the study, devised the intranasal approach, and wrote the manuscript. Emil F. Coccaro provided advisory support, helped to design the experiment, and contributed to the manuscript. Henk Cremers played a key role in processing of fMRI data, choice of analysis parameters, and assistance with critical revisions. Rosemary McCarron designed the fMRI experiment and played a key role in fMRI data analysis. Shi-Fang Lu contributed to the design and the study. Michael J. Brownstein helped to design the experiment and contributed to the manuscript. Neal G. Simon helped to design the experiment and contributed to the manuscript.

#### **ACKNOWLEDGMENTS**

At the University of Chicago, the authors recognize the assistance of Margaret Wieczorek, Bing Chen, Eleana Fisher, and Eryka Nosal. At Azevan, the authors recognize the assistance of Debra Itzkowitz, Eve Damiano, Christophe Guillon, and Ned Heindel.

## **FINANCIAL SUPPORT**

The study was supported by NIMH awards R44 MH063663, R44 MH087001, and Azevan Pharmaceuticals, Inc.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnsys. 2013.00100/abstract

#### **REFERENCES**


and the central nucleus of the amygdala. *Neurosci. Lett.* 524, 79–83. doi: 10.1016/j.neulet.2012.07.008


**Conflict of Interest Statement:** Emil F. Coccaro is on the scientific advisory board for Azevan. Shi-Fang Lu, Michael J. Brownstein, and Neal G. Simon hold equity in Azevan Pharmaceuticals. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 26 July 2013; accepted: 14 November 2013; published online: 12 December 2013.*

*Citation: Lee RJ, Coccaro EF, Cremers H, McCarron R, Lu S-F, Brownstein MJ and Simon NG (2013) A novel V1a receptor antagonist blocks vasopressin-induced changes in the CNS response to emotional stimuli: an fMRI study. Front. Syst. Neurosci. 7:100. doi: 10.3389/fnsys.2013.00100*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2013 Lee, Coccaro, Cremers, McCarron, Lu, Brownstein and Simon. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Resting state low-frequency fluctuations in prefrontal cortex reflect degrees of harm avoidance and novelty seeking: an exploratory NIRS study

*Takashi Nakao1†\*, Tomoya Matsumoto2†, Daisuke Shimizu3, Machiko Morita3, Shinpei Yoshimura4, Georg Northoff 5, Shigeru Morinobu2,6, Yasumasa Okamoto2 and Shigeto Yamawaki <sup>2</sup>*

*<sup>1</sup> Department of Psychology, Graduate School of Education, Hiroshima University, Hiroshima, Japan*

*<sup>2</sup> Department of Psychiatry and Neurosciences, Institute of Biomedical and Health Sciences, Hiroshima University, Hiroshima, Japan*

*<sup>3</sup> Faculty of Medicine, Hiroshima University, Hiroshima, Japan*

*<sup>4</sup> Department of Psychology, Faculty of Psychology, Otemon Gakuin University, Osaka, Japan*

*<sup>5</sup> Institute of Mental Health Research, University of Ottawa, Ottawa, ON, Canada*

*<sup>6</sup> Department of Neuropsychiatry, Kochi University, Kochi, Japan*

#### *Edited by:*

*Andrew J. Greenshaw, University of Alberta, Canada*

#### *Reviewed by:*

*Takashi Yamamoto, Osaka University Graduate School of Dentistry, Japan Matthew R. G. Brown, University of Alberta, Canada*

#### *\*Correspondence:*

*Takashi Nakao, Department of Psychology, Graduate School of Education, Hiroshima University, 1-1-1, Kagamiyama, Higashi-Hiroshima, Hiroshima 739-8524, Japan e-mail: takana818@gmail.com †These authors have contributed equally to this work.*

Harm avoidance (HA) and novelty seeking (NS) are temperament dimensions defined by Temperament and Character Inventory (TCI), respectively, reflecting a heritable bias for intense response to aversive stimuli or for excitement in response to novel stimuli. High HA is regarded as a risk factor for major depressive disorder and anxiety disorder. In contrast, higher NS is linked to increased risk for substance abuse and pathological gambling disorder. A growing body of evidence suggests that patients with these disorders show abnormality in the power of slow oscillations of resting-state brain activity. It is particularly interesting that previous studies have demonstrated that resting state activities in medial prefrontal cortex (MPFC) are associated with HA or NS scores, although the relation between the power of resting state slow oscillations and these temperament dimensions remains poorly elucidated. This preliminary study investigated the biological bases of these temperament traits by particularly addressing the resting state low-frequency fluctuations in MPFC. Regional hemodynamic changes in channels covering MPFC during 5-min resting states were measured from 22 healthy participants using near-infrared spectroscopy (NIRS). These data were used for correlation analyses. Results show that the power of slow oscillations during resting state around the dorsal part of MPFC is negatively correlated with the HA score. In contrast, NS was positively correlated with the power of resting state slow oscillations around the ventral part of MPFC. These results suggest that the powers of slow oscillation at rest in dorsal or ventral MPFC, respectively, reflect the degrees of HA and NS. This exploratory study therefore uncovers novel neural bases of HA and NS. We discuss a neural mechanism underlying aversion-related and reward-related processing based on results obtained from this study.

**Keywords: low-frequency fluctuations, resting state, medial prefrontal cortex (MPFC), personality, reward, aversion, harm avoidance, novelty seeking**

#### **INTRODUCTION**

Temperament and character are the basic elements of personality that vary among individuals. In contrast to character, which is strongly influenced by experiential factors, temperament is probably more biologically based and stable across a person's life span. Harm avoidance (HA) and novelty seeking (NS) are temperament dimensions defined by the Temperament and Character Inventory (TCI), reflecting a heritable bias for responding intensely to aversive stimuli or for excitement in response to novel stimuli, respectively, (Cloninger, 1987; Cloninger et al., 1993). It is particularly interesting that extreme expression on these temperaments is associated with vulnerability to psychiatric disorders (Richter and Brandstrom, 2009). Increased levels of HA are thought to play a role as a risk factor for development of depression (Joffe et al., 1993; Richter et al., 2000; Farmer et al., 2003; Abrams et al., 2004; Smith et al., 2005; Celikel et al., 2009; Quilty et al., 2010) and anxiety disorders (Jylha and Isometsa, 2006; Mertol and Alkin, 2012). In contrast, a high level of NS is associated with increased risk of exhibiting substance abuse (Cloninger et al., 1988; Gerra et al., 2003) and pathological gambling disorder (Won Kim and Grant, 2001). Therefore, it is important to characterize the biological bases of these temperament traits widely, not only in terms of psychology but of psychiatry.

Neurally, HA and NS are known to be associated with resting state activities in various brain regions including prefrontal cortex (PFC). Positron-emission tomography (PET) reports have described that medial PFC (MPFC) glucose metabolism during resting state is negatively correlated with the HA score (Youn et al., 2002; Hakamata et al., 2006, 2009). Studies measuring cerebral blood flow (Sugiura et al., 2000; O'gorman et al., 2006) also tend to show negative correlation between HA score and activities within frontal regions including MPFC. Functional magnetic resonance imaging (fMRI) studies have demonstrated that functional connectivity between MPFC and amygdala is negatively correlated with the HA score (Li et al., 2012). In contrast, only a few studies have currently addressed the neural characteristics of NS trait from the perspective of resting-state activity. A single photon emission computed tomography (SPECT) study demonstrated that the resting state cerebral blood flow in anterior cingulate and insula are positively correlated with the NS score (Sugiura et al., 2000). Youn and colleagues reported that the NS score is positively associated with the glucose metabolic rate in the right PFC including MPFC (Youn et al., 2002). Taken together, resting state brain activity within MPFC is apparently an important neural basis underlying the temperament traits: HA and NS.

In recent years, interest in the brain's synchronous slow oscillations during a resting state has increased immensely, particularly in the field of psychiatry. Slow oscillations have been observed using measurements of different types, fMRI (Biswal et al., 1995; Fransson, 2006; Chepenik et al., 2010) and electroencephalography (Horovitz et al., 2008; Helps et al., 2010; Broyd et al., 2011; EEG). Although the mechanisms underlying the slow oscillations are not fully understood, slow oscillations of the fMRI blood oxygenation level-dependent (BOLD) signal are known to correlate with local field potentials (LFPs) in a broad frequency range (1– 100 Hz) (He et al., 2008; Scholvinck et al., 2010; Pan et al., 2011, 2013; Wang et al., 2012b). Moreover, slow oscillations reportedly modulate higher-frequency activity (Canolty and Knight, 2010; Wang et al., 2012b; Valencia et al., 2013). It is particularly interesting that the slow oscillations have been used to identify the neural characteristics of psychiatric disorders such as major depression disorder (Wang et al., 2012a; Fan et al., 2013; Liu et al., 2013), anxiety disorders (Yin et al., 2011; Hou et al., 2012; Bing et al., 2013), and substance abuse (Jiang et al., 2011). Considering that HA and NS are reported as risk factors for these disorders, it would be interesting to address the question of whether these temperament traits correlate to the slow oscillation activities at rest. However, this question remains to be answered.

This preliminary study was undertaken to characterize the neural bases of temperament dimensions (i.e., HA and NS) by particularly addressing resting state low-frequency fluctuations using near-infrared spectroscopy (NIRS). This non-invasive technique uses near-infrared light to evaluate spatiotemporal characteristics of brain functions near the brain surface. As with fMRI and EEG, NIRS enables the detection of spontaneous slow oscillations in oxygenated hemoglobin (oxy-Hb) (Obrig et al., 2000). Based on earlier studies described above, we specifically focused on the examination of MPFC resting state activity. It is noteworthy that MPFC is characterized by large amplitudes of spontaneous slow oscillations during a resting state (Raichle et al., 2001; Fransson, 2005; Zou et al., 2008). TCI (Cloninger et al., 1993) was used to assess HA and NS temperament traits. We examined whether HA or NS is related with the power of resting-state slow oscillations in the MPFC.

## **METHOD**

#### **PARTICIPANTS**

Twenty two healthy volunteer participants (12 males; age range = 21–27 years, mean age = 22.7 years) were recruited from Hiroshima University. All participants were right-handed, with normal or corrected-to-normal vision. All were free of neurological and psychiatric disorders. To control possible confounding factors of brain activity (Duncan and Northoff, 2012), participants who were habitual drinkers or taking medication were not recruited. Participants were not permitted to smoke tobacco from 3 h before the experiment started. Written informed consent was obtained from each participant before the investigation, in line with a protocol approved by the Research Ethics Committee of Hiroshima University. Each participant was paid a small fee for participating.

#### **SELF-REPORT MEASURES**

Temperament traits including HA and NS were quantified using the TCI (Cloninger et al., 1993). The TCI is a 240-item questionnaire that assumes a human personality consisting of four temperament and three character dimensions. The temperament dimensions include HA, NS, reward dependence, and persistence. The character dimensions include self-directedness, cooperativeness, and self-transcendence. In this study, the measures of HA and NS were particularly addressed.

#### **RESTING STATES**

After NIRS probe placement, participants were seated on a comfortable chair facing a computer screen in a dark shielded room. During recording, a chin rest was used to help participants maintain the head position. Participants performed counterbalanced resting eyes-closed (EC) and eyes-open (EO) baseline periods of 5 min each. Each participant was instructed to relax and allow the mind to disengage during these periods. During the EO resting state, participants were asked to gaze with fixation at a cross presented at the center of the computer screen, but were allowed to blink normally. Because the EC and EO resting states were thought to reflect baseline brain activity of different types (Marx et al., 2004; Barry et al., 2009; Yan et al., 2009), we included resting states of these two types in the present study. After each type of resting state measurement, participants were asked to fill out a questionnaire that included the question: "Did you fall asleep during the resting state scan?" No participant reported that they had fallen asleep during resting state recordings.

#### **NIRS DATA ACQUISITIONS**

Relative changes in the concentration of oxy-Hb and deoxy-Hb were measured using a multichannel NIRS imaging system (FOIRE-3000; Shimadzu Corp., Kyoto, Japan) with three wavelengths (780, 805, and 830 nm) of infrared light based on Matcher et al. (1995). The data sampling time was 115 ms. The source– detector probes were placed in fronto-temporal regions. The probe set was mounted on a cap for fixation (**Figure 1B**). The

lower frontal probes were positioned along the Fp1–Fp2 line according to the international 10–20 system used for electroencephalography. The distance between pairs of source–detector probes was set at 3 cm. Each measuring area between the pairs of source–detector probes was defined as a channel. It is inferred that the machine, with source–detector spacing of 3 cm, measures points at 2–3 cm depth from the scalp [i.e., measurements are taken from the surface of the cerebral cortex; Hock et al. (1997); Toronov et al. (2001); Okada and Delpy (2003a,b)]. Because the exact optical path length is unknown, the unit used to measure these values is the molar concentration multiplied by length (mM•cm). The 43 measuring points were labeled as ch1–ch43 (see **Figure 1A**). Of 43 channels, 15 channels in MPFC regions (ch3, ch4, ch5, ch9, ch10, ch17, ch18, ch19, ch25, ch26, ch32, ch33, ch34, ch40, ch41) were used in correlation analyses (see below) for reasons described in the Introduction. Because of a technical problem, data of three channels (ch25, ch28, and ch41) from eight participants failed to record a signal. Unless otherwise indicated, 22 participants' data were used. Three-dimensional locations of the NIRS probe were measured using a Fastrak System (TX-2; Polhemus, USA). Using the MATLAB toolbox NFRI functions (http://www*.*jichi*.*ac*.*jp/brainlab/tools*.*html), statistical results for each channel were shown for the surface of a standardized brain (Singh et al., 2005).

study, where **(B)** shows the NIRS probe position.

#### **NIRS ANALYSIS**

The NIRS data analysis was conducted using software (MATLAB 8.0; The MathWorks Inc., Natick, MA, USA). Resting state oxy-Hb data were filtered using a low-pass filter of 0.4 Hz. The linear trend caused by drift was removed (Tachtsidis et al., 2004). A Fast Fourier Transform (FFT) was performed on oxy-Hb data EC and EO resting state data. The Welch technique with a Hanning window of 1024 sample points (117.76 s sliding window) and an overlap of 512 points was used. Power spectral density (mM•cm2/Hz) was calculated for each channel over the range of 0.02–0.15 Hz. The Welch technique (Welch, 1967) involves sectioning the timeseries data into many sub-sections and converting them to a modified estimate of the spectral density before averaging the signals of the sections. Subsequently, the band-limited power in the following two frequency bands was calculated based on previous studies (Obrig et al., 2000; Tachtsidis et al., 2004; Näsi et al., 2011; Pierro et al., 2012): very low-frequency oscillations (VLFO; 0.02–0.04 Hz) and low-frequency oscillations (LFO; 0.04–0.15 Hz). The VLFO and LFO are lower frequency ranges known to be differentiated from other oscillatory phenomena such as eye blinking, heart beat, and respiratory cycles (Obrig et al., 2000; Aminoff, 2012; Pierro et al., 2012; Sassaroli et al., 2012; Li et al., 2012).

### **CORRELATION ANALYSIS**

To investigate the relations between the temperament traits and resting state activity derived from 15 channels covering MPFC, we performed separate correlation analyses for each combination among temperament traits (HA, NS), different frequency band (VLFO, LFO), and resting states of two types (EC, EO). Before calculating Pearson correlation coefficients, outliers of each datum were excluded from the correlation analysis using an upper limit of the mean ± 3 SD of the participants' data. For cases in which there were outliers for Pearson's correlation analysis, we also calculated Spearman's rank correlation coefficient, which is insensitive to outliers, using all participants' data. In both correlation analyses, Benjamini and Hochberg (BH) false discovery rate (FDR) (Benjamini and Hochberg, 1995) was applied to avoid an increase in false positives for the 15 channels. A bootstrap procedure (Efron and Tibshirani, 1986) with *n* = 1000 resamples was used to establish 95% confidence intervals (*CI*) around the *r* value.

## **RESULTS**

## **SELF-REPORT DATA**

The mean scores of HA and NS were, respectively, 51.41 (*SD* = 7*.*48, range = 35–65) and 48.73 (*SD* = 7*.*03, range = 36–63). No significant correlation was found between the HA and NS score (*r* = −0*.*37, *p* = 0*.*09, *CI* = –0.78–0.13).

#### **RESTING STATE DATA**

#### *Resting state power spectrum density*

**Table 1** presents the averaged power across all NIRS channels for each resting-state condition (EC and EO) and for each frequency band (VLFO and LFO). The mean VLFO power of the EC resting state was 0.0005 mM•cm2/Hz (*SD* <sup>=</sup> <sup>0</sup>*.*0002). That of the EO resting state was 0.0007 mM•cm2/Hz (*SD* <sup>=</sup> <sup>0</sup>*.*0006). The mean LFO power of the EC resting state was 0.00008 mM•cm2/Hz (*SD* <sup>=</sup> 0*.*00004). That of the EO resting state was 0.0001 mM•cm2/Hz (*SD* = 0*.*00006). In both frequency bands, the EO resting state showed significantly greater power than the EC resting state **Table 1 | Summary of averaged power (mM•cm2/Hz) across all NIRS channels for each resting state condition (EC and EO) and for each frequency band (VLFO and LFO).**


*M, mean; SD, standard deviation; EC, eyes-closed resting state; EO, eyesopen resting state; VLFO, very low-frequency oscillation; LFO, low-frequency oscillation.*

did [VLFO, *t(*21*)* = 2*.*15, *p* = 0*.*04; LFO, *t(*21*)* = 2*.*98, *p* = 0*.*007]. These results resemble those reported from earlier studies (Obrig et al., 2000; Tachtsidis et al., 2004; Yan et al., 2009).

These resting state data reported in this manuscript have been published previously (Nakao et al., 2013) and were included as part of a larger data collection. Nakao et al. (2013) reported the results for relations among the power of resting state slow oscillations, early life stress, and frontal activation during decision making tasks. The present manuscript describes a specific examination of the relations between the power of resting state slow oscillations and temperament traits (HA or NS).

#### *Correlation between resting state and temperament scores*

**Figure 2A** presents some correlation results between the powers of VLFO during EC resting state and the HA score. The power of VLFO at right dorsal MPFC (ch9, Brodmann area: BA9) was negatively correlated with the HA score (Pearson *r* = −0*.*61, *FDR* adjusted *p <* 0*.*05, *CI* = –0.80 to –0.34, *N* = 21; Spearman *r*s = – 0.71, *FDR* adjusted *p <* 0*.*01 *CI* = –0.88 to –0.41, *N* = 22). The power of VLFO or LFO during EC or EO resting state in other channels showed no significant correlation with the HA score.

**Figure 2B** presents the correlation results between the powers of LFO during EO resting state and NS score. The powers of LFO at bilateral ventral MPFC (ch40, BA10; ch41 BA10) were positively correlated with the NS score (ch40, Pearson *r* = 0*.*64, *FDR* adjusted *p <* 0*.*05, *CI* = 0.33–0.85, *N* = 21; Spearman *r*s = 0.58, *FDR* adjusted *p <* 0*.*07 *CI* = 0.19–0.79, *N* = 22; ch41, Pearson *r* = 0*.*62, *FDR* adjusted *p <* 0*.*05, *CI* = 0.21–0.90, *N* = 14). The power of VLFO or LFO during EC or EO resting state in other channels showed no significant correlation with NS score.

#### **DISCUSSION**

This study was undertaken to investigate the relations between the power of slow oscillation during resting state and HA or NS. As **Figure 2** shows, slow oscillations during resting state at the dorsal MPFC were negatively correlated with the HA score. In contrast, NS was correlated positively with resting-state slow oscillations around the ventral MPFC. These results provide new insights into the neural bases of HA or NS by particularly addressing low-frequency fluctuations.

Previous reports have described that HA is associated with decreased resting state cerebral blood flow (Sugiura et al., 2000; O'gorman et al., 2006) within frontal regions including dorsal MPFC. Although our index of resting state brain activity (i.e., the power of NIRS oxy-Hb slow oscillations) differed from those earlier studies, our results were consistent with those in that HA was found to be associated with the attenuated resting state activity in the dorsal regions of MPFC (**Figure 2A**). In contrast, our results showed that NS is associated with amplified resting state activation within ventral regions of the MPFC (**Figure 2B**). These results are consistent with those of previous studies which reported that the NS was associated with increased resting state glucose metabolism in the prefrontal regions including ventral MPFC (Youn et al., 2002). Consequently, these exploratory data provide new evidence that the neural bases of HA or NS can be assessed by low-frequency fluctuations during a resting state measured by NIRS, in addition to other indexes such as the glucose metabolism and cerebral blood flow. It would be interesting to investigate the relations among NIRS low frequency fluctuations and other measurements of brain activity (e.g., the glucose metabolism and cerebral blood flow) in terms of neural bases of temperament traits.

Considering our finding about the relation between HA and the power of resting state slow oscillation, resting state activity in dorsal MPFC might be related to aversion-related processing. Indeed, dorsal MPFC is known as a part of neural network activated by aversive stimuli (Hayes and Northoff, 2011, 2012). The dorsal MPFC is reported to serve an important role in sustaining fear response (Vidal-Gonzalez et al., 2006; Laurent and Westbrook, 2009; Furlong et al., 2010; Robinson et al., 2012). Laurent and Westbrook (2009) demonstrated that inactivation of the rat's paralimbic neurons, which are thought to have similar function with human dorsal MPFC in fear conditioning (Milad et al., 2007, 2009; Robinson et al., 2012), prevents fear response to conditioned aversive stimulus. In addition, Vidal-Gonzalez et al. (2006) demonstrated that microstimulation of that region increased fear response. Robinson et al. (2012) conducted a human fMRI study that showed that the functional connectivity between dorsal MPFC and amygdala was increased during the processing of fearful faces under anxious conditions, and that the amount of coupling was stronger in participants with higher trait anxiety. Based on this evidence, people with high HA personality are expected to show sustained fear response and greater activity in dorsal MPFC under aversive conditions. It would be interesting to examine whether and how the attenuated resting state activity in dorsal MPFC relates to the enhanced aversive-stimulus-induced activity in the same region in high HA people.

Ventral PFC, resting state activity of which correlated positively with NS, is known as a part of the reward-related network (Liu et al., 2011). The activity of ventral PFC is thought to represent the expected value of the outcome which guides reward-based decision making (Hampton and O'Doherty, 2007; O'doherty, 2007; Nakao et al., 2012). Bermpohl et al. (2008) revealed that people with high NS showed enhanced ventral MPFC activity during the expectancy of emotional stimuli. In the relation with resting state brain activity, Li et al. (2013) reported that the resting state functional connectivity in the reward-related network including ventral MPFC was associated with high impulsivity in decision making (i.e., higher preference

for an immediate small reward than a larger delayed reward). It is possible that enhanced activity of ventral MPFC at rest observed in people with higher NS scores influences the intensity of the response to rewarding stimuli. Future studies must be undertaken to elucidate how resting state activity in ventral MPFC influences reward-based decision making.

Although we used TCI, which was developed to assess the seven dimensions of the psychobiological model of personality, another line of personality model exists: the five factor model (FFM; Costa and Maccrae, 1992). Neuroticism and extroversion are dimensions of the FFM. These are known to correlate, respectively, with HA and NS (Zuckerman and Cloninger, 1996; De Fruyt et al., 2000; Sher et al., 2000). Like HA, neuroticism is known to be associated with depression and anxiety disorders (Boyce et al., 1991; Rosellini and Brown, 2011). Similarly to NS, a higher extroversion score is associated with alcohol abuse (Flory et al., 2002; Merenäkk et al., 2003). Kunisato et al. (2011) and Wei et al. (2012) examined the relation between resting-state slow oscillation and neuroticism or extroversion using fMRI. They reported that extroversion correlated positively with the amplitude of slow oscillation in the prefrontal regions including ventral MPFC, which are similar to our results for NS. However, they reported no significant correlation between neuroticism and prefrontal regions, which is inconsistent with our results for HA. De Fruyt et al. (2000) reported that 23–51% of the variance of the TCI scales is explainable using the FFM, and concluded that although a substantial overlap exists between the TCI and the FFM, these two cannot be regarded as an equivalent tool to assess individual differences of personality. It would be interesting to examine the differences and similarities between the two personality models in terms of resting state brain activity.

Despite the importance of our data for revealing the neural bases of temperament traits, these findings leave several questions unresolved. First, although NIRS is expected to be useful to assess the bases of HA traits, it was impossible to address the question of how changes of the frontal power of slow oscillation in relation with HA traits are associated with the resting-state activity in the amygdala, where functional connectivity to the MPFC regions was reported previously to correlate to HA (Li et al., 2012; Wang et al., 2013). Additional fMRI studies are expected to be useful to provide further integrative understanding about the neural basis of temperament traits. Second, our data demonstrate that the HA correlated strongly with VLFO power during the EC resting state (**Figure 2A**), whereas the NS score correlated strongly with LFO power during the EO resting state (**Figure 2B**). However, although several studies addressed the differences in the frequencies of slow oscillation (Schroeter et al., 2004; Harrison et al., 2008) and the resting state eye conditions (Yang et al., 2007; Qin et al., 2013; Tan et al., 2013), the characteristics in brain function related to these frequencies/conditions remain poorly understood. Further studies investigating the characteristics of VLFO and LFO, and those of EC and EO resting states in the brain function are expected to contribute to the elucidation of the neural bases of temperament traits. Third, we did not record physiological data of eye blink, heat rate, or respiratory cycles because the ranges of slow oscillation can be differentiated from these artifacts (Obrig et al., 2000; Aminoff, 2012; Li et al., 2012; Pierro et al., 2012; Sassaroli et al., 2012). However, recording these artifact data and careful assessment of the pollution on cortical activity data are preferred for future study.

#### **CONCLUSION**

This study was undertaken to investigate the relations between temperament dimensions (i.e., HA and NS) and the power of slow oscillation in a resting state. We demonstrated a unique relation between them in that HA and NS are oppositely associated, respectively, with the power of slow oscillations in different subregions in the MPFC. These results suggest that the degrees of HA and NS might be predicted by the power of lowfrequency fluctuations at rest. Further research on this matter must be conducted using data of more participants. Considering that both slow oscillation activity and temperament traits are involved in the pathophysiology of various psychiatric disorders, the results of this study are expected to be of great interest in the field not only of personality research but also that of psychiatric research. It would therefore be interesting to extend this study to the assessment of patients with such disorders. Beyond elucidating the neural bases of the temperament traits, this line of investigation is expected to contribute to improvement of our understanding of resting-state brain activity.

#### **ACKNOWLEDGMENTS**

We thank N. Kanayama for his useful comments. This work was supported by the following grants: Hiroshima University Grants-in-Aid for a scientific research project to overcome stressvulnerability; the Strategic Research Program for Brain Sciences by the Ministry of Education, Culture, Sports, Science and Technology of Japan; JSPS KAKENHI Grant Numbers 24390284 and 25870467.

#### **REFERENCES**


an exploratory NIRS study during resting state and self-oriented task. *Front. Hum. Neurosci.* 7:339. doi: 10.3389/fnhum.2013.00339


by resting-state functional MRI. *Neuroimage* 36, 144–152. doi: 10.1016/j.neuro image.2007.01.054


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 September 2013; accepted: 02 December 2013; published online: December 2013. 17*

*Citation: Nakao T, Matsumoto T, Shimizu D, Morita M, Yoshimura S, Northoff G, Morinobu S, Okamoto Y and Yamawaki S (2013) Resting state low-frequency fluctuations in prefrontal cortex reflect degrees of harm avoidance and novelty seeking: an exploratory NIRS study. Front. Syst. Neurosci. 7:115. doi: 10.3389/fnsys.2013.00115 This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2013 Nakao, Matsumoto, Shimizu, Morita, Yoshimura, Northoff, Morinobu, Okamoto and Yamawaki. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## fMRI investigation of response inhibition, emotion, impulsivity, and clinical high-risk behavior in adolescents

Matthew R. G. Brown1, 2 \*, James R. A. Benoit <sup>1</sup> , Michal Juhás <sup>1</sup> , Ericson Dametto<sup>1</sup> , Tiffanie T. Tse<sup>1</sup> , Marnie MacKay <sup>1</sup> , Bhaskar Sen<sup>2</sup> , Alan M. Carroll <sup>1</sup> , Oleksandr Hodlevskyy <sup>1</sup> , Peter H. Silverstone<sup>1</sup> , Florin Dolcos 1, 3, Serdar M. Dursun<sup>1</sup> and Andrew J. Greenshaw<sup>1</sup>

<sup>1</sup> Department of Psychiatry, University of Alberta, Edmonton, AB, Canada, <sup>2</sup> Department of Computing Science, University of Alberta, Edmonton, AB, Canada, <sup>3</sup> Psychology Department, Neuroscience Program, and the Beckman Institute for Advanced Science and Technology, University of Illinois Urbana-Champaign, Urbana, IL, USA

#### Edited by:

Mikhail Lebedev, Duke University, USA

#### Reviewed by:

Giovanni Mirabella, Sapienza University of Rome, Italy Guillaume Sescousse, Donders Institute for Brain, Cognition and Behaviour, Netherlands

#### \*Correspondence:

Matthew R. G. Brown, Department of Psychiatry, University of Alberta, 12-127A Clinical Sciences Building, Edmonton, AB T6G2B3, Canada mbrown2@ualberta.ca

> Received: 27 November 2014 Accepted: 20 August 2015 Published: 29 September 2015

#### Citation:

Brown MRG, Benoit JRA, Juhás M, Dametto E, Tse TT, MacKay M, Sen B, Carroll AM, Hodlevskyy O, Silverstone PH, Dolcos F, Dursun SM and Greenshaw AJ (2015) fMRI investigation of response inhibition, emotion, impulsivity, and clinical high-risk behavior in adolescents. Front. Syst. Neurosci. 9:124. doi: 10.3389/fnsys.2015.00124 High-risk behavior in adolescents is associated with injury, mental health problems, and poor outcomes in later life. Improved understanding of the neurobiology of high-risk behavior and impulsivity shows promise for informing clinical treatment and prevention as well as policy to better address high-risk behavior. We recruited 21 adolescents (age 14–17) with a wide range of high-risk behavior tendencies, including medically high-risk participants recruited from psychiatric clinics. Risk tendencies were assessed using the Adolescent Risk Behavior Screen (ARBS). ARBS risk scores correlated highly (0.78) with impulsivity scores from the Barratt Impulsivity scale (BIS). Participants underwent 4.7 Tesla functional magnetic resonance imaging (fMRI) while performing an emotional Go/NoGo task. This task presented an aversive or neutral distractor image simultaneously with each Go or NoGo stimulus. Risk behavior and impulsivity tendencies exhibited similar but not identical associations with fMRI activation patterns in prefrontal brain regions. We interpret these results as reflecting differences in response inhibition, emotional stimulus processing, and emotion regulation in relation to participant risk behavior tendencies and impulsivity levels. The results are consistent with high impulsivity playing an important role in determining high risk tendencies in this sample containing clinically high-risk adolescents.

Keywords: response inhibition, high-risk behavior, impulsivity, emotional Go/NoGo, adolescent, ARBS

## 1. Introduction

Adolescence is a period of increasing high-risk behavior for many individuals. Examples of highrisk behavior include alcohol binging, substance abuse, unsafe sex, physical violence, physical risk-taking, and self-harm. At a population level, high-risk behavior tendencies start to increase between ages 13–17 (Statistics Canada, 2010a,b; Viner et al., 2012). Many risk behaviors such as binge drinking and impaired driving peak between ages 20–24 and then decline (Statistics Canada, 2010a,b). Although many important risk behaviors peak later in life, engagement in these behaviors begins to increase in adolescence, creating attendant risks for poor outcomes. For example, deaths due to suicide and accidental injury peak in the 50–54 and 90+ age ranges, respectively (Statistics Canada, 2010a,b). Nonetheless, suicide and accidental injury consistently remain the two leading causes of death in persons aged 15–21 in Canada, with rates higher for males than females (Statistics Canada, 2010b). In 2004, suicide was the third leading cause of death among persons aged 10–24 years in the United States (NCHS, 2007). High-risk behaviors in adolescence and young adulthood are also associated with poor physical and mental health outcomes in later life (Anda et al., 2006; Eaton et al., 2006; Hawton and O'Connor, 2012). Moran et al. (2012) have reviewed self-harm in adolescence and point to the need to understand the underlying factors for this vulnerable age group. Such understanding is also important given that self-harm is an indicator of future mental health problems (see also Hawton and O'Connor, 2012). Apart from our shared concern in terms of human suffering, the economic implications of an increased health system burden related to neurodevelopmental challenges are staggering (Stephens and Joubert, 2001). Research that can inform us about mechanisms of brain function that underlie high-risk behavior in adolescents and young adults is very important and has implications for adolescent and subsequent mental health, health services delivery, and health policy.

The population-level risk patterns described are comprised of subgroups with different risk behavior patterns, which likely have different etiologies (Romer, 2010). First, it is important to note that many adolescents do not engage in high-risk behavior to any great degree. In a study of adolescent binge drinking, Hill et al. (2000) reported that the majority (70%) of adolescents age 13–18 did not binge drink. 23% began binge drinking around age 16 and increased the frequency of binge drinking into young adulthood. A small minority (4%) began binge drinking earlier, around age 14–15, and exhibited increasing and substantially higher binge drinking frequencies into young adulthood. Another small minority (3%) exhibited a very early onset of binge drinking before age 13 combined with a peak at age 14–15 followed by declining binge drinking into young adulthood. One of this study's conclusions is that a small minority of individuals account for a substantial proportion of adolescent binge drinking. A study of aggressive behavior in children and adolescents (Nagin and Tremblay, 1999) supports a similar conclusion with regard to physical aggression.

The interaction of emotion and executive control of behavior is a crucial focal point for understanding the neural basis of decision making in high-risk situations such as those involving drug use or self-harm. Explanations of high risk behavior tendencies have emphasized individual differences and developmental changes in emotion processes, reward processing, and executive control of behavior and emotions (Jessor, 1991; Arnett, 1992, 1994, 1996; Ernst et al., 2006; Steinberg, 2007; Casey et al., 2008; Ernst and Mueller, 2008; Gullo and Dawe, 2008; Steinberg, 2008; Ernst and Fudge, 2009; Romer et al., 2009; Romer, 2010; Casey et al., 2011; Dalley et al., 2011; Mitchell, 2011; Blakemore and Robbins, 2012; Whelan et al., 2012).

We used an emotional Go/NoGo task with functional magnetic resonance imaging (fMRI) to investigate processes related to response inhibition and emotion processing in a sample of 21 adolescents (age 14–17 years) with a range of risk behavior tendencies. Risk behavior was assessed using the Adolescent Risk Behavior Screen (ARBS; Jankowski et al., 2007). Our sample included participants recruited from pediatric psychiatry clinics with high ARBS risk scores. We used a variant of the classic Go/NoGo task (Donders, 1868/1969). Our task presented emotionally neutral or aversive distractor pictures simultaneously with Go/NoGo stimuli. Participants had to ignore the distractors, respond to Go stimuli by pressing a button, and respond to NoGo stimuli by withholding the button press. Go trials were 4 times as frequent as NoGo trials to increase prepotency of Go trials. See Section 2.3 for task details.

Our group has previously reported fMRI results using this emotional Go/NoGo task with young adult participants (Brown et al., 2012). Brown et al. (2012) found differences in fMRI activation related to response-inhibition and emotion processing in multiple regions in all lobes of the brain. Activation in left motor cortex and other regions was significantly larger for Go vs. NoGo trials. In an analysis of response inhibition (NoGo vs. Go), ventrolateral prefrontal (vlPFC) cortex as well as other cortical regions exhibited larger activation for NoGo compared to Go trials. These findings are consistent with previous Go/NoGo studies (Garavan et al., 1999, 2002; Watanabe et al., 2002; Mostofsky et al., 2003; Aron et al., 2004b; Fassbender et al., 2004; Kelly et al., 2004; Rubia et al., 2005; Wager et al., 2005; Aron et al., 2007b; Mitchell, 2011). In an analysis of emotional valence (aversive vs. neutral distractors), greater activation for aversive distractor pictures was displayed in orbitofrontal cortex, lateral prefrontal cortex, insula, the amygdala and surrounding cortex, anterior cingulate cortex, medial prefrontal cortex, and bilateral posterior middle temporal gyrus and angular gyrus. These results are also consistent with previous fMRI studies of emotional picture processing (Irwin et al., 1996; Bermpohl et al., 2006; Meseguer et al., 2007). Brown et al. (2012) investigated interactions of response inhibition and emotion processing in vlPFC by emotional context. These two sources of fMRI activation changes summated in a straight-forward manner; emotional context (aversive vs. neutral distractors) did not suppress or potentiate fMRI signals related to response inhibition in vlPFC. See Brown et al. (2012) for further details.

More broadly, the results of Brown et al. (2012) are consistent with the literature on response inhibition, cognitive control, emotion processing, and emotion regulation. Dorsolateral prefrontal cortex (dlPFC), vlPFC, orbitofrontal cortex (OFC), and ventromedial PFC (vmPFC) are involved in response inhibition in the Go/NoGo task as well as inhibition in other executive control tasks (see Aron et al., 2004a, 2007b; Chikazoe, 2010; Dolcos et al., 2011; Mitchell, 2011; Mahmood et al., 2013). Anterior cingulate cortex (ACC) has been implicated in error detection and conflict monitoring in the Go/NoGo and other cognitive tasks (Carter et al., 1998, 1999; Garavan et al., 1999; Botvinick et al., 2004; Kerns et al., 2004; Brown and Braver, 2005; Mitchell, 2011). Dorsomedial PFC (dmPFC) may also contribute to response conflict processing (see Ridderinkhof et al., 2004) as well as to response selection and response inhibition in the Go/NoGo task (Simmonds et al., 2008). dmPFC is also thought to be involved in resolution of response conflict and outcome value-related aspects of decision making (Venkatraman et al., 2009). There are suggestions that primary motor and premotor cortex may be involved in response inhibition, in addition to generation of motor responses (see Coxon et al., 2006; Mirabella et al., 2011; Mattia et al., 2012, 2013). OFC and vlPFC are thought to be involved in processing emotional stimuli, for example to evaluate valence (Dolcos et al., 2011; Mitchell, 2011). Multiple prefrontal regions including OFC, vmPFC, dmPFC, vlPFC, and dlPFC are also associated with emotion regulation (Dolcos et al., 2011; Mitchell, 2011; Golkar et al., 2012).

We are aware of one other fMRI study using an emotional Go/NoGo task with adolescent participants (Hare et al., 2008). In this study, the emotional valence of the stimulus served as the Go/NoGo signal, necessitating a blocked aspect in the trial design to accommodate different stimulus valence to Go/NoGo assignments. For example, Hare et al. (2008) presented specific valence to Go/NoGo assignments in different functional runs. This approach introduces potential extraneous task set effects neutral Go and aversive NoGo trials occur in the context of one assignment; aversive Go and neutral NoGo trials occur in the context of a different assignment. In contrast, we presented emotional distractors with non-emotional Go/NoGo stimuli, such that task performance did not require use of any information from the distractors. This design choice allowed us to interleave all trial types in the same task set context, resulting in cleaner methodological and logical separation of differences related to NoGo vs. Go effects and aversive vs. neutral distractor effects.

We characterized participant impulsivity using the Barratt Impulsivity Scale (BIS; Barratt, 1959) and compared fMRI results against both ARBS risk scores and Barratt impulsivity scores. Individual differences in impulsivity and impulse control are thought to play a role in high-risk behavior. Higher impulsivity scores on psychometric questionnaires have been associated with increased risk behavior tendencies (Levitt, 1991; Moore and Rosenthal, 1993; Luengo et al., 1994; Stanford et al., 1996; Gullo and Dawe, 2008; Romer et al., 2009; Romer, 2010; Dalley et al., 2011; Mishra and Lalumière, 2011; Christiansen et al., 2012). The relationship between impulsivity and high-risk behavior is complex (Romer, 2010; Dalley et al., 2011; Blakemore and Robbins, 2012), and some aspects of high-risk behavior do not seem to be associated particularly strongly with impulsivity (Steinberg et al., 2008; Romer, 2010; Brown et al., 2015). However, certain aspects of dangerous high-risk behavior do seem to be associated with elevated impulsivity (see Romer, 2010). The inclusion of BIS scores allowed us to investigate potential relationships between high-risk tendencies and impulsivity.

One study reported increased activation on NoGo trials in left dmPFC, right ACC, right dlPFC, and left precentral gyrus in adolescent participants with internet gaming addiction compared to controls (Ding et al., 2014). Impulsivity scores from the BIS were also associated with internet gaming addiction and left dmPFC activation in that study. Goldenberg et al. (2013) found that risky sexual tendencies in adolescent participants were inversely related to response inhibition fMRI activation in a Go/NoGo task evoked in multiple brain regions in left and right dlPFC, vlPFC, and insula. In adults, impulsivity measures have been associated with differences in fMRI activation patterns in the Go/NoGo task. Greater BIS scores were associated with less response inhibition-related fMRI activation in right dlPFC (Asahi et al., 2004) and in dmPFC (Horn et al., 2003). Horn et al. (2003) also reported a positive correlation between Eysenck's Impulsivity Scale and response inhibitionrelated activation in right vlPFC. The literature suggests, then, that many prefrontal regions may be involved in individual differences in impulsivity levels and risk tendencies, although different papers implicate somewhat different sets of specific regions. The precise relationships between brain functions related to behavioral control, emotion representation, and emotion regulation in various prefrontal regions, fMRI activation patterns, and individual impulsivity and high-risk behavior tendencies constitute an open area of research.

## 1.1. Hypotheses

Given the discussion above and the well-known involvement of prefrontal cortex (PFC) in executive control, emotion processing, and emotion regulation (Rubia et al., 2001; Aron et al., 2004a, 2007b; Chikazoe, 2010; Dolcos et al., 2011; Mitchell, 2011; Mahmood et al., 2013), we expected that one or more prefrontal regions would show a relationship between participant ARBS risk scores and/or BIS impulsivity scores and task-related fMRI activation patterns, as specified in the following hypotheses:


Due to the large number of different prefrontal regions previously reported to show differences in response inhibition based on participant impulsivity or risk tendencies (see discussion above and Horn et al., 2003; Asahi et al., 2004; Goldenberg et al., 2013; Ding et al., 2014), we did not constrain Hypotheses 1a–1c to specific prefrontal brain regions. That is, these hypotheses were specific in predicting associations between task-related prefrontal fMRI changes with risk and/or impulsivity scores but exploratory in terms of which prefrontal brain regions would exhibit modulation by risk and/or impulsivity scores. Despite the broad prefrontal focus, vlPFC, especially on the right side, was of particular interest, given the well-replicated finding that fMRI activation in vlPFC is associated with response inhibition (see Chikazoe, 2010). We also did exploratory analyses of potential relationships between task-related fMRI activation patterns and risk and/or impulsivity scores and brain regions outside prefrontal cortex.

We sought to address the specific question of how high impulsivity is related to high risk behavior. Despite previous associations between impulsivity and high risk behavior

<sup>1</sup>The interaction of response inhibition and emotion processing refers specifically to the interaction contrast (aversive NoGo - aversive Go) - (neutral NoGo - neutral Go). See Section 2.6 for details.

tendencies (Levitt, 1991; Moore and Rosenthal, 1993; Luengo et al., 1994; Stanford et al., 1996; Gullo and Dawe, 2008; Romer et al., 2009; Romer, 2010; Dalley et al., 2011; Mishra and Lalumière, 2011; Christiansen et al., 2012), recent results from our group suggest a possible dissociation between impulsivity and risk behavior (Brown et al., 2015). Therefore, we did not have an a priori prediction about whether risk and impulsivity scores would have similar or distinct associations with differences in prefrontal fMRI activations. We anticipated that either outcome would contribute to this open question. We formulated this question as two opposing hypotheses:


We also tested predictions related to two models of highrisk behavior in adolescents. The Triadic Model of Ernst and colleagues (Ernst et al., 2006; Ernst and Mueller, 2008; Ernst and Fudge, 2009) suggests that elevated adolescent risk tendencies (relative to younger or older age groups) are caused in part by developmental changes in emotion-related processing in limbic regions. Specifically, the Triadic Model posits increased approach signals, including signals driven by reward or other emotionally positive stimuli, in the nucleus accumbens and other limbic regions in adolescents. It also posits decreased harm-avoidance signal, including signals driven by emotionally aversive stimuli, in the amygdala and other limbic regions in adolescents. In addition, the Triadic Model attributes elevated adolescent risk behavior partially to incomplete development of prefrontal regulatory functions. Though this model does not address individual differences in risk behavior among adolescents (Nagin and Tremblay, 1999; Hill et al., 2000; Berns et al., 2009; Romer, 2010), we suggest that this model would be more consistent with reduced limbic fMRI responses to aversive distractors in high risk participants (Hypothesis 3a below) as well as reduced activation related to response inhibition in prefrontal regions in high risk participants (Hypothesis 4 below). The model of Casey et al. (2008, 2011) attributes elevated adolescent risk behavior in part to elevated emotional responses in limbic regions, including the amygdala, as well as to incompletely developed prefrontal regulatory function. We propose that this model would be more consistent with increased limbic responses to aversive distractors in high risk participants (Hypothesis 3b below, opposite to the Ernst Triadic Model's prediction) as well as reduced response inhibition activation in prefrontal regions in high risk participants (Hypothesis 4 below, identical to the Ernst Triadic Model). We investigated fMRI responses to the distractor images in limbic regions including the amygdala, as well as possible modulation based on participant risk and impulsivity levels. We did not have a prediction about which of the two models would be better supported. This question is formulated as two opposing hypotheses:


We also investigated differences in prefrontal response inhibition activation related to risk tendencies. Our prediction based on the discussion above was that higher risk participants would exhibit reduced activation related to response inhibition in one or more prefrontal regions, with a particular focus on vlPFC:

• **Hypothesis 4:** Prefrontal regions including vlPFC will exhibit reduced response inhibition activation in participants with higher risk scores, consistent with the Ernst Triadic Model and the Casey Model.

## 2. Methods

The Health Research Ethics Board at the University of Alberta approved this study.

Due to substantial similarities between the methods described here and those of Brown et al. (2015), from our research group, parts of the methods descriptions below were adapted from that paper.

## 2.1. Participants

Twenty-one adolescents were recruited into the study (13 female and 8 male, age range 14.8–17.7 years, mean age 16.0 ± 1.1 years). Based on the Edinburgh Handedness Inventory (Oldfield, 1971), 19 participants were right-handed, and two participants were ambidextrous. All participants gave informed, written assent in English, and a parent or guardian gave informed, written consent. Participants did not receive any financial incentive to participate in the study. They were given a Tim Horton's gift card worth \$10 at the end of the study as a thank-you. The gift card did not provide any incentive toward study participation or task performance as participants were not told they would receive it at any time before receiving it.

Fourteen participants were recruited from the general population. These participants reported no history of psychiatric disorder, neurological disorder, or learning disability, and they were not taking any psychoactive medication. Three of these fourteen participants had ARBS risk scores in the high-risk range (≥17, see Questionnaires Section 2.2 below). The other 11 participants had ARBS scores in the low-risk range (≤13, see Questionnaires Section 2.2 below).

To enable investigation of clinical high-risk behavior, we also recruited seven participants with high-risk behavior tendencies from pediatric psychiatry clinics in the Edmonton area. These participants did not report any history of neurological disorder. One reported no history of psychiatric diagnosis and was not on psychoactive medication. The other six participants reported

#### TABLE 1 | Psychiatric symptoms.


Psychiatric symptoms reported by the seven participants recruited from pediatric psychiatric clinics.

some history of psychiatric symptoms (see **Table 1**). All seven of these participants had ARBS scores ≥17.

## 2.2. Questionnaires

Participant risk behavior tendencies were characterized with the Adolescent Risk Behavior Screen (ARBS; Jankowski et al., 2007). This provides a score from 9 (lowest risk) to 30 (highest risk). Jankowski et al. (2007) suggest a cutoff of >17 for defining highrisk status. We used an ARBS cutoff of ≥17 as our participants exhibited a natural break with ARBS scores being either ≤13 or ≥17.

To assess participants' impulsivity, we used the Barratt Impulsivity Scale, version 11 (BIS; Patton et al., 1995). The BIS includes six first order subscales: attentional, cognitive instability, motor, perseverance, self-control, and cognitive complexity. We took the sum over all 30 questions in the BIS (after reversing scores for appropriate items) as a participant's impulsivity score. This is equivalent to taking the sum of the first order subscale scores. Overall BIS scores can range from 30 (least impulsive) to 120 (most impulsive). BIS scores from 52 to 71 represent a normal range of impulsivity, with scores at or below 51 indicating a very controlled, non-impulsive individual and scores at or above 72 representing a highly-impulsive individual (Stanford et al., 2009).

### 2.3. Task

We employed an emotional Go/NoGo task (see Donders, 1868/1969; Hester and Garavan, 2004), which presented emotional distractor pictures simultaneously with the Go and NoGo stimuli. In each trial, the participant was shown a square or circle, lasting 2 s, which served as the Go or NoGo stimulus (see **Figure 1**). The assignment of shape to trial type was counterbalanced across participants. Each Go or NoGo stimulus was superimposed on a task-irrelevant distractor image. Each distractor image was either emotionally neutral or aversive. Distractor images were taken from the International Affective Pictures System (IAPS) (Lang et al., 2008). On Go trials, the participant had to press a button with their right index finger. On NoGo trials, the participant had to withold the button press response. To make the Go response more automatic (prepotent), Go and NoGo trials were presented at a 4:1 ratio. The task included four trial types: neutral Go, neutral NoGo, aversive Go, and aversive NoGo. Between trials, participants fixated a dot located at screen centre (**Figure 1B**). Participants were asked to perform each trial quickly and accurately.

IAPS images were chosen as follows. IAPS images were screened by two child psychiatrists to be acceptable for use with adolescent psychiatric participants. From the screened images, aversive and neutral distractor pictures were selected based on the IAPS measures of valence and arousal from the normative sample reported in Lang et al. (2008). To maximize the effect of distractor valence, we used image selection criteria that created two non-overlapping clusters of images in two-dimensional arousal-valence space, one cluster for aversive distractors and one for neutral distractors (see Supplementary Figure 1). Specifically, we selected the 100 aversive IAPS images that had valence ratings ≤3.6 and were closest to [arousal, valence] target position [9, 1]. Position [9, 1] represents the most aversive (lowest valence rating), most arousing possible score. We selected the 104 neutral images with valence ratings >3.6 and <6.4 that were closest to [arousal, valence] target position [1, 5], which represents a neutral valence and the smallest possible arousal score. It would have been preferable to match distractor images for scene complexity, number of objects, and so on across the different trial types. Unfortunately, the IAPS set did not include enough images to permit such matching while also satisfying the above-described criteria: screening by psychiatrists and separation into two non-overlapping clusters (as can be seen in Supplementary Figure 1). Aversive distractor pictures presented a variety of scenes including threatening animals, aggressive human faces, individuals wielding guns in a threatening manner, human injuries, surgical scenes, vehicle accidents, terrorismrelated scenes, individuals vomiting, and dirty toilets including feces.

Trials were presented in a rapid event-related design. Each Go or NoGo trial lasted one volume, i.e. 2 s. Inter-trial intervals were pseudo-randomized from the set {2, 4, 6 s}, distributed 30% 2 s, 40% 4 s, 30% 6 s with a mean of 4 s. Trial sequences and timings were derived using custom Python code to ensure linear independence of trial activations (see Burock et al., 1998). First-order counterbalancing of trial sequences was used to avoid first-order interaction effects between adjacent trials. To avoid interaction of BOLD non-linearity with inter-trial intervals and trial types, each of the four trial types was preceded in equal proportions by the 2, 4, and 6 s inter-trial intervals. Participants each completed four 330 s functional runs with a combined total of 204 trials including 84 neutral Go trials, 80 aversive Go trials, 20 neutral NoGo trials, and 20 aversive NoGo trials. The first trial of every run was always a neutral Go trial.

#### 2.4. MRI Scanning

Magnetic resonance imaging was done on the 4.7 Tesla Varian Inova scanner at the Peter S. Allen MR Research Centre at the University of Alberta. We acquired blood oxygenation level dependent (BOLD) fMRI images with a T2\*-weighted echo planar imaging sequence using these parameters: volume time 2.0 s, single shot, repeat time 2.0 s, echo time 19.0 ms, 3.0 mm isotropic voxels, 80 <sup>×</sup> 80 matrix, 240 <sup>×</sup> 240 mm<sup>2</sup> field of view, 3.0 mm slice thickness, 36 axial slices, 108 mm

through-plane coverage, interleaved slice collection order. We used 80% partial k-space in the phase encode direction (anteriorposterior). The fMRI scanning volume covered the entire cerebral cortex except for the ventral-posterior tip of occipital cortex in participants with larger heads. A high resolution T1-weighted structural scan was also acquired for each participant. This scan utilized a magnetization-prepared rapid acquisition gradient echo (MPRAGE) sequence with parameters: TR 9.4 ms, inversion time 300.0 ms, relaxation delay time (after readout prior to inversion) 300.0 ms, linear phase encoding, TE 3.7 ms, matrix <sup>240</sup> <sup>×</sup> <sup>192</sup> <sup>×</sup> 128, field of view 240 <sup>×</sup> <sup>192</sup> <sup>×</sup> 192 mm<sup>3</sup> , 1.0 × 1.0 <sup>×</sup> 1.5 mm<sup>3</sup> voxels, whole brain coverage.

## 2.5. Analysis of Task Performance and Questionnaires

Error rates were not normally distributed (heavily skewed toward zero). Bootstrap and permutation tests were used to compare NoGo commission error rates vs. zero and commission rates for aversive vs. neutral NoGo trials. T-tests were used to compare Go trial latencies vs. zero and to compare neutral vs. aversive Go trial latencies. A participant's overall Barratt impulsivity score was computed as the sum of the six BIS subscales: attentional, cognitive instability, motor, perseverance, self-control, and cognitive complexity. We did a correlation analysis of ARBS risk vs. BIS impulsivity scores. Bootstrap regression tests (resampling residuals) were used to compare NoGo error rates against ARBS and BIS scores. Standard linear regression was used to compare Go trial latencies against ARBS and BIS scores.

## 2.6. fMRI Analysis

SPM8 and in-house MATLAB code were used for preprocessing of fMRI data. The preprocessing steps for each participant included: (1) 6 parameter rigid body motion correction of fMRI volumes, (2) non-linear spatial warping of fMRI data to the MNI EPI template space (interpolated to 3 <sup>×</sup> <sup>3</sup> <sup>×</sup> 3 mm<sup>3</sup> spatial resolution), (3) 8 mm full width at half maximum (FWHM) Gaussian spatial smoothing of fMRI data, (4) non-linear spatial warping of MPRAGE structural scans to the MNI T1 template space (interpolated to 1 <sup>×</sup> <sup>1</sup> <sup>×</sup> 1 mm<sup>3</sup> resolution). The first 6 volumes were discarded from each functional run to account for spin saturation effects.

Statistical modeling of fMRI data was done in two steps using custom-built MATLAB code. We first performed separate firstlevel, within-subjects general linear model (GLM) analyses on each participant. Within-subjects results were then combined using three different between-subjects mixed-effects analyses (Worsley et al., 2002). A manually-constructed mask was used to exclude voxels outside the brain. The mask included 82,244 voxels (size 3 × 3 × 3 mm) inside the brain. Each withinsubject GLM included either four or five sets of finite impulse response (FIR) predictors, one set for each of the four trial types (neutral Go, aversive Go, neutral NoGo, aversive NoGo) and, for participants who made errors, one set of predictors for error trials (collapsed across trial types). Error trials were rare (0–17% of trials, per participant). The GLM included 7 FIR impulse predictors (corresponding to 7 functional volume times, each lasting 2 s) per trial type. The FIR predictors represented deconvolved activation timecourses for the different trial types (see Serences, 2004). The GLM also incorporated a set of nuisance predictors for each run. These included constant run offset, linear drift, cosine, and sine with period equal to twice the run length, and 6 rigid body motion parameters. In addition, for each run, we computed three nuisance predictors by taking the mean activation timecourse over voxels in three regions expected a priori to contain only noise signals: the region outside the brain, a region entirely within the white matter, and a region inside the ventricles. The region outside the brain was defined based on the above-mentioned mask. The regions inside the white matter and ventricles were defined manually using the Marsbar toolbox. We also included two nuisance predictors for each run that were derived from independent component analyses (ICA) as follows. Two ICA analyses were applied to all four fMRI runs from all 21 participants using the GIFT software package version 2.0 from Vincent Calhoun's group (http://mialab.mrn.org/software). In one analysis, the four runs for each participant were treated as separate sessions. For the other analysis, the four runs were concatenated and treated as a single session. For each analysis, 20 components were computed using the default settings. Components were visually inspected and categorized as containing task-related brain signal or noise. One component containing motion-related noise was selected from each ICA analysis, and the corresponding timecourses for each fMRI run were included as nuisance predictors in each participant's first-level GLM. Each within-subject GLM was fit to the data using weighted least squares that corrected for autocorrelated noise. Ten autocorrelation coefficients (lags of 1– 10 volume times) were computed for each functional slice across the whole brain using the residuals from a non-corrected initial GLM fit. Then, the design matrix and each voxel's timecourse were pre-whitened, and auto-correlation-corrected beta weights were computed as described in Burock and Dale (2000) and Worsley et al. (2002).

For each subject separately, we computed three first-level (within-subjects) statistical contrast maps (two-tailed t statistic maps) from the GLM beta weights. The response inhibition contrast was (aversive NoGo + neutral NoGo) - (aversive Go + neutral Go). The emotional valence contrast was (aversive NoGo + aversive Go) - (neutral NoGo + neutral Go). The interaction contrast was (aversive NoGo - aversive Go) - (neutral NoGo neutral Go). Contrasts were computed from the FIR beta weights representing activation across the 3rd and 4th time points of the FIR deconvolved timecourses. The 3rd and 4th time points, which correspond to 4 and 6 s from trial start, were chosen a priori based on the typical BOLD hemodynamic peak time around 4–6 s (Aguirre et al., 1998).

For each of the three first-level statistical contrasts, we performed three second-level linear regression analyses using the mixed-effects method of Worsley et al. (2002). The first second-level analysis examined mean contrast magnitude across all 21 participants for each of the three first-level contrasts. The second-level design matrix in this case consisted only of a column of ones. The other two second-level analyses modeled the linear relationship between either ARBS risk scores or BIS impulsivity scores and the first-level contrast magnitude. For these models, the second-level design matrix included one constant offset predictor column and one column with either the mean-centred ARBS scores or the mean-centred BIS scores for all 21 participants. To summarize, for the three firstlevel contrasts (response inhibition contrast, emotional valence contrast, interaction contrast), statistical t-maps were computed testing for significant contrast (independent of ARBS or BIS scores), significant linear relationship between contrast and ARBS scores, and significant linear relationship between contrast and BIS scores.

Statistical t-maps were thresholded voxelwise at p < 0.01 (|t| > 2.626, two-tailed, df = 98). A cluster size threshold of 349 voxels (9423 mm<sup>3</sup> ) was then applied to achieve global correction for multiple comparisons at p < 0.05 across the voxel population. The cluster size threshold value was determined using permutation testing (Winkler et al., 2014). Briefly, we simulated a simpler version of the analysis 5000 times. On each iteration, the four trial types were randomly permuted independently for each participant. Second-level analyses were then performed using a basic GLM model (as opposed to the computationally-expensive method of Worsley et al. (2002), which would have made the simulation run time infeasible). After voxelwise thresholding at p < 0.01 (|t| > 2.863, two-tailed, df = 19), maximum cluster sizes were counted across the 5000 simulations. A maximum cluster size of 349 voxels or larger was found to occur in 5% of simulated analyses, and this was used as the cluster size threshold to achieve global p < 0.05. Note that this threshold is much larger than that computed by Monte Carlo simulation based on the method of AlphaSim (Ward, 2000). Our implementation of the AlphaSim method (fMRIMonteCluster, available at github.com/mbrown/fmrimontecluster) provides a cluster size threshold of 106 voxels for our data. This estimate was computed using a residual smoothness FWHM of 12.5 mm, which was the mean residual smoothness derived from all participants' residuals volumes using AFNI's 3dFWHMx function. Woo et al. (2014) demonstrated with fMRI analysis simulations that, with a voxelwise threshold of p < 0.01, the cluster size threshold returned by Gaussian random field theory can be too small to fully correct for multiple comparisons. We built a simulation similar to that presented in Woo et al. (2014) and confirmed that the cluster threshold of 349 does fully correct for multiple comparisons at the expected familywise error rate of α = 0.05 when using a voxelwise threshold of p < 0.01.

In total, there were nine second-level statistical maps:


We did additional quality assurance analyses on the second-level maps described above. For each of the second-level maps, an automated algorithm was used to grow a cluster (or region) of voxels around each positive or negative statistical peak (local extremum) in the associated t-map. For a given cluster, each participant's mean BOLD signal was computed by averaging across all voxels in the cluster. First-level GLM analyses were then conducted on the average timecourses, and event-related activation timecourses were computed for each of the four trial types based on the finite impulse response predictors from the fitted GLMs. Unpublished results from our group indicate that fMRI statistical contrast maps that look acceptable on visual inspection can be generated from underlying timecourses that seem to contain substantial signal noise and that may not reflect BOLD signals from the brain. Though consideration of event-related timecourses for quality assurance purposes is not standard practice in fMRI analysis, we suggest that it should be. Accordingly, we discarded regions whose activation timecourses were severely dissimilar to the expected difference of gammas hemodynamic response function shape (see Huettel et al., 2008, ch. 7), as determined by visual inspection. Note that the regression analyses done upon the mean timecourses extracted from each region were used only for quality assurance purposes. Summary data from significant clusters (median pand t-values for regression vs. ARBS and regression vs. BIS presented in tables in the Results section) were computed from the whole-brain statistical maps. Specifically, median p- and tvalues were computed (across the voxels comprising a given cluster) from the various second-level statistical maps.

#### 2.7. Correlation Pattern Analysis

To address Hypotheses 2a and 2b (see Section 1.1), we compared correlation relationships between ARBS risk scores and fMRI contrast values with correlation relationships between BIS impulsivity scores and fMRI contrast values. We chose 26 prefrontal regions from the Harvard-Oxford atlas (distributed with FSL, see http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Atlases). Specifically, we used the HarvardOxford-cortl-maxprob-thr25- 1mm.nii file from the series of files defining the Harvard-Oxford atlas. The 26 regions are listed in the Results in Section 3.6. For each of the three first-level statistical contrasts (response inhibition, emotional valence, interaction), we computed the average contrast value for each participant for each of the 26 regions. For each of the three contrasts, for each of the 26 regions, we computed the correlation between participants' contrast values and their ARBS scores as well as between contrast values and BIS scores. This created a set of 78 correlation values for ARBS scores and another set of 78 correlation values for BIS scores. We will call the 78 correlation values from a given instrument (ARBS or BIS) that instrument's correlation pattern. The correlation pattern characterized the relationship of the instrument's scores with fMRI contrast patterns in prefrontal cortex. We present the correlation patterns computed from all 21 participants in Section 3.6.

To test whether correlation patterns for ARBS and BIS scores were similar or different, we used bootstrap sampling with 10,000 iterations. For each iteration, we randomly assigned participants into two groups with each group containing similar proportions of low-risk (ARBS score ≤13) and high-risk (ARBS ≥17) participants. We computed the correlation pattern for ARBS (denoted Eca) using the first group and the correlation pattern for BIS (denoted Ec<sup>b</sup> ) using the second group. We then computed the following similarity measure between the two correlation patterns S(Eca, Ec<sup>b</sup> ):

$$S(\vec{c}\_a, \vec{c}\_b) = 1 - \frac{||\vec{c}\_a - \vec{c}\_b||}{\sqrt{78}}.\tag{1}$$


Hypotheses 2a and 2b were tested as follows. The mean similarity measure was computed across the 10,000 iterations. A similarity measure of 0.3 or below was defined as indicating a large difference between the correlation patterns (i.e. "substantially different"). A similarity measures of 0.7 or above was defined as indicating that the correlation patterns are identical or close to identical (i.e. "very similar"). To test Hypothesis 2a, which held that risk and impulsivity measures have similar associations with fMRI patterns, we tested against a null hypothesis that the similarity measures were substantially different by computing the proportion of iterations in which the similarity measure was <0.3. To test Hypothesis 2b, which stated that risk and impulsivity measures have different associations with fMRI patterns, we tested against a null hypothesis that the similarity measures were very similar by computing the proportion of similarity measures >0.7.

## 3. Results

#### 3.1. Risk and Impulsivity Scores

Mean (± standard deviation) participant ARBS risk score was 15.6 ± 4.6, and ARBS scores ranged from 9 to 23. ARBS scores range from 9 (lowest risk) to 30 (highest risk), and Jankowski et al. (2007) recommend a cutoff of >17 for determining clinical highrisk status. In our sample, 10 participants had ARBS score ≥17, while the remainder had ARBS scores ≤13, in the low-risk range. Participants' Barratt impulsivity scale (BIS) scores had a mean of 67.6 ± 16.1 with a range of 44–95. BIS scores range from 30 (least impulsive) to 120 (most impulsive). Normal impulsivity is represented by BIS scores in the 52–71 range, with scores at or below 51 indicating a very controlled individual, and scores at or above 72 indicating high impulsivity (Stanford et al., 2009). Differences in overall BIS scores were driven by all six BIS subscales (see **Table 2**). BIS and ARBS scores were highly correlated (<sup>r</sup> <sup>=</sup> 0.78, <sup>p</sup> <sup>=</sup> <sup>3</sup> <sup>×</sup> <sup>10</sup>−<sup>5</sup> , t = 5.42, df = 19) as illustrated in **Figure 2**. BIS scores explained 59.7% of the variance in the ARBS scores.

#### 3.2. Task Performance

Participants made commission errors on NoGo trials at a mean rate of 5.8 ± 6.9%, which was low but significantly above zero (p < 0.0001, bootstrap test). Error rates did not differ significantly between NoGo trials with aversive vs. neutral distractors (p = 0.23, permutation test). NoGo error rates did not vary significantly as a function of participant ARBS risk scores (p = 0.36) or Barratt impulsivity scores (p = 0.12) on bootstrap regression tests. Fifteen of twenty-one participants made no omission errors on Go trials. Five of twenty-one participants had low omission error rates in the 0.6–3.7% range. One clinical participant fell asleep intermittently toward the end of the study and exhibited an omission error rate of 20.1%. Due to the difficulty of recruiting clinical participants, we did not exclude this participant from the analysis, although we did separate out error trials in the fMRI analysis (see Section 2.6).

Go trial latencies were 772 ± 128 ms with neutral distractors and 808 ± 141 ms with aversive distractors, and this difference was significant (p = 0.0003, t = 4.7, df = 19). Go trial latencies did not show significant relationships with either ARBS risk scores (p = 0.42, t = 0.82, df = 19) or Barratt impulsivity scores (p = 0.14, t = 1.6, df = 19) on linear regression tests.

<sup>2</sup>The similarity measure defined in Equation (1) could theoretically assume values in the range [−1, 1]. In our analysis, no similarity measures <0 were encountered, so we ignored the negative case. Alternatively, one could eliminate the possibility of negative similarity values by truncating negative values to 0.


TABLE 2 | Summary of BIS scores and subscale scores.

R denotes correlation coefficient comparing subscore against overall BIS score (sum of 1st order subscores). P indicates p-value of statistical test for significant correlation with df = 19.

Similarly, Horn et al. (2003) and Asahi et al. (2004) did not find significant relationships between participant impulsivity scores and Go/NoGo task performance.

#### 3.3. fMRI Main Effects Results

The response inhibition contrast (NoGo – Go) was significant in many brain regions. Left motor/premotor cortex (**Figure 3**, left panel), supplementary motor area, posterior insula, and other regions showed greater activation for Go vs. NoGo regions (also see Supplementary Figure 2). The left motor/premotor cortex region shown in **Figure 3** exhibited activation significantly above baseline for neutral NoGo trials (p < 0.0001, t = 5.78, df = 20) and non-significantly above baseline for aversive NoGo trials (p = 0.11, t = 1.69, df = 20). Multiple regions exhibited greater activation for NoGo vs. Go trials including right posterior ventrolateral prefrontal cortex (vlPFC)/anterior insula as shown in **Figure 3** as well as dorsolateral prefrontal cortex (dlPFC), parietal and occipital regions associated with attention and visual processing, and other regions as shown in Supplementary Figure 2.

The emotional valence contrast (aversive − neutral distractors) also showed significance in many regions throughout the brain. Multiple regions exhibited positive contrast (aversive > neutral) including the amygdala, vlPFC, and angular gyrus (see **Figure 4**) as well as supplementary motor area, vlPFC, anterior and posterior cingulate cortex, medial frontal gyrus, orbitofrontal regions, parietal and occipital regions, and anterior temporal regions as shown in Supplementary Figure 3. Several regions showed negative emotional valence contrast (neutral > aversive) including regions in vlPFC, dlPFC, posterior insula, OFC, and other regions (Supplementary Figure 3).

## 3.4. fMRI Emotional Valence Contrast vs. Risk and Impulsivity Scores

We examined relationships between the emotional valence contrast (aversive − neutral distractor pictures) vs. ARBS risk scores (Map 5). See Section 2.6 for methodological details. There was a positive relationship between emotional valence contrast amplitude and participants' ARBS scores in the temporooccipital part of right middle temporal gyrus (MTG) (**Table 3**, **Figure 5**). Participants with low ARBS scores (≤13) exhibited greater activation for neutral vs. aversive distractor trials resulting in negative values for the first-level (aversive − neutral) contrast. Participants with high ARBS scores (≥17) showed the opposite pattern. The analysis of emotional valence contrast vs. BIS scores (Map 6) did not include a significant cluster surviving multiple comparison correction in right MTG. Follow-up region of interest analysis on the right MTG region from Map 5 did find a significant relationship between emotional valence contrast and BIS scores (**Table 3**), but caution is recommended in interpreting this result due to potential double-dipping issues (see Kriegeskorte et al., 2009; Vul et al., 2009). This result is explained by the fact that, in Map 6, there is an MTG cluster exhibiting a relationship between emotional valence and BIS scores, but this cluster is too small to survive correction for multiple comparisons. Our results are consistent with, but do not provide conclusive support for, a relationship between emotional valence contrast and BIS scores in right MTG.

## 3.5. Other Statistical Contrasts

The other statistical contrasts (listed below) did not exhibit any significant regions surviving cluster size threshold correction for multiple comparisons and quality assurance checks (see last

paragraph of Section 2.6). The statistical comparisons with no results included:


#### 3.6. Correlation Pattern Analysis

For each of 26 prefrontal regions from the Harvard-Oxford atlas, we computed correlations between participant fMRI firstlevel contrast values and ARBS risk scores as well as BIS impulsivity scores (see Section 2.7). The 26 regions are listed in **Table 4**. The resulting correlation values comprised a correlation pattern for either ARBS or BIS scores. **Table 4** shows correlation patterns computed using all participants. Bootstrap testing with 10,000 iterations was used for statistical testing (see Section 2.7). The mean similarity measure between correlation patterns for ARBS scores and for BIS scores was 0.53 ± 0.09 (std.). A histogram of similarity measures from all iterations is shown in **Figure 6**. Over all bootstrap iterations, the proportion of similarity measures <0.3 was 0.01, and the proportion of measures >0.7 was also 0.01. These results indicate that the mean similarity measure was significantly above and below the cutoffs, respectively, defining correlation patterns as substantially different and very similar (see Section 2.7). That is, correlation patterns for ARBS and BIS scores were neither identical (or close to identical) nor completely dissimilar but were instead partially similar while still exhibiting some differences.

## 4. Discussion

We investigated fMRI activation in adolescents (age 13–17) performing a Go/NoGo task with emotional distractor images. In the Introduction, we outlined eight specific hypotheses, which are discussed below. Our analyses also included exploratory aspects. In discussing the results from exploratory analyses, we

TABLE 3 | From emotional valence contrast vs. ARBS score analysis.

activation time course values.


Summary data for significant clusters identified in statistical t-maps comparing fMRI emotional valence contrast (aversive − neutral distractors) vs. ARB scores. X, Y, Z: MNI coordinates of region's peak statistical voxel. P- and t-values are median values across all voxels in each region (df = 98). Positive and negative t-values indicate, respectively, greater and lesser emotional valence contrast values for participants with larger ARBS or BIS scores. See Section 2.6 for details of analysis. MTG: Middle Temporal Gyrus.

had necessarily to rely on post-hoc interpretations, and standard cautions apply, for example related to reverse inference (see Poldrack, 2006). The ultimate test of any result is that it must survive independent replication.

#### 4.1. Emotional Go/NoGo Task in Adolescents

The response inhibition contrast (NoGo − Go) and emotional valence contrast (aversive − neutral distractors) revealed significant differences in many regions throughout the brain (see Section 3.3). These results were mostly similar to those previously reported by Brown et al. (2012) using the same task with young adult participants. Unlike Brown et al. (2012), the current study found regions exhibiting greater activation for neutral vs. aversive distractor images. In dlPFC and vlPFC, Brown et al. (2012) found greater activation for aversive distractors compared to neutral distractors, while the current study found the opposite pattern in dlPFC and parts of vlPFC. This difference may be due to differences in the participant groups. Brown et al. (2012) included young adults age 18–28 with low- to mediumrisk behavior tendencies whereas the current study included adolescents age 14–17 with low- as well as high-risk behavior tendencies. The current study's findings are also consistent with previous Go/NoGo studies (Garavan et al., 1999, 2002; Watanabe et al., 2002; Mostofsky et al., 2003; Aron et al., 2004b; Fassbender et al., 2004; Kelly et al., 2004; Rubia et al., 2005; Wager et al., 2005; Aron et al., 2007b; Mitchell, 2011) and with studies of emotional picture processing (Irwin et al., 1996; Bermpohl et al., 2006; Meseguer et al., 2007).

The finding of significant activation above baseline in left motor/premotor cortex for neutral NoGo trials is consistent with a potential role of motor and premotor cortex in response inhibition (see Aron et al., 2007a; Mirabella, 2014). It has been suggested by Mirabella (2014) that initiating a motor response and withholding one may involve interaction among overlapping brain regions based on evaluation of the outcomes of an action (or lack of action). He also proposed that the network of involved brain regions varies based on the decision-making context.

FIGURE 5 | Top row: Statistical t-map for regression of ARBS risk scores against fMRI emotional valence contrast (aversive − neutral distractors). Red/yellow regions exhibited larger contrast magnitudes in participants with higher ARBS risk scores. All results p < 0.05 (corrected for multiple comparisons). Color bars indicate t-value scaling. Slice X- or Z-coordinate in MNI space shown in upper-left. Axial image's left side corresponds to left side of brain. MTG: middle temporal gyrus. Second row: Mean deconvolved timecourses for four trial types for region outlined in green in first row. Error bars denote mean across all participants' standard error of mean activation time course values. Timecourses denoted Low-Risk were computed from 11 low-risk participants with ARBS risk scores ≤13. Timecourses denoted High-Risk were computed from 10 high-risk participants with ARBS risk scores ≥17. Bottom row: Scatter plots show emotional valence contrast magnitude vs. participants' ARBS risk scores (left) and vs. BIS impulsivity scores (right) for right MTG region outlined in green in first row. Red line shows linear regression of contrast magnitude against participant ARBS or BIS scores.

For example, the presence of emotional stimuli would recruit emotion processing regions such as the amygdala. This view provides a possible explanation for some regions' involvement in motor control as well as a variety of other tasks. For example, right vlPFC is involved in redirecting selective attention

(Corbetta and Shulman, 2002), in motor response inhibition (Aron et al., 2014), and in suppression of memories (Benoit and Anderson, 2012). The overlap between significant regions revealed by our response inhibition and emotional valence contrasts is consistent with this view.

#### TABLE 4 | Results from correlation pattern analysis.


Correlation pattern analysis. 26 regions were used from the Harvard-Oxford atlas. # denotes region numbering from the atlas. X, Y, Z denote region centroid coordinates in mm. Volume is in mm<sup>3</sup> . Correlations between values from each of three first-level fMRI contrasts and either ARBS or BIS scores were computed (denoted Corr ARBS and Corr BIS, respectively). First level contrasts included the response inhibition contrast (denoted Inhib), the emotional valence contrast (Emot), and the interaction contrast (Inter). Also see Section 3.6.

## 4.2. Prefrontal Executive Control, Emotion Processing, Risk Tendency, and Impulsivity

Our exploratory statistical comparisons (Maps 1–9, see Section 2.6) did not confirm Hypotheses 1a, 1b, or 1c. We did not find significant prefrontal clusters showing relationships between either ARBS risk scores or BIS impulsivity scores and any of the three first-level contrasts (response inhibition, emotional valence, or interaction contrast). The lack of results may be due to our use of a larger-than-usual cluster size threshold to correct for multiple comparisons in accordance with recent results from Woo et al. (2014) (also see discussion in Section 2.6). Though the large threshold properly corrects for multiple comparisons, it may reduce sensitivity, thereby increasing false negative errors. Our own simulation results (unpublished) support this proposition.

## 4.3. Emotion Processing, Risk Tendency, and Impulsivity in MTG

In exploratory analyses, we found that emotional valence contrast values in right MTG were modulated by ARBS risk. High-risk participants exhibited greater right MTG activation for aversive distractor pictures, whereas low-risk participants exhibited greater right MTG activation for neutral distractors. We also found partial but not conclusive evidence for a relationship in right MTG between emotional valence contrast and participant BIS impulsivity scores (see Section 3.4).

Right MTG has been implicated in higher visual processing, response inhibition, and processing emotional stimuli (Schäfer et al., 2005; Sabatinelli et al., 2011; Bhaijiwala et al., 2014). ADHD patients have been shown to exhibit less MTG activity than healthy controls when performing tasks involving inhibition (Bhaijiwala et al., 2014), suggesting that MTG plays a role in inhibition and impulse control disorders. Bhaijiwala et al. (2014) also suggest that right MTG is part of a circuit for taskrelated functions and inhibition<sup>3</sup> . Associations have also been found between MTG activity, reward processing, and emotional processing. In a meta-analysis, Sabatinelli et al. (2011) located clusters in the MTG that were significantly associated with images of emotional facial expression or emotional scenes, suggesting a role in emotional stimulus processing. Moreover, Schäfer et al. (2005) found increased fMRI activation in MTG for facial

<sup>3</sup>Although our results included differences related to emotion processing but not response inhibition in right MTG, the results from Bhaijiwala et al. (2014) provide important context in this area.

expressions and other visual stimuli that elicited fear (as opposed to neutral or disgust-evoking stimuli). They suggest that fearinducing images may promote a fight-or-flight response involving MTG-based visual processing. Our findings in right MTG are consistent with its suggested role in processing emotional stimuli. Risk- and impulsivity-related modulation of right MTG indicate changes in emotional stimulus processing that may contribute to individual risk tendencies and impulsivity levels.

#### 4.4. Impulsivity and High-Risk Behavior

As indicated by the analysis of correlation patterns (Section 3.6), there were many similarities in the associations that ARBS risk scores and BIS impulsivity scores exhibited with fMRI activation levels related to response inhibition or emotional valence as predicted by Hypothesis 2a. However, the correlation pattern analysis also showed differences between these associations, supporting Hypothesis 2b. We conclude that ARBS risk scores and BIS impulsivity scores show relationships with fMRI activation related to response inhibition and emotional valence that are partially similar while still maintaining differences.

Higher impulsivity scores based on self-report instruments are associated with increased risk behavior tendencies (Levitt, 1991; Moore and Rosenthal, 1993; Luengo et al., 1994; Stanford et al., 1996; Gullo and Dawe, 2008; Romer et al., 2009; Romer, 2010; Dalley et al., 2011; Mishra and Lalumière, 2011; Christiansen et al., 2012; Stautz and Cooper, 2013). However, dissociations between risk behavior and impulsivity have also been reported (Ryan et al., 2013; Brown et al., 2015). The sample of adolescents recruited in the current study exhibited a high correlation (0.78) between ARBS risk scores and BIS impulsivity scores, consistent with a role of impulsivity in contributing to high-risk behavior in the higher risk participants. Accordingly, ARBS riskrelated modulation of fMRI activation patterns in the emotional Go/NoGo task were associated with similar BIS impulsivityrelated modulation. The relationship between impulsivity and risk behavior is complex and not necessarily consistent (see Romer, 2010; Dalley et al., 2011; Blakemore and Robbins, 2012; Whelan et al., 2012), but our results suggest a relationship between elevated impulsivity and high-risk behavior, both in terms of psychometric testing and modulation of fMRI activation patterns, in groups such as the high-risk adolescent participants included in this study.

## 4.5. Models of Risk Behavior

Models of high-risk behavior in adolescents proposed by Ernst and colleagues (Ernst et al., 2006; Ernst and Mueller, 2008; Ernst and Fudge, 2009) and by Casey and colleagues (Casey et al., 2008, 2011) posit that limbic responses to emotional stimuli are altered in adolescence, that prefrontal regulatory mechanisms are not fully developed, and that higher rates of risk behavior result from this imbalance. The models differ in details of limbic emotionrelated responses. In the Introduction, we outlined Hypotheses 3a and 3b that the models of Ernst et al. and Casey et al. would be consistent with, respectively, reduced and increased emotional valence contrast (aversive − neutral pictures) in limbic regions, particularly in the amygdala, in high-risk participants. Contrary to expectation, we did not observe risk- or impulsivityrelated differences in emotional valence contrast in the amygdala, other deep brain nuclei, or other limbic structures associated with emotion processing. As such, our results do not provide differential support for either the Ernst model or the Casey model. We also suggested that both models would be consistent with an association between elevated individual risk behavior tendencies and reduced prefrontal fMRI activation related to response inhibition (Hypothesis 4). Prefrontal regions, such as vlPFC (**Figure 3**), that exhibited response inhibition-related activation in the form of larger positive BOLD deflections for NoGo vs. Go trials did not exhibit modulation by ARBS or BIS scores. As such, our results do not provide clear support for Hypothesis 4.

## 4.6. Conclusions

We observed partial similarities in how participant risk tendencies and impulsivity levels were associated with changes in fMRI activation patterns related to response inhibition and emotional stimulus processing in prefrontal regions. These changes in activation patterns may reflect changes in processing related to response inhibition and decision-making in emotional contexts that could underlie high-risk behavior tendencies and high impulsivity status.

## Acknowledgments

This work was supported by the Norlien Foundation, the Women and Children's Health Research Institute, Alberta Innovates—Health Solutions, the Canadian Institutes of Health Research, the Alberta Mental Health Board, the University of Alberta, Alberta Health Services, the University of Alberta Hospital Foundation, and the Alberta Innovates Centre for Machine Learning. The work described here was undertaken under the auspices of the Child/Adolescent At Risk Population Imaging, neuropsychology, and population health team (CARPI), of which the authors are members.

## References


## Supplementary Material

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fnsys. 2015.00124


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Brown, Benoit, Juhás, Dametto, Tse, MacKay, Sen, Carroll, Hodlevskyy, Silverstone, Dolcos, Dursun and Greenshaw. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

## Neural correlates of high-risk behavior tendencies and impulsivity in an emotional Go/NoGo fMRI task

Matthew R. G. Brown<sup>1</sup> \*, James R. A. Benoit <sup>1</sup> , Michal Juhás <sup>1</sup> , R. M. Lebel 2 † , Marnie MacKay <sup>1</sup> , Ericson Dametto<sup>1</sup> , Peter H. Silverstone<sup>1</sup> , Florin Dolcos 1, 3, Serdar M. Dursun<sup>1</sup> and Andrew J. Greenshaw<sup>1</sup>

#### Edited by:

Mikhail Lebedev, Duke University, USA

#### Reviewed by:

Christia Schmahl, Medical Faculty Mannheim/Heidelberg University, Germany Liangsuo Ma, University of Texas Health Science Center at Houston, USA Jane Elizabeth Joseph, Medical University of South Carolina, USA Annegret Krause-Utz, Zentralinstitut für Seelische Gesundheit, Germany (in collaboration with Christian Schmahl)

#### \*Correspondence:

Matthew R. G. Brown, Department of Psychiatry, University of Alberta Campus, 12-127A Clinical Sciences Building, Edmonton, AB T6G2B3, Canada mbrown2@ualberta.ca

#### †Present Address:

R. M. Lebel, General Electric, Calgary, AB, Canada

> Received: 27 November 2014 Accepted: 12 February 2015 Published: 10 March 2015

#### Citation:

Brown MRG, Benoit JRA, Juhás M, Lebel RM, MacKay M, Dametto E, Silverstone PH, Dolcos F, Dursun SM and Greenshaw AJ (2015) Neural correlates of high-risk behavior tendencies and impulsivity in an emotional Go/NoGo fMRI task. Front. Syst. Neurosci. 9:24. doi: 10.3389/fnsys.2015.00024 <sup>1</sup> Department of Psychiatry, University of Alberta, Edmonton, AB, Canada, <sup>2</sup> Department of Biomedical Engineering, University of Alberta, Edmonton, AB, Canada, <sup>3</sup> Department of Psychology, Neuroscience Program, and the Beckman Institute for Advanced Science and Technology, University of Illinois Urbana-Champaign, Urbana-Champaign, IL, USA

Improved neuroscientific understanding of high-risk behaviors such as alcohol binging, drug use, and unsafe sex will lead to therapeutic advances for high-risk groups. High-risk behavior often occurs in an emotionally-charged context, and behavioral inhibition and emotion regulation play important roles in risk-related decision making. High impulsivity is an important potential contributor to high-risk behavior tendencies. We explored the relationships between high-risk behavior tendencies, impulsivity, and fMRI brain activations in an emotional Go/NoGo task. This task presented emotional distractor pictures (aversive vs. neutral) simultaneously with Go/NoGo stimuli (square vs. circle) that required a button press or withholding of the press, respectively. Participants' risk behavior tendencies were assessed with the Cognitive Appraisal of Risky Events (CARE) scale. The Barratt Impulsivity Scale 11 (BIS) was used to assess participant impulsivity. Individuals with higher CARE risk scores exhibited reduced activation related to response inhibition (NoGo−Go) in right orbital frontal cortex (OFC) and ventromedial prefrontal cortex. These regions did not show a significant relationship with impulsivity scores. Conversely, more impulsive individuals showed reduced emotionrelated activity (aversive−neutral distractors) in dorsomedial prefrontal cortex, perigenual anterior cingulate cortex, and right posterior OFC. There were distinct neural correlates of high-risk behavior tendency and impulsivity in terms of brain activity in the emotional Go/NoGo task. This dissociation supports the conception of high-risk behavior tendency as a distinct construct from that of impulsivity. Our results suggest that treatment for high-risk behavior may be more effective with a nuanced approach that does not conflate high impulsivity necessarily with high-risk behavior tendencies.

#### Keywords: high-risk behavior, impulsivity, emotional Go/NoGo, CARE, BIS, fMRI

## 1. Introduction

High-risk behaviors such as binge drinking, substance abuse, unsafe sex, and physical violence create increased potential for harm to mental and physical health and general wellbeing (Jessor, 1991; Arnett, 1992; Blakemore and Robbins, 2012). High-risk behaviors account for a substantial proportion of deaths and injuries among adolescents and young adults (NCHS, 2007; Statistics Canada, 2010; Viner et al., 2012) as well as poor health outcomes in later life (Anda et al., 2006; Eaton et al., 2006; Hawton and O'Connor, 2012). In addition to personal costs, high-risk behaviors impose large economic costs on society (MacKersie et al., 1995), for example in health services. Improved understanding of the neurobiology of high-risk behavior is important and shows promise for improved treatments and policies for addressing high-risk behavior.

Many high-risk behaviors occur in emotionally-charged circumstances such as "wild" parties or interpersonal confrontations. High-risk behavior is complex, but behavioral inhibition, emotional responses, and emotion regulation are thought to play important roles in risk-related decision making in many such circumstances. Previous studies have emphasized individual differences and developmental changes in impulsivity and emotion processing as important factors contributing to high-risk behavior tendencies (Jessor, 1991; Arnett, 1992, 1994, 1996; Ernst et al., 2006; Steinberg, 2007; Casey et al., 2008; Ernst and Mueller, 2008; Gullo and Dawe, 2008; Steinberg, 2008; Ernst and Fudge, 2009; Romer et al., 2009; Romer, 2010; Casey et al., 2011; Dalley et al., 2011; Mitchell, 2011; Blakemore and Robbins, 2012; Whelan et al., 2012; Bari and Robbins, 2013). In addition to circumstantial decision making, there is an important interplay between highrisk behaviors, emotional dysregulation, and impulsive decision making in a clinical context, for example in borderline personality disorder (BPD), attention-deficit/hyperactivity disorder (ADHD), obsessive-compulsive disorder (OCD), substance use disorder (SUD), pathological gambling, and bulimia nervosa (Aron and Poldrack, 2006; Chamberlain and Sahakian, 2007; Kemps and Wilsdon, 2010; Reid et al., 2014; Sebastian et al., 2014). In this study, we investigate impulsivity as it relates to high-risk behavior and fMRI brain activity patterns associated with behavioral control in emotional contexts.

Impulsivity is a complex construct, and there are multiple proposals on the ontology of impulsivity and its different possible components and sub-processes (see Whiteside and Lynam, 2001; Dalley et al., 2011; Bari and Robbins, 2013). Dalley et al. (2011) define impulsivity informally as "the tendency to act prematurely without foresight." Impulsivity can be operationally measured using self-report instruments such as the Barratt Impulsivity Scale (BIS; Barratt, 1959; Patton et al., 1995). Other widely-used instruments that assess impulsivity, subcomponents of impulsivity, or constructs related to impulsivity include the Urgency, Premeditation, Perseverance, and Sensation Seeking Scale (UPPS; Whiteside and Lynam, 2001); the Sensation Seeking Scale (SSS; Zuckerman, 1994); the Tridimensional Personality Questionnaire (TPQ; Cloninger et al., 1991); and the I-7 Impulsiveness Questionnaire (I7; Eysenck et al., 1985). The choice of questionnaire used to assess impulsivity implies a certain conception of the impulsivity construct. For example, the UPPS includes sensation seeking as a subcomponent of impulsivity, whereas the BIS does not include sensation seeking as a subscale nor does it include the sort of questions used to assess sensation seeking in the UPPS or SSS. At present, there is no consensus on a single, "correct" version of the impulsivity construct. We focus on impulsivity as captured by the BIS instrument, while acknowledging that other conceptions also provide valuable perspective and insight.

The BIS and other impulsivity scales provide numerical scores for an individual's overall impulsivity level, as well as subscores for various subcomponents of impulsivity. Greater impulsivity scores on standardized self-report questionnaires are known to be associated with increased risk behavior tendencies (Levitt, 1991; Moore and Rosenthal, 1993; Luengo et al., 1994; Stanford et al., 1996; Cyders et al., 2007; Gullo and Dawe, 2008; Romer et al., 2009; Zapolski et al., 2009; Romer, 2010; Dalley et al., 2011; Mishra and Lalumière, 2011; Christiansen et al., 2012; Stautz and Cooper, 2013). However, it is noteworthy that at least one dissociation between risk behavior tendencies, in this case smoking tendencies, and impulsivity has been reported (Ryan et al., 2013). In addition to impulsivity, other contributors to high-risk behavior have been proposed, such as reward seeking and sensation seeking (see Romer et al., 2009; Romer, 2010; Dalley et al., 2011; Blakemore and Robbins, 2012).

Individual differences in impulsivity may be related to differences in cognitive control of behavior, emotions, and other mental processes (Dalley et al., 2011; Bari and Robbins, 2013). Inhibition has been suggested to be an important component of cognitive control (see Ainslie, 1975; Smith, 1992; Dempster and Brainerd, 1995; Aron, 2007), with response inhibition being one widely-studied example of inhibition (see Wager et al., 2005; Aron, 2007). The classic Go/NoGo task (Donders, 1868/1969) provides a means of recruiting and investigating response inhibition processes. This task presents the participant with frequent Go stimuli requiring a button press as well as rare NoGo stimuli requiring inhibition of the button press. The button press response is made automatic, or prepotent, by the frequent Go trials, requiring the participant to actively inhibit that response in NoGo trials. Psychometric measures of impulsivity, such as the BIS, do not correlate significantly with behavioral performance measures on the Go/NoGo task, including reaction times and error rates (Horn et al., 2003; Asahi et al., 2004; Reynolds et al., 2006, 2008; Christiansen et al., 2012). However, Dalley et al. (2011) differentiate between impulsivity based on motor disinhibition from that based on temporal discounting. It is possible that psychometrically-derived behavioral impulsivity may be more related to temporal discounting, while poor Go/NoGo performance may be more related to motor disinhibition. Nonetheless, psychometric impulsivity measures have been shown to be correlated with changes in fMRI activation patterns evoked by the Go/NoGo task. Individuals with greater Barratt Impulsivity Scores were found to exhibit less response inhibition-related fMRI activation in right dorsolateral prefrontal cortex (dlPFC) (Asahi et al., 2004) and in dorsomedial prefrontal cortex (dmPFC) (Horn et al., 2003). Horn et al. (2003) also found that scores on Eysenck's Impulsivity Scale were positively correlated with response inhibition-related activation in right ventrolateral prefrontal cortex (vlPFC<sup>1</sup> ). Therefore,

<sup>1</sup>Note on anatomical terminology: To make the manuscript more accessible to readers outside the brain imaging/neuroanatomy subfields, we employ widelyused, broad subdivisions of prefrontal cortex including vlPFC, dlPFC, vmPFC, and dmPFC, as opposed to anatomically more specific terms such as inferior, middle, and superior frontal gyri. It has been pointed out, however, that precise definitions of these broad prefrontal subdivisions exhibit some variability in terms of extent and anatomical boundaries across different studies (for example, see Cieslik et al., 2013).

there is evidence in the literature that individual differences in psychometrically-measured impulsivity may be related to differences in recruitment of cognitive processes during Go/NoGo task performance.

This study examined the relationships between participants' high-risk behavior tendencies, levels of impulsivity, and recruitment of response inhibition and emotional stimulus processing during performance on an emotional Go/NoGo task using functional magnetic resonance imaging (fMRI). The emotional Go/NoGo task used here presented a distractor image that was emotionally neutral or aversive simultaneously with each Go or NoGo stimulus. This task also allowed us to investigate response inhibition specifically in aversive emotional contexts by comparing NoGo vs. Go activation in the presence of aversive distractor images.

A variety of prefrontal brain regions are thought to have roles in the executive and emotion processing needed to perform this emotional Go/NoGo task. dlPFC, vlPFC, orbitofrontal cortex (OFC), and ventromedial PFC (vmPFC) are involved in response inhibition in the Go/NoGo task as well as inhibition in other executive control tasks (see Aron et al., 2004a, 2007; Dolcos et al., 2011; Mitchell, 2011; Mahmood et al., 2013). Anterior cingulate cortex (ACC) has been implicated in error detection and conflict monitoring in the Go/NoGo and other cognitive tasks (Carter et al., 1998, 1999; Garavan et al., 1999; Botvinick et al., 2004; Kerns et al., 2004; Brown and Braver, 2005; Mitchell, 2011). Dorsomedial PFC (dmPFC) may also contribute to response conflict processing (see Ridderinkhof et al., 2004) as well as to response selection and response inhibition in the Go/NoGo task (Simmonds et al., 2008). dmPFC is also thought be involved in resolution of response conflict and outcome value-related aspects of decision making (Venkatraman et al., 2009). OFC and vlPFC are thought to be involved in processing emotional stimuli, for example to evaluate valence (Dolcos et al., 2011; Mitchell, 2011). Multiple prefrontal regions including OFC, vmPFC, dmPFC, vlPFC, and dlPFC are also associated with emotion regulation (Dolcos et al., 2011; Mitchell, 2011; Golkar et al., 2012).

In the work described here, we performed a new analysis of fMRI data previously presented by Brown et al. (2012). The original Brown et al. (2012) paper did not consider participant risk tendencies nor impulsivity levels. The current work features a new analysis of relationships between individual risk behavior tendencies or impulsivity scores and fMRI activation patterns in the emotional Go/NoGo task. The analysis presented here is statistically independent of the previous analysis presented in Brown et al. (2012).

Brown et al. (2012) found fMRI changes related to responseinhibition and emotion processing in many brain regions. In the response inhibition contrast, they found significantly larger Go vs. NoGo activation in left motor cortex and other regions and larger NoGo vs. Go activation in ventrolateral prefrontal cortex as well as other cortical regions. These findings are consistent with previous Go/NoGo studies (Garavan et al., 1999, 2002; Watanabe et al., 2002; Mostofsky et al., 2003; Aron et al., 2004b; Fassbender et al., 2004; Kelly et al., 2004; Rubia et al., 2005a; Wager et al., 2005; Aron et al., 2007; Mitchell, 2011). In the emotional valence contrast, they found greater activation for aversive vs. neutral distractor pictures in orbitofrontal cortex, lateral prefrontal cortex, insula, the amygdala and surrounding cortex, anterior cingulate cortex, medial prefrontal cortex, and bilateral posterior middle temporal gyrus and angular gyrus. These results are also consistent with studies of emotional picture processing (Irwin et al., 1996; Bermpohl et al., 2006; Meseguer et al., 2007). Brown et al. (2012) looked specifically at interaction of fMRI activation patterns related to response-inhibition and emotion processing in vlPFC. They found that these two sources of fMRI activation changes summated in a straightforward manner; emotional context (aversive vs. neutral distractors) did not suppress or potentiate fMRI signals related to response inhibition in vlPFC (see Brown et al., 2012 for further details).

In the current study, we tested several hypotheses. We expected fMRI activation patterns evoked in prefrontal regions by the response inhibition and emotion processing components of the emotional Go/NoGo task to exhibit modulation based on participants' high-risk tendencies and impulsivity levels. Based on Horn et al. (2003) and Asahi et al. (2004), we expected impulse-control activation in right dlPFC and dmPFC to be modulated by psychometric impulsivity scores. We did exploratory analyses looking for other prefrontal regions that might show modulation of fMRI activation related to response inhibition and/or emotional stimulus processing based on participants' risk tendencies and impulsivity scores. Given the previouslysuggested role of impulsivity in contributing to high-risk behavior (see discussion above), we expected this exploratory analysis to reveal a subset of prefrontal brain regions exhibiting similar modulation of fMRI activity patterns by both risk tendency and impulsivity. In addition, given that aversive emotional contexts have been suggested to promote impulsive decision-making to escape aversive stimuli or circumstances in certain individuals (see negative urgency as discussed in Whiteside and Lynam, 2001; Cyders and Smith, 2008), we expected that the exploratory analysis might reveal a relationship between participant risk behavior or impulsivity tendencies and fMRI activation patterns related to response inhibition in the presence of aversive distractors.

## 2. Methods

The Health Research Ethics Board at the University of Alberta approved this study.

## 2.1. Participants

Nineteen young adults were recruited into the study (12 female and 7 male, age range 18–28 years, mean age 22.7 ± 2.3 years). All participants were undergraduate or graduate students recruited from the University of Alberta student population. Based on the Edinburgh Handedness Inventory (Oldfield, 1971), 16 participants were right-handed. One participant was left-handed, and two were ambidextrous. All participants gave informed, written consent in English. Participants reported no history of diagnosed psychiatric or neurological disorder and no history of learning disability. Participants exhibited low to moderate risk behavior tendencies based on the CARE risk questionnaire. Based on the BIS impulsivity questionnaire, participants fell in the low to moderate impulsivity range. No participants exhibited very high risk behavior or very high impulsivity tendencie (see Sections 2.2, 3.1 for details of CARE and BIS scores).

### 2.2. Questionnaires

We used the expected involvement component of the Cognitive Appraisal of Risky Events (CARE) questionnaire (Fromme et al., 1997) to assess each participant's risk behavior tendencies. This instrument includes 30 questions that ask a participant to rate how likely they are to engage in various risk-related behaviors in the next 6 months. A seven-point Likert scale is used with ratings ranging from 1 (not at all likely) to 7 (extremely likely). The CARE provides six risk behavior subscores: illegal drug use, fighting and petty crime, high-risk sex, alcohol abuse, high-risk sports, and cheating at or neglect of academic/employment work. To derive a single risk score for each participant, we took the mean score over the six subscores. This overall CARE risk score could range from a minimum of 1 (lowest risk tendency) to a maximum of 7 (highest risk tendency).

To assess participants' impulsivity, we used the Barratt Impulsivity Scale, version 11 (BIS) (Patton et al., 1995). This questionnaire includes 30 questions that assess a participant's frequency of engaging in impulsive or non-impulsive activities and mental states. Assessment is on a four-point scale with the values (1) rarely/never, (2) occasionally, (3) often, and (4) almost always/always. The BIS includes six first order subscales: attentional, cognitive instability, motor, perseverance, self-control, and cognitive complexity. We took the sum over all 30 questions (after reversing scores for appropriate items) as a participant's impulsivity score. This is equivalent to taking the sum of the six first order subscale scores. Overall BIS scores can range from 30 (least impulsive) to 120 (most impulsive). BIS scores from 52 to 71 represent a normal range of impulsivity, with scores at or below 51 indicating a very controlled, non-impulsive individual and scores at or above 72 representing a highly-impulsive individual (Stanford et al., 2009).

#### 2.3. Task

We employed an emotional Go/NoGo task (see Donders, 1868/1969; Hester and Garavan, 2004), which presented emotional distractor pictures simultaneously with the Go and NoGo stimuli. In each trial, the participant was shown a square or circle, lasting 2 s, which served as the Go or NoGo stimulus (see **Figure 1**). The assignment of shape to trial type was counterbalanced across participants. Each Go or NoGo stimulus was superimposed on a task-irrelevant distractor image. Each distractor image was either emotionally neutral or aversive. Distractor images were taken from the International Affective Pictures System (IAPS; Lang et al., 2008). On Go trials, the participant had to press a button with their right index finger. On NoGo trials, the participant had to withold the button press response. To make the Go response more automatic (prepotent), Go and NoGo trials were presented at a 4:1 ratio. The task included four trial types: neutral Go, neutral NoGo, aversive Go, and aversive NoGo. Between trials, participants fixated a dot located at screen center (**Figure 1B**).

IAPS images were chosen as follows. IAPS images were screened by two child psychiatrists to be acceptable for use with our participant population and with adolescent psychiatric participants in a concurrent study (Brown et al., under review). From the screened images, aversive and neutral distractor pictures were selected based on the IAPS measures of valence and arousal from the normative sample reported in Lang et al. (2008). To maximize the effect of distractor valence, we used image selection criteria that created two non-overlapping clusters of images in two-dimensional arousal-valence space, one cluster for aversive distractors and one for neutral distractors (see Supplementary Figure 1). Specifically, we selected the 100 aversive IAPS images that had valence ratings ≤ 3.6 and were closest to [arousal, valence] target position [9, 1]. Position [9, 1] represents the most aversive (lowest valence rating), most arousing possible score. We selected the 104 neutral images with valence ratings > 3.6 and < 6.4 that were closest to [arousal, valence] target position [1, 5], which represents a neutral valence and the smallest possible arousal score. It would have been preferable to match distractor images for scene complexity, number of objects, and so on across the different trial types. Unfortunately, the IAPS set did not include enough images to permit such matching while also satisfying the above-described criteria, namely, screening by psychiatrists and separation into two non-overlapping clusters (as can be seen in Supplementary Figure 1). Aversive distractor pictures presented a variety of scenes including threatening animals, aggressive human faces, individuals wielding guns in a threatening manner, human injuries, surgical scenes, vehicle accidents, terrorism-related scenes, individuals vomiting, and dirty toilets including feces.

Brown et al. fMRI, risk behaviour, and impulsivity

Trials were presented in a rapid event-related design. Each Go or NoGo trial lasted one volume, i.e., 2 s. Inter-trial intervals were pseudo-randomized from the set {2, 4, 6 s}, distributed 30% 2 s, 40% 4 s, 30% 6 s with a mean of 4 s. Trial sequences and timings were derived using custom Python code to ensure linear independence of trial activations (see Burock et al., 1998). First-order counterbalancing of trial sequences was used to avoid first-order interaction effects between adjacent trials. To avoid interaction of BOLD non-linearity with inter-trial intervals and trial types, each of the four trial types was preceded in equal proportions by the 2, 4, and 6 s inter-trial intervals. Participants each completed four 330 s functional runs with a combined total of 204 trials including 84 neutral Go trials, 80 aversive Go trials, 20 neutral NoGo trials, and 20 aversive NoGo trials. The first trial of every run was always a neutral Go trial.

#### 2.4. IAPS Distractor Picture Ratings

After completing the fMRI scanning component of the study, participants rated the 204 IAPS distractor images used in the emotional Go/NoGo task for valence and arousal using the 9 point Likert scale (range 1–9) described in Lang et al. (2008). Valence ratings indicate participants' judgements of how pleasant or unpleasant a picture is (1: most unpleasant, 5: neutral, 9: most pleasant). Arousal ratings indicate how exciting or not exciting a picture is (1: not at all exciting, 5: neutral, 9: most exciting).

## 2.5. Analysis of Questionnaires, IAPS Ratings, and Task Performance

We computed a participant's overall CARE risk score as the mean of the six CARE subscales (see above and Fromme et al., 1997). To assess relative contributions of each subscale to the overall risk score, we did a separate correlation analysis of scores from each subscale vs. the overall CARE score. A participant's overall Barratt impulsivity score was computed as the sum across all questionnaire items (see above and Patton et al., 1995). We did a separate correlation analysis of scores from each of the six BIS first order subscales vs. the overall BIS score. Finally, we did a correlation analysis of overall CARE risk vs. BIS impulsivity scores. Twotailed t-tests were used to test whether each of the above correlations was significantly different from zero. Kolmogorov-Smirnov tests were used to check for non-normality in the distributions of participants' CARE and BIS scores.

Valence and arousal ratings for IAPS distractor pictures were analyzed with separate mixed effects ANOVAs in the R statistical language using the within-subject factors Response Inhibition (Go vs. NoGo) and Valence (aversive vs. neutral). We found differences in valence and arousal ratings for aversive vs. neutral distractor images but not between images used in Go vs. NoGo trials (see Section 3.2). A subsequent analysis examined relationships between valence and arousal ratings and CARE risk and BIS impulsivity scores. After collapsing across Go vs. NoGo trials, each participant's average rating for aversive distractor images and for neutral distractor images was computed, as was the difference between these two (aversive−neutral distractor ratings). We then computed the correlations of these differences vs. CARE risk scores and vs. BIS impulsivity scores. Two-tailed t-tests were used to test whether these correlations were significantly different from zero.

Behavioral data from task performance (error rates and latencies) were analyzed as follows. A Kolmogorov-Smirnov test was used to test for non-normality in the distribution of commission error rates for NoGo trials (collapsed across distractor valence). This test did not indicate a significant difference from normality (see Section 3.3). Commission error rates on NoGo trials (collapsed across distractor valence) were compared against zero with a one-tailed t-test. Kolmogorov-Smirnov tests for commission error rates for neutral NoGo trials and aversive NoGo trials considered as separate sets of data, not collapsed across neutral and aversive distractors—did indicate that the distributions were non-Gaussian (see Section 3.3). Therefore, error rates for NoGo trials with neutral vs. aversive distractors were compared using permutation testing. Omission errors on Go trials were very rare with 17 of 19 participants making no such errors and the other two participants making very few commission errors (4.3% and 0.6%, see Section 3.3). Therefore, we simply reported the Go trial omission error rates without doing statistical comparisons. Latencies for Go trials with neutral vs. aversive distractor pictures were compared using a two-tailed t-test. Kolmogorov-Smirnov tests were used to test normality in the distributions of neutral Go and aversive Go trial latencies. These tests did not indicate significant deviation from normality (see Section 3.3). We tested for relationships between NoGo trial error rates (collapsing across neutral vs. aversive distractors) and CARE risk scores, as well as BIS impulsivity scores, using separate linear regression models and associated two-tailed t-tests. Similarly, we tested Go trial latencies (collapsing across neutral vs. aversive distractors) against CARE and BIS scores using separate linear regression models and associated two-tailed t-tests.

## 2.6. MRI Scanning

Magnetic resonance imaging was done on the 4.7 Tesla Varian Inova scanner at the Peter S. Allen MR Research Center at the University of Alberta. We acquired blood oxygenation level dependent (BOLD) fMRI images with a T2∗ -weighted echo planar imaging sequence using these parameters: volume time 2.0 s, single shot, repeat time 2.0 s, echo time 19.0 ms, 3.0 mm isotropic voxels, 80 <sup>×</sup> 80 matrix, 240 <sup>×</sup> 240 mm<sup>2</sup> field of view, 3.0 mm slice thickness, 36 axial slices, 108 mm through-plane coverage, interleaved slice collection order. We used 80% partial k-space in the phase encode direction (anterior-posterior). The fMRI scanning volume covered the entire cerebral cortex except for the ventral-posterior tip of occipital cortex in participants with larger heads. A high resolution T1-weighted structural scan was also acquired for each participant. This scan utilized a magnetization-prepared rapid acquisition gradient echo (MPRAGE) sequence with parameters: TR 9.4 ms, inversion time 300.0 ms, relaxation delay time (after readout prior to inversion) 300.0 ms, linear phase encoding, TE 3.7 ms, matrix 240 × 192 × 128, field of view 240 <sup>×</sup> <sup>192</sup> <sup>×</sup> 192 mm<sup>3</sup> , 1.0 <sup>×</sup> 1.0 <sup>×</sup> 1.5 mm<sup>3</sup> voxels, whole brain coverage.

### 2.7. fMRI Analysis

SPM8 and in-house MATLAB code were used for preprocessing of fMRI data. The preprocessing steps for each participant included: (1) 6 parameter rigid body motion correction of fMRI volumes in SPM8, (2) coregistration of fMRI data to MPRAGE anatomical scan in SPM8, (3) non-linear spatial warping (estimation and interpolation) of MPRAGE anatomical volume to MNI T1 template space at 1 × 1 × 1 mm resolution in SPM8, (4) interpolation of fMRI volumes into the T1 template space at 3 × 3 × 3 mm spatial resolution using warping parameters from step (3), (5) 8 mm full width at half maximum (FWHM) Gaussian spatial smoothing of fMRI volumes in SPM8.

Statistical modeling of fMRI data was done in two steps using custom-built MATLAB code. We first performed separate first-level, within-subjects general linear model (GLM) analyses on each participant. Within-subjects results were then combined using two different between-subjects mixed-effects analyses (Worsley et al., 2002). Each within-subject GLM included either four or five sets of finite impulse response (FIR) predictors, one set for each of the four trial types (neutral Go, aversive Go, neutral NoGo, aversive NoGo) and, for participants who made errors, one set of predictors for error trials (collapsed across trial types). Error trials were rare (0–10% of trials, per subject). The GLM included 10 FIR impulse predictors (corresponding to 10 functional volumes) per trial type. The FIR predictors represented deconvolved activation timecourses for the different trial types (see Serences, 2004). The GLM also included a set of nuisance predictors for each run consisting of constant run offset, linear drift, cosine, and sine with period equal to twice the run length, 6 rigid body motion parameters, and 6 impulses at the start of each run for spin saturation. We used a manually-constructed mask that excluded voxels outside the brain. The mask included 79,044 voxels (size 3 × 3 × 3 mm) inside the brain. Each within-subject GLM was fit to the data using weighted least squares that corrected for autocorrelated noise. Specifically, 10 autocorrelation coefficients (lags of 1–10 volume times) were computed for each functional slice across the whole brain using the residuals from a non-corrected initial GLM fit. Then, the design matrix and each voxel's timecourse were pre-whitened, and auto-correlation-corrected beta weights were computed as described in Burock and Dale (2000) and Worsley et al. (2002). For each subject separately, we computed three first-level (within-subjects) statistical contrast maps (two-tailed t statistic maps) from the GLM beta weights. The response inhibition contrast was (aversive NoGo + neutral NoGo) − (aversive Go + neutral Go). The emotional valence contrast was (aversive NoGo + aversive Go) − (neutral NoGo + neutral Go). The emotional response inhibition contrast was (aversive NoGo − aversive Go). Contrasts were computed from the FIR beta weights representing activation across the 3rd and 4th time points of the FIR deconvolved timecourses. The 3rd and 4th time points, which correspond to 4 and 6 s from trial start, were chosen a priori based on the typical BOLD hemodynamic peak time around 4–6 s (Aguirre et al., 1998).

For each of the three first-level statistical contrasts, we performed second-level analyses combining results across participants and testing for significant relationships between first level contrast values and CARE risk scores and/or BIS impulsivity scores. One goal was to identify brain regions that showed a relationship between a first level contrast and one of the questionnaires (either CARE or BIS scores) but not the other one. One incorrect approach would have been to do second level regressions against a given questionnaire (e.g., CARE scores), identify significant regions of interest (ROIs), and perform follow-up F-tests comparing regression against CARE vs. BIS scores on each ROI. This approach would have created dangers from double-dipping (Kriegeskorte et al., 2009; Vul et al., 2009). Instead, we used the approach described below.

For each of the three first-level statistical contrasts, we performed three second-level, mixed-effects analyses combining results across participants. The first analysis tested for significant relationships with CARE scores where CARE scores accounted for significantly more variance in first level contrast values than BIS scores (CARE > BIS), assessed using F-tests. The second analysis tested for significant relationships with BIS scores where BIS scores accounted for significantly more variance than CARE scores (BIS > CARE), assessed using F-tests. The third analysis tested for significant relationships with both CARE and BIS scores (conjunction analysis). Specific computational details for the second-level analyses are provided in Supplementary Methods Section 1.1. In total, there were nine second-level statistical maps:


Statistical t-maps were thresholded voxelwise at p < 0.05. A cluster mass threshold of 465 was also applied to each map to correct for multiple comparisons at p < 0.05 across the voxel population as well as the nine second-level statistical comparisons. Cluster mass is the sum of absolute tvalues from all voxels in the cluster. The cluster mass threshold was determined using Monte Carlo simulation based on the method of AlphaSim (Ward, 2000), modified to account for comparisons across all nine second-level maps. Statistical results were visualized using MATLAB and EasyFMRI (www.easyfmri.com).

We did follow-up quality assurance analyses on the nine second-level maps described above. None of the three conjunction maps (Maps 3, 6, and 9) revealed any significant regions surviving multiple comparison correction. For the other secondlevel maps, each of which did reveal one or more significant voxel clusters, an automated algorithm was used to grow a cluster of voxels around each positive or negative statistical peak (local extremum) in the associated t-map (after filtering via an Fmap, if appropriate, see Supplementary Methods Section 1.1). In some cases, two or more of the resulting clusters fell within the same anatomical structure. In these cases, we combined those clusters. There were 36 such clusters in total from Maps 1, 2, 4, 5, 7, and 8. For a given cluster, each participant's mean BOLD signal was computed by averaging across all voxels in the cluster. First-level GLM analyses were then conducted on the average timecourses, followed by second-level regression against CARE or BIS scores. Event-related activation timecourses for each of the four trial types were derived from the first-level finite impulse response models and averaged across participants. We discarded two clusters whose activation timecourses were severely dissimilar to the expected difference of gammas hemodynamic response function shape (see Huettel et al., 2008, ch. 7), as determined by visual inspection. In some clusters, statistical significance in the second-level analyses was dependent on one or two outlier participants. We re-ran all second-level analyses excluding the two most extreme participants, either the participants with the smallest and largest fMRI contrast values for the given cluster or the participants with the smallest and largest CARE scores or BIS scores, as appropriate. Clusters that failed to reach significance without these two participants were discarded. 23 of the 36 clusters were discarded in this way. We discarded a total of 25 clusters based on the above quality assurance criteria, and we present results only for the 11 remaining clusters.

For exploratory purposes, we computed correlations between significant regions identified in Maps 1–9 and participant scores on the CARE and BIS subscales (see Supplementary Methods Section 1.2 for details).

## 3. Results

#### 3.1. Risk and Impulsivity Scores

Participant CARE risk scores had a mean of 2.69 ± 0.58 and ranged from 1.66 to 4.06. That is, the participants in this study exhibited low to medium risk tendencies based on the CARE questionnaire, with no participants exhibiting very high-risk tendencies. The distribution of CARE scores did not differ significantly from the Gaussian (p = 0.90, Kolmogorov-Smirnov test). Differences in participants' CARE risk scores were driven primarily by differences in the heavy drinking CARE subscale, as well as by differences in drug use, aggression, and academic/work subscales (see **Table 1**).

Participants' Barratt impulsivity scale (BIS) scores had a mean of 58.11 ± 8.14 with a range of 43–71. That is, participants ranged from non-impulsive to moderately-impulsive. No participants fell in the highly-impulsive range based on Stanford et al. (2009)'s criterion (BIS score ≥ 72). The distribution of BIS scores did not differ significantly from the Gaussian (p = 0.39, Kolmogorov-Smirnov test). Differences in overall BIS scores were driven most strongly by differences in the BIS self-control, cognitive instability, and attentional subscales, while differences in the motor and perseverance subscales also contributed (see **Table 2**).

There was a partial linear relationship between participant BIS and CARE scores (**Figure 2**). The correlation coefficient between BIS and CARE scores was 0.50, which was significantly different


R denotes correlation coefficient comparing subscore against CARE score (mean of subscores). P indicates p-value from t-test comparing correlation against zero with df = 17.

#### TABLE 2 | Summary of BIS scores and subscale scores.


R denotes correlation coefficient comparing subscore against overall BIS score (sum of 1st order subscores). P indicates p-value from t-test comparing correlation against zero with df = 17.

from zero (p = 0.030, t = 2.36, df = 17). BIS scores explained only 20.3% of the variance in the CARE scores.

#### 3.2. Distractor Picture Ratings

Participants rated IAPS distractor pictures for valence and arousal on a 9-point Likert scale (see Section 2.4). Mean valence rating for aversive distractors was 3.18 ± 0.64 (mean ± std). Mean valence rating for neutral distractors was 5.32 ± 0.21. Valence ratings were significantly lower for aversive distractors (mixed effects ANOVA, <sup>p</sup> <sup>&</sup>lt; <sup>1</sup>.<sup>0</sup> <sup>×</sup> <sup>10</sup>−<sup>10</sup> , F = 175.3, df = 1, 18). There were no significant differences in valence scores for distractors on Go vs. NoGo trials (p = 0.58, F = 0.31, df =

variance in the CARE scores. The largest statistical influence (Cook's distance

D) across all data point was 0.487.

FIGURE 3 | Left: Participants' mean valence ratings for neutral and

1, 18), nor was there a significant interaction effect of Go vs. NoGo × aversive vs. neutral (p = 0.68, F = 0.18, df = 1, 18). Arousal ratings were significantly higher for aversive distractor trials (4.94 ± 1.19) compared to neutral distractor trials (2.10 ± 0.92). The main effect of aversive vs. neutral distractors on arousal ratings was significant (mixed effects ANOVA, <sup>p</sup> <sup>=</sup> <sup>1</sup>.<sup>3</sup> <sup>×</sup> <sup>10</sup>−<sup>9</sup> , F = 128.3, df = 1, 18). The main effect on arousal for Go vs. NoGo trials was not significant (p = 0.90, F = 0.016, df = 1, 18), nor was the interaction effect significant (p = 0.063, F = 3.92, df = 1, 18).

We did separate correlation analyses of CARE risk scores and Barratt impulsivity scores against the difference between ratings for aversive vs. neutral pictures. (See Section 2.5 for analysis details.) The correlation between CARE risk scores and the difference in valence scores (aversive−neutral pictures) was not significantly different from zero (r = −0.40, p = 0.090, t = −1.80, df = 17), nor was the correlation between CARE risk scores and the difference in arousal scores (r = 0.35, p = 0.14, t = 1.54, df = 17). The correlation between BIS impulsivity scores and the difference in valence scores was significantly different from zero (r = −0.67, p = 0.002, t = −3.76, df = 17). More impulsive participants rated the aversive pictures as more unpleasant (lower valence score) as shown in **Figure 3**. The correlation between BIS impulsivity scores and the difference in arousal scores was not significantly different from zero (r = 0.32, p = 0.18, t = 1.40, df = 17).

#### 3.3. Task Performance

Participants made commission errors on NoGo trials (collapsed across distractor valence) at a mean rate of 3.7 ± 3.8%, which was low but significantly above zero (p = 0.00022, t = 4.3, df = 18, one-tailed t-test). The distribution of participants' error rates for NoGo trials (collapsed across distractor valence) was not significantly different from the Gaussian (p = 0.35, Kolmogorov-Smirnov test). Without collapsing across distractor valence, the distribution of neutral NoGo trial error rates approached significant difference from the Gaussian (p = 0.057,

fit linear regression line. More impulsive participants rated the aversive pictures as more unpleasant (lower valence score). The correlation was significant (r = −0.67, p = 0.002, t = −3.76, df = 17).

Kolmogorov-Smirnov test), and aversive NoGo trial error rates were significantly non-Gaussian (p = 0.039, Kolmogorov-Smirnov test). Error rates did not differ significantly between NoGo trials with aversive vs. neutral distractors (p = 0.56, permutation test). Seventeen of nineteen participants made no omission errors on Go trials, while the other two participants had low omission error rates of 4.3% and 0.6%. Go trial latencies were 616 ± 135 ms with neutral distractors and 631 ± 148 ms with aversive distractors. The distributions of neutral Go and aversive Go trial latencies did not differ significantly from the Gaussian (respectively, p = 0.95 and p = 0.98, Kolmogorov-Smirnov tests). The difference between neutral Go and aversive Go trial latencies was significant (p = 0.033, t = 2.3, df = 18, twotailed t-test). NoGo error rates and Go trial latencies did not show any significant relationships with either CARE risk scores or Barratt impulsivity scores on linear regression tests (p > 0.5, |t| < 0.69, df = 17). Similarly, Horn et al. (2003) and Asahi et al. (2004) did not find significant relationships between participant impulsivity scores and Go/NoGo task performance.

## 3.4. fMRI Results Independent of Risk and Impulsivity Scores

Brown et al. (2012) previously presented an analysis of fMRI activation related to response inhibition and distractor picture valence in the fMRI dataset used in the current study. Here, we excluded one participant from Brown et al. (2012)'s analysis, as this person did not complete the BIS. This exclusion did not significantly change the results as presented in Brown et al. (2012). Briefly, in the response inhibition contrast, we found significantly larger Go vs. NoGo activation in left motor cortex and other regions and larger NoGo vs. Go activation in ventrolateral prefrontal cortex as well as other cortical regions (see Supplementary Figure 2). In the emotional valence contrast, we found greater activation for aversive vs. neutral distractor pictures in orbitofrontal cortex, lateral prefrontal cortex, insula, the amygdala and surrounding cortex, anterior cingulate cortex, medial prefrontal cortex, and bilateral posterior middle temporal gyrus and angular gyrus (see Supplementary Figure 3) (See Brown et al., 2012 for further details). The emotional response inhibition contrast (aversive NoGo−aversive Go), also revealed large regions of significant difference in all major lobes of the brain, including regions in dlPFC, vlPFC, right anterior insula, and right OFC showing greater activation for aversive NoGo trials (see Supplementary Figure 4). There was a right side laterality in that the right vlPFC and right dlPFC clusters were larger than the left ones.

### 3.5. fMRI Response Inhibition Contrast vs. Risk and Impulsivity Scores

We examined relationships between the response inhibition contrast (NoGo−Go) and CARE risk scores and between the inhibition contrast and BIS impulsivity scores (see Section 2.7 for methodological details). Statistical Map 1 tested regression of response inhibition contrast against CARE scores where CARE scores also accounted for significantly more variance than BIS scores (see Section 2.7). Map 1 revealed significant inverse relationships (p < 0.05, corrected) in a large cluster with two foci, one in right orbitofrontal cortex (OFC) and the other in ventromedial prefrontal cortex (vmPFC) (See **Figure 4** and **Table 3**). As required by Map 1's inclusion criteria, CARE scores accounted for significantly more variance in the inhibition contrast values than did BIS scores in these regions (also see **Table 3**). These regions did not show a significant relationship between inhibition contrast and BIS scores (also see **Table 3**). The right OFC region included voxels in the medial orbital gyrus, rostral and caudal parts of the medial orbital sulcus, and medial portions of the anterior and posterior orbital gyri (based on Chiavaras and Petrides (2000)'s description of orbitofrontal anatomy). The vmPFC cluster was bilateral, centered supero-inferiorly on the suborbital sulcus, and included voxels in the adjacent ventromedial part of the medial aspect of the superior frontal gyrus as well as the medial aspect of the gyrus rectus. In the vmPFC region, the four trial types evoked deactivation of the BOLD signal (**Figure 4** third row, middle panel). Specifically, in this region, participants with lower CARE risk scores exhibited larger negative BOLD deflections for Go vs. NoGo trials (resulting in positive NoGo-Go contrast values), while higher CARE score participants exhibited larger negative BOLD deflections for NoGo vs. Go trials (resulting in negative NoGo-Go contrast values). Also see scatterplot, middle of bottom row, **Figure 4**. Map 1 indicated that there was also a significant positive relationship between response inhibition contrast and CARE scores in right occipital cortex (**Figure 4**, **Table 3**). Measures of influence (Cook's distance) were below 1 for all participants for all of the above regions (see Supplementary Table 1). Correlation analyses revealed significant relationships, in these regions, between response inhibition contrast and several CARE subscales (see Supplementary Table 2).

The statistical comparison of response inhibition contrast vs. BIS scores (Map 2, see Section 2.7) did not reveal any significant regions after correction for multiple comparisons and quality assurance exclusions. The conjunction analysis of regressions of response inhibition contrast against CARE and against BIS scores (Map 3, see Section 2.7) revealed no significant clusters surviving multiple comparison correction.

### 3.6. fMRI Emotional Valence Contrast vs. Risk and Impulsivity Scores

We examined relationships between the emotional valence contrast (aversive−neutral distractor pictures) vs. CARE risk scores and vs. BIS impulsivity scores. See Section 2.7 for methodological details.

Map 4 (see Section 2.7) revealed a positive relationship between emotional valence contrast amplitude and participants' CARE scores in right occipital cortex and dorsomedial cerebellum (**Table 4**). As required by Map 4's inclusion criteria, CARE scores accounted for significantly more variance in emotion contrast amplitudes compared to BIS scores (also see **Table 4**) in these regions. These regions did not exhibit significant relationships with BIS scores (also see **Table 4**). Measures of influence (Cook's distance) were below 1 for all participants for all of the above regions (see Supplementary Table 1). Correlation analyses revealed significant relationships between emotional valence contrast and several CARE subscales in these regions (see Supplementarypreviously presented an analysis of fMRI Table 3).

Map 5 (see Section 2.7) revealed a significant inverse relationship between emotional valence contrast amplitude and BIS impulsivity scores in a large cluster in dorsomedial prefrontal

Axial images' left side corresponds to left side of brain. OFC: orbitofrontal

cortex (dmPFC) (See **Figure 5**, **Table 4**). This region included a large portion of the dorsomedial aspect of the superior frontal gyrus as well as adjacent anterior cingulate sulcus. There was a

0.760, right occipital 0.247 (see Supplementary Table 1).

#### TABLE 3 | Summary of significant regions from Map 1.


Summary of significant regions from Map 1: comparison of fMRI response inhibition contrast (NoGo—Go) vs. CARE risk scores where CARE scores also accounted for significantly more variance than BIS scores. Statistical comparison of inhibition contrast vs. BIS impulsivity scores (Map 2) did not yield any significant regions. X, Y, Z: MNI coordinates of region's peak statistical voxel. P- and t-values are median values across all voxels in a region (df = 98). Positive and negative t-values indicate, respectively, greater and lesser inhibition contrast values for participants with larger CARE scores. F-test CARE > BIS are median values across each region from the F-map testing whether CARE scores accounted for significantly more variance in response inhibition contrast values than BIS scores (df = 1, 97). See Section 2.7 for analysis details. OFC: orbitofrontal cortex. vmPFC: bilateral ventromedial prefrontal cortex. Occipital: occipital cortex.


Summary of significant regions from Maps 4 and 5. Map 4 tested comparison of fMRI emotional valence contrast (aversive - neutral distractors) vs. CARE scores where CARE scores also accounted for more variance than BIS scores. Map 5 tested regression of emotional valence contrast vs. BIS scores where BIS scores also accounted for more variance than CARE scores. X, Y, Z: MNI coordinates of region's peak statistical voxel. P- and t-values are median values across all voxels in each region (df = 98). F-test CARE > BIS values are median values across each region from the F-map testing whether CARE scores accounted for significantly more variance in emotional valence contrast values than BIS scores (df = 1, 97). F-test BIS > CARE shows median values from the F-map testing whether BIS scores accounted for significantly more variance in emotional valence contrast values than CARE scores (df = 1, 97). See Section 2.7 for details of analysis. Occipital: occipital cortex. dmCereb: dorsomedial cerebellum. dmPFC: dorsomedial prefrontal cortex. pgACC: perigenual anterior cingulate cortex. pOFC: posterior orbitofrontal cortex. Temp Pole: temporal pole.

similar relationship in voxel clusters in perigenual anterior cingulate cortex (pgACC), right posterior orbitofrontal cortex (pOFC), and right temporal pole (**Figure 5**, **Table 4**). The pgACC region was bilateral, centered antero-posteriorly on the anterior cingulate sulcus rostral to the genu of the corpus callosum, and extended into the adjacent anterior cingulate gyrus and slightly into the posterior part of the medial aspect of the superior frontal gyrus. pgACC exhibited BOLD deactivation in response to the four trial types (**Figure 5** second row, middle panel). In this region, participants with low BIS impulsivity scores exhibited larger negative BOLD deflections for neutral distractor trials, whereas participants with higher BIS scores exhibited greater negative BOLD deflections for aversive distractor trials (see scatterplot, middle of bottom row, **Figure 5**). The pOFC region was located in the posterior orbital gyrus. This region overlapped partially with the most posterior part of the OFC region from Map 1 (response inhibition contrast vs. CARE scores). The dmPFC

and pgACC regions did not overlap with the vmPFC region from Map 1. BIS scores accounted for significantly more variance in the emotional valence contrast values compared to CARE scores in all of these regions, as required by Map 5's inclusion criteria (also see **Table 4**). Map 4 (regression of emotional valence contrast vs. CARE scores) did not include significant clusters in these regions. In terms of regression of emotional valence contrast vs. CARE scores, the median across voxels in the right pOFC region from Map 5 did exhibit a significant inverse relationship between emotional valence contrast values and CARE scores (p = 0.047, **Table 4**), but this region did not survive multiple comparison correction (cluster mass thresholding, see Section 2.7) in the t-map calculation for Map 4. Measures of influence (Cook's distance) were below 1 for all participants for all of the above regions (see Supplementary Table 1). Correlation analyses revealed significant relationships between emotional valence contrast and several BIS subscales in these regions (see Supplementary Table 4).

Map 6 (conjunction of t-maps for emotional valence contrast vs. CARE scores and vs. BIS scores) did not reveal any significant regions surviving correction for multiple comparisons.

## 3.7. fMRI Emotional Response Inhibition Contrast vs. Risk and Impulsivity Scores

We examined relationships between the emotional response inhibition contrast (aversive NoGo−aversive Go trials) vs. CARE risk scores and vs. BIS impulsivity scores. See Section 2.7 for methodological details. Map 7 (see Section 2.7) revealed a region in right occipital cortex showing a significant positive relationship between emotional response inhibition and CARE scores (see **Table 5**). Map 8 (see Section 2.7) showed a left occipital region exhibiting a significant positive relationship between emotional response inhibition and BIS scores (see **Table 5**). Measures of influence (Cook's distance) were below 1 for all participants for all of the above regions (see Supplementary Table 1). Correlation analyses revealed significant relationships between the emotional response inhibition contrast and several CARE subscales and BIS subscales in these regions (see Supplementary Table 5). Map 9 (conjunction of t-maps for emotional response inhibition contrast vs. CARE scores and vs. BIS scores) revealed no significant regions surviving correction for multiple comparisons.

#### TABLE 5 | Summary of significant regions from Maps 7 and 8.


Summary of significant regions from Maps 7 and 8. Map 7 tested comparison of fMRI emotional response inhibition contrast (aversive NoGo−aversive Go) vs. CARE scores where CARE scores also accounted for more variance than BIS scores. Map 8 tested regression of emotional response inhibition contrast vs. BIS scores where BIS scores also accounted for more variance than CARE scores. X, Y, Z: MNI coordinates of region's peak statistical voxel. P- and t-values are median values across all voxels in each region (df = 98). F-test CARE > BIS values are median values across each region from the F-map testing whether CARE scores accounted for significantly more variance in emotional response inhibition contrast values than BIS scores (df = 1, 97). F-test BIS > CARE shows median values from the F-map testing whether BIS scores accounted for significantly more variance in emotional response inhibition contrast values than CARE scores (df = 1, 97). See Section 2.7 for details of analysis.

#### 3.8. Outliers

As described at the end of Section 2.7, we excluded 23 regions from Maps 1, 2, 4, 5, 7, and 8 which did not retain significance after excluding outlier participants (two participants with smallest and largest CARE or BIS scores, or two participants with smallest or largest first-level fMRI contrast values). All retained regions exhibited maximum influence measures (maximum Cook's distance across all 19 participants) less than 1 (see Supplementary Table 1).

Three of the 19 participants were not right-handed (one lefthanded, two ambidextrous). None of these three participants had the lowest or highest CARE or BIS scores. Their CARE scores were 2.82, 2.92, and 2.91, compared to the CARE score range of 1.66–4.06 for all 19 participants. Their BIS scores were 57, 68, and 50, compared to the BIS score range of 43–71. For the dorsomedial cerebellum region identified in Map 4 (see **Table 4**), an ambidextrous participant exhibited the highest value. For the dmPFC and pgACC regions identified in Map 5 (see **Table 4**), two non-right-handed participants exhibited the lowest emotional valence contrast values. Otherwise, the participants with the lowest and highest first-level contrast values were right-handed. Nonetheless, all regions presented above retained significance with outlier participants removed, as previously discussed. Based on visual inspection of first-level fMRI contrast values from the significant regions presented for Maps 1–9, the three nonright-handed participants did not display any consistent trend toward deviating from the other 16 right-handed participants in terms of fMRI contrast values. With only three non-righthanded participants, there was not enough statistical power to do a proper statistical comparison of right-handed vs. non-righthanded participants. We do not think that the inclusion of the three non-right-handed participants skewed the results presented here.

## 4. Discussion

Many studies have emphasized the role of impulsivity as a potential contributor to high-risk behavior tendencies (Levitt, 1991; Moore and Rosenthal, 1993; Luengo et al., 1994; Stanford et al., 1996; Ernst et al., 2006; Casey et al., 2008; Ernst and Mueller, 2008; Ernst and Fudge, 2009; Romer et al., 2009; Romer, 2010; Casey et al., 2010a; Dalley et al., 2011; Blakemore and Robbins, 2012). Accordingly, we expected to observe substantial similarities in how fMRI brain activation patterns from the emotional Go/NoGo task related to impulsivity and to risk behavior tendencies. Contrary to expectations, we found a dissociation between impulsivity and risk behavior tendencies in terms of fMRI activations. All but one of the regions detected in the statistical analyses reported in Sections 3.5 and 3.6 exhibited a significant relationship between fMRI first-level contrast amplitudes and either CARE risk scores or BIS impulsivity scores, but not both (see **Tables 3**, **4**). F-tests comparing amounts of variance in fMRI contrast amplitude explained by CARE or BIS scores also supported this dissociation (see **Tables 3**, **4**). The sole partial exception was one small cluster of voxels in right pOFC which showed an inverse relationship between emotional valence contrast and both CARE and BIS scores (see **Table 4**) although the relationship with CARE scores did not survive multiple comparison correction. Our results support the proposition that impulsivity and high-risk behavior tendencies are distinct (but related) constructs. Furthermore, a high impulsivity level is not equivalent to an elevated highrisk behavior tendency. We suggest that impulsivity may contribute to high-risk behavior in some cases but that greater impulsivity does not necessarily contribute to higher risk behavior tendencies.

High-risk behavior is complex, and it is acknowledged that various factors other than impulsivity are important potential contributors to risk tendency, such as reward seeking and sensation seeking (see Whiteside and Lynam, 2001; Romer et al., 2009; Romer, 2010; Dalley et al., 2011; Blakemore and Robbins, 2012). Relatively little emphasis has been placed on possible dissociations between high-risk behavior tendencies and impulsivity profiles. We are aware of one study that reported a dissociation between risk behavior tendencies as assessed using the Balloon Analog Risk Task and BIS impulsivity in a population of cigarette smokers (Ryan et al., 2013). Our observed dissociation is also consistent with studies that propose contributing factors to highrisk behavior other than impulsivity, such as reward seeking and sensation seeking (see Romer et al., 2009; Romer, 2010) as well as the influence of peers and social cues on behavior, particularly in adolescents (Gardner and Steinberg, 2005; Blakemore and Robbins, 2012).

The current study focused on impulsivity as measured by the BIS. Impulsivity is a complex construct; Bari and Robbins (2013) have suggested that impulsivity may involve multiple subdivisions of cognitive processes with as many as 9 distinct components (also see Whiteside and Lynam, 2001). Aspects of impulsivity not captured by the BIS Scale may show different relationships with fMRI activity patterns.

## 4.1. BIS Impulsivity vs. Aversive Distractor Valence Ratings

Participants with higher BIS impulsivity scores rated aversive distractor images as being more unpleasant (lower valence scores, see **Figure 3**). This may reflect a greater sensitivity to aversive stimuli in more impulsive participants, commensurate with a possible contribution of negative urgency to higher impulsivity (see Whiteside and Lynam, 2001; Cyders and Smith, 2008).

## 4.2. Response Inhibition Activity vs. Risk Tendency and Impulsivity

In right OFC and a region in vmPFC, participants with higher CARE risk scores exhibited lower response inhibition (NoGo−Go) contrast amplitude. These regions are not traditionally associated with motor response inhibition but rather with representing reward and value (see Mitchell, 2011). It is possible that, in participants with lower risk tendencies, successful completion of NoGo trials generated larger reward responses in these regions. Interestingly, our results in vmPFC and OFC are commensurate with findings in BPD. Diminished recruitment of BOLD activation during impulse control tasks has been reported in OFC and medial prefrontal regions in patients with BPD, a condition which is also associated with elevated high-risk behavior tendencies (Krause-Utz et al., 2014; Sebastian et al., 2014). We did find greater response inhibition-related activity for higher risk score individuals in right occipital cortex. It is possible that our emotional Go/NoGo task evoked greater attention responses in this region in those participants.

vlPFC has an established role in response inhibition (see Aron et al., 2004a, 2007; Dolcos et al., 2011; Mitchell, 2011). It has also been suggested that high-risk behavior tendencies might be caused by impulsivity as a result of reduced prefrontal behavioral control (see Ernst et al., 2006; Casey et al., 2008; Ernst and Mueller, 2008; Ernst and Fudge, 2009; Casey et al., 2010b). For these reasons, we expected to observe risk-related differences and/or impulsivity-related differences in response inhibition activation in vlPFC, but we did not observe this. In addition, Asahi et al. (2004) found that NoGo response inhibition-related activity was inversely correlated with individual impulsivity in a region they called right dlPFC. This region was actually located in the right inferior frontal sulcus on the border between right vlPFC and dlPFC. We expected to replicate this finding but did not. Horn et al. (2003) also previously found a relationship between reduced inhibition activation in the Go/NoGo task and increased participant impulsivity, in dmPFC. We did not replicate this finding, though we did find reduced emotional valence contrast amplitude in more impulsive participants in dmPFC as discussed below. Asahi et al. (2004) and Horn et al. (2003) both used a blocked design comparing blocks of pure Go trials with blocks of 50:50 mixed Go and NoGo trials. We used a rapid event-related design with an 80:20 ratio of Go:NoGo trials. These task design differences could account for differences between our results and those of Asahi et al. (2004) and Horn et al. (2003).

Conditions specific to poor impulse control in adolescents have also been examined in Go/NoGo fMRI studies (without a paired emotional task component). Risk behavior specific to impulsivity has been measured in drug naive adolescents diagnosed with ADHD (Rubia et al., 2005b), confirming a relationship between behavioral impulsiveness scores on screening questionnaires specific to the untreated disorder and reduced activation related to response inhibition in right vlPFC during the stop signal task. Adolescents at high-risk for developing Alcohol Use Disorder who later transitioned into heavy drinking adults also showed similarities between risk-based behavior and reduced brain activation related to response inhibition in various brain regions including vlPFC (Norman et al., 2011). Impulsive drinking behavior measured in heavy vs. light drinking college students has also indicated that those students who were prone to heavier episodes of drinking did show reduced fMRI activation during NoGo trials as compared to the students who did not drink as much alcohol in various brain regions including dlPFC (Ahmadi et al., 2013). In contrast, we did not observe any relationship between response inhibition-related activation in vlPFC or dlPFC and either CARE risk scores or Barratt impulsivity scores. Our participants exhibited low to medium CARE risk scores and low to medium Barratt impulsivity scores (see Section 3.1). It would be interesting to investigate whether including participants from the highest end of the risk and impulsivity spectra may reveal relationships between fMRI activation from the emotional Go/NoGo task and CARE or BIS scores in these regions.

## 4.3. Emotional Processing and Regulation vs. Risk Tendency and Impulsivity

We observed reduced emotion-related activity (aversive−neutral distractor picture activation difference) in more impulsive participants in dmPFC, pgACC, and right pOFC. These regions have various roles in emotion-related processing and regulation as well as emotion-based response processing. Dorsomedial PFC (dmPFC) is involved in a variety of regulatory functions, recruited by various cognitive tasks. [Note: Some authors include dorsal ACC in dmPFC (e.g., Mitchell, 2011), whereas we take dmPFC to include only the medial aspect of the superior frontal gyrus excluding ACC.] dmPFC is thought to have roles in flexible behavior control, including resolution of response conflict and outcome value-related aspects of decision making (Venkatraman et al., 2009) and in emotion regulation (see Ochsner and Gross, 2005; Mitchell, 2011). Anterior cingulate cortex (ACC), inferiorly adjacent to dmPFC, has been implicated in performance monitoring, error detection, and process conflict monitoring (Carter et al., 1998; Botvinick et al., 2004; Botvinick, 2007; Mitchell, 2011). In particular, Carter et al. (1998) suggest that ACC may detect conditions that make errors more likely, such as the presence of emotional distractors in our emotional Go/NoGo task. dmPFC may also contribute to these functions (Stuss et al., 2001; Stemmer et al., 2004; Lvstad et al., 2012). pgACC is thought to be involved in processing salience related to emotion and motivation and is implicated in response preparation (Schulz et al., 2011). This region is also thought to provide regulatory control over emotion response circuitry in the amygdala and nucleus accumbens as well as autonomic functions in the hypothalamus and brainstem (Quirk and Gehlert, 2003; Ochsner and Gross, 2005; Mitchell, 2011, see). OFC is known to be involved in representing emotional stimulus value as well as learning and flexible control of emotions and of behavior in emotional contexts (Ochsner et al., 2002; O'Doherty, 2003; Ochsner et al., 2004; O'Doherty, 2004; Murray et al., 2007; O'Doherty, 2007; Wallis, 2007; Rolls and Grabenhorst, 2008; Dolcos et al., 2011; Mitchell, 2011; Golkar et al., 2012). Cyders et al. (2014b) found a relationship between increased fMRI activation for aversive stimulus processing in right lateral OFC, negative urgency (a component of impulsivity and subscale in the UPPS conception of impulsivity, see Whiteside and Lynam, 2001; Cyders and Smith, 2008), and general risk-taking. Negative urgency has also been associated with fMRI responses to alcohol-related cues in a ventromedial PFC region (Cyders et al., 2014a). Patients with BPD, which is associated with emotional dysregulation, show reduced fMRI activation in OFC, ACC, dmPFC, and dlPFC in response to emotion regulation tasks (Krause-Utz et al., 2014; Sebastian et al., 2014).

We interpret our findings in dmPFC, pgACC, and right pOFC as reflecting reduced recruitment of emotion regulation in these regions in more impulsive individuals. It is possible that reduced emotion regulation processing in all three regions could lead to increased impulsivity in a straight-forward manner. Participants with higher BIS impulsivity scores rated aversive distractor pictures as being more unpleasant (see **Figure 3**). Therefore, it seems unlikely that the reduced emotional valence contrast in dmPFC, pgACC, and right pOFC for higher BIS score participants could reflect a reduction in emotional stimulus salience in participants with higher impulsivity.

We observed greater aversive emotional valence contrast activation in right occipital cortex and dorsomedial cerebellum in participants with greater risk behavior tendencies. The metaanalysis by Kober et al. (2008) suggests that limbic projections enhance visual stream activity in the presence of emotional stimuli. Combined with changes in eye movement patterns, this results in changes to visual stream activity patterns. Kober et al. (2008) also point out that cerebellar efferents are more active during emotional states, possibly as the situational context portion of a larger pattern recognition network activated during emotional states. To our knowledge, ours is the first demonstration that emotion-related activity patterns in occipital cortex and dorsomedial cerebellum may be modulated by participant risk behavior tendency.

### 4.4. Emotional Response Inhibition Activity vs. Risk Tendency and Impulsivity

The emotional response inhibition contrast (aversive NoGo−aversive Go) did not show any significant relationships with CARE risk scores or BIS impulsivity scores in frontal regions. Several large clusters in vlPFC, dlPFC, and right anterior insula did show significantly larger activation for aversive NoGo vs. aversive Go trials, independently of CARE or BIS scores (see Supplementary Figure 4). Interestingly, Wilbertz et al. (2014) did not find differences related to high and low BIS scores in fMRI signals evoked by performance of the stop signal task, but they did find that participants' scores on the negative urgency subscale of the UPPS were negatively correlated with fMRI activation related to response inhibition in right vlPFC and anterior insula. Individuals scoring high for negative urgency are thought to exhibit impulsive decision-making specifically in the context of negative situations to escape from or minimize exposure to aversive stimuli (Whiteside and Lynam, 2001; Cyders and Smith, 2008). It would be informative to compare the fMRI emotional Go/NoGo task used here with UPPS subscores, including negative urgency, to further investigate this issue. It is also possible that inclusion of participants with more extreme risk and impulsivity tendencies may reveal relationships between emotional response inhibition fMRI activation and high risk or impulsivity tendencies. See discussion below in Section 4.6 for further details.

#### 4.5. Negative BOLD Responses

The vmPFC region from Map 1 (**Figure 4**) and pgACC region from Map 5 (**Figure 5**) exhibited negative BOLD signals (deactivation), with certain trial types inducing larger negative BOLD responses than others. The precise pattern of trial type-induced deactivation was modulated by CARE or BIS scores (see Sections 3.5, 3.6 for details). In vmPFC from Map 1, participants with higher CARE risk scores exhibited greater negative BOLD deflection for NoGo trials, while lower CARE score participants showed the opposite pattern. In pgACC from Map 5, higher BIS score participants exhibited greater negative BOLD deflection for aversive compared to neutral distractor trials, while low BIS participants showed the converse pattern. The vmPFC and pgACC regions are part of the default mode or task negative network, which is well known to show negative BOLD responses when participants perform focused tasks such as the emotional Go/NoGo task (see Raichle et al., 2001; Buckner et al., 2008; Andrews-Hanna et al., 2010; Buckner, 2012). It has been suggested that medial prefrontal (and posterior cingulate) parts of the default mode network are involved in self-referential, emotional decisions and regulation (Andrews-Hanna et al., 2010). It is possible that the fMRI activation patterns described above reflect a deeper disengagement of self-referential functions in the anterior medial default mode network in higher CARE score participants engaged in response inhibition in NoGo trials and in higher BIS score participants performing trials with aversive distractors.

### 4.6. Limitations and Future Directions

Our task included aversive but not pleasant emotional stimuli. Both Casey et al. (2010b) and Ernst and Fudge (2009) have suggested that higher risk behavior tendencies are associated with increased responsivity to pleasant, rewarding stimuli. Many risk behaviors, such as high-risk sex, recreational drug use, and excess alcohol consumption, are pursued in part for their rewarding properties, and the current study design would not assess rewardrelated aspects of risk-related neuronal processing. A future fMRI study using an emotional Go/NoGo task with positive, aversive, and neutral distractors would allow investigation of reward- and approach- as well as avoidance-related brain activity patterns as they relate to risk behavior tendencies and impulsivity. In addition, participants in the current study fell into low- to mediumrisk and low- to medium-impulsivity categories. A future fMRI study recruiting participants with a broader range of risk behavior tendencies and impulsivity levels would allow a more complete picture of brain activity patterns related to risk and impulsivity.

We focused on impulsivity measures provided by the BIS. In the introduction, we discussed the complexity of the impulsivity construct and the lack of a single consensus on how to define that construct. It would be informative to extend the approach used here to include alternative measures of impulsivity as well as related concepts such as sensation and reward seeking from other psychometric instruments including the SSS (Zuckerman, 1994), UPPS (Whiteside and Lynam, 2001), TPQ (Cloninger et al., 1991), and the I7 (Eysenck et al., 1985).

We assessed risk-behavior tendencies using the CARE questionnaire, which provides the advantage that it addresses a wide range of real-world risk behaviors that are influenced by emotions (unsafe sex, alcohol binging, and so on). That the CARE is a self-report instrument raises the possibility that participants may distort their answers, for example by under-reporting socially-undesirable risk behaviors. We note, however, that risk behavior scores from the CARE questionnaire have been shown to associate strongly with objective risk-related measures such as frequency of emergency room visits due to alcohol misuse (Kelly et al., 2005). It would also be informative to combine the approach used here with objective risk behavior measures such as those from the balloon analog risk task (BART; Lejuez et al., 2002) or risky decision-making tasks or gambling tasks from the neuroeconomics literature (for review, see Loewenstein et al., 2008). Though these tasks do not directly assess the same real-world risk behaviors that are the focus of this article, these tasks do provide objective measures of risk-related decision-making.

#### 4.7. Conclusions

We investigated differences related to participants' risk behavior tendencies and impulsivity levels in fMRI brain activity patterns

## References


evoked by an emotional Go/NoGo task. We focused on impulsivity as it is assessed using the BIS instrument and risk behavior tendencies measured using the CARE self-report instrument. Our results showed a dissociation between brain activity profiles related to CARE risk tendencies and to BIS impulsivity, supporting a view of BIS impulsivity and high-risk behavior tendencies as distinct constructs. This view is consistent with previous suggestions that risk behavior can be driven not just by impulsivity as assessed using the BIS, which emphasizes cognitive processes, executive control, and response inhibition, but also by other factors such as reward seeking or sensation seeking (Romer, 2010) and by social behaviors (Gardner and Steinberg, 2005; Blakemore and Robbins, 2012). Higher BIS impulsivity levels may contribute to increased risk behavior tendencies in some cases, but elevated BIS impulsivity is not equivalent to elevated risk behavior tendency. Our results suggest that treatment for high-risk behavior in highly impulsive patients may be more effective if a nuanced approach is taken to understanding potential multi-faceted causes of the high-risk behavior, rather than attributing it to high impulsivity from poor cognitive control.

## Acknowledgments

This work was supported by the Norlien Foundation, the Women and Children's Health Research Institute, Alberta Innovates— Health Solutions, the Canadian Institutes of Health Research, the Alberta Mental Health Board, the University of Alberta, Alberta Health Services, the University of Alberta Hospital Foundation, and the Alberta Innovates Center for Machine Learning. We thank Dr. Alan Carroll and Dr. Oleksandr Hodlevskyy for advice on IAPS picture selection. The work described here was undertaken under the auspices of the Child/Adolescent At Risk Population Imaging, neuropsychology, and population health team (CARPI), of which the authors are members.

## Supplementary Material

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnsys. 2015.00024/abstract

Eur. Arch. Psychiatry Clin. Neurosci. 256, 174–186. doi: 10.1007/s00406-005- 0624-4


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Brown, Benoit, Juhás, Lebel, MacKay, Dametto, Silverstone, Dolcos, Dursun and Greenshaw. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

## Altered frontocingulate activation during aversive interoceptive processing in young adults transitioning to problem stimulant use

## *Jennifer L. Stewart 1, Jason M. Parnass 2, April C. May1, Paul W. Davenport <sup>3</sup> and Martin P. Paulus 1,2\**

*<sup>1</sup> Department of Psychiatry, University of California, La Jolla, CA, USA*

*<sup>2</sup> Psychiatry Service, Veterans Affairs San Diego Healthcare System, San Diego, CA, USA*

*<sup>3</sup> Department of Physiological Sciences, University of Florida, Gainesville, FL, USA*

#### *Edited by:*

*Dave J. Hayes, Toronto Western Research Institute, University of Toronto, Canada*

#### *Reviewed by:*

*Matthew R. G. Brown, University of Alberta, Canada Harriet De Wit, University of Chicago, USA*

#### *\*Correspondence:*

*Martin P. Paulus, Department of Psychiatry, University of California San Diego, 8939 Villa La Jolla Dr., Suite 200, La Jolla, CA 92037-0855, USA*

*e-mail: mpaulus@ucsd.edu*

Problems associated with stimulant use have been linked to frontocingulate, insular, and thalamic dysfunction during decision making and alterations in interoceptive processing. However, little is known about how interoception and decision making interact and contribute to dysfunctions that promote the transition from recreational drug use to abuse or dependence. Here, we investigate brain activation in response to reward, punishment, and uncertainty during an aversive interoceptive challenge in current and former stimulant (cocaine and amphetamine) users using functional magnetic resonance imaging (fMRI). Young adults previously identified as recreational users (*n* = 184) were followed up 3 years later. Of these, 18 individuals progressed to problem stimulant use (PSU), whereas 15 desisted stimulant use (DSU). PSU, DSU, and 14 healthy comparison subjects (CTL) performed a two-choice prediction task at three fixed error rates (20% = reward, 50% = uncertainty, 80% = punishment) during which they anticipated and experienced episodes of inspiratory breathing load. Although groups did not differ in insula activation or subjective breathing load ratings, PSU exhibited lower right inferior frontal gyrus (IFG) and bilateral anterior cingulate (ACC) activation than DSU and CTL during aversive interoceptive processing as well as lower right IFG in response to decision making involving uncertainty. However, PSU exhibited greater bilateral IFG activation than DSU and CTL while making choices within the context of punishing feedback, and both PSU and DSU showed lower thalamic activation during breathing load than CTL. Findings suggest that frontocingulate attenuation, reflecting reduced resources devoted to goal maintenance and action selection in the presence of uncertainty and interoceptive perturbations, may be a biomarker for susceptibility to PSU.

**Keywords: functional magnetic resonance imaging (fMRI), stimulants, decision making, error processing, interoception, breathing load**

### **INTRODUCTION**

A growing literature suggests that brain regions involved in interoception, such as insular cortex, are dysfunctional in substance abuse and dependence and may be involved in the maintenance, escalation, and/or relapse of drug use (Paulus et al., 2009; Naqvi and Bechara, 2010; Verdejo-Garcia et al., 2012a). More specifically, substance dependence may reflect a discrepancy between an individual's predicted vs. actual internal state known as the bodily prediction error, an imbalance disconnected from accurate valuation of external stimuli (e.g., current and future rewards and punishments) that may in turn influence the degree of future drug-related approach vs. avoidance behavior (Paulus and Stewart, 2014). For example, inadequate insular functioning in drug users may result in persistent but undetected aversive states, which are unable to modulate cognitive control mechanisms implemented by the prefrontal cortex (PFC) during decision making (Paulus et al., 2009; Verdejo-Garcia et al., 2012a).

With respect to models of the interoceptive system, researchers (Damasio, 1996; Bechara, 2005) have postulated that the insular cortex coordinates with other brain regions to process and integrate somatosensory feeling states in order to guide future decisions. It has been argued that the thalamus delivers sensory information first to the posterior insula and then to the anterior insula, resulting in bodily feeling states that are registered by the anterior cingulate cortex (ACC), which initiates motivated action to regain internal homeostasis and minimize bodily prediction error (Craig, 2003; Paulus et al., 2009). Neuroimaging research supports the role of thalamic, insular, and genual/subgenual ACC function during probing of the interoceptive system (Critchley, 2004; Critchley et al., 2004; Pollatos et al., 2007; Paulus et al., 2012; Zaki et al., 2012). Individuals with substance dependence may have inadequate function in this relay system in response to positive and/or negative body signals to engage in adaptive approach or avoidance behaviors (Paulus and Stewart, 2014). In particular, the interaction between compromised interoception and cognitive control systems involving regions of the PFC may lead to suboptimal decision making (making poor choices such as using drugs despite anticipating and facing negative outcomes). This hypothesis is supported by recent neuroimaging research showing that amphetamine dependent individuals exhibit lower insula, thalamus, ACC, and PFC activation than healthy comparison subjects while making choices in a simple decision making task and concurrently experiencing pleasant interoceptive stimuli via soft bristle brush (May et al., 2013).

Some studies have shown that stress alters frontocingulate, thalamic, and insular regions in stimulant dependent individuals, leading to heightened craving and relapse (Sinha et al., 2006, 2007; Sinha, 2007). For instance, cocaine-dependent patients exhibit lower ACC activation than healthy comparison subjects during exposure to non-drug related stressful imagery (Sinha et al., 2005), whereas the presence of stress (mild shock to the wrist) is associated with greater thalamus and ACC activation than the absence of stress within the context of drug cues in a small sample of cocaine dependent men (Duncan et al., 2007). Moreover, a recent study examining gender differences in responses to neutral, stress, and cocaine imagery scripts indicates that although cocaine dependent men and women both exhibit greater thalamus activation during stress provocation than healthy subjects, cocaine dependent women show greater insula, ACC, and PFC activation than healthy women in response to stress (Potenza et al., 2012). Taken together, these findings support the assertion that in stimulant dependent individuals, aversive states are associated with heightened neural processing as well as urges to engage in drug-related approach behavior. However, additional research is warranted to determine whether aversive interoceptive states influence valuation of external stimuli when stimulant users are making decisions in the face of both positive and negative consequences.

Individuals with stimulant dependence demonstrate neural and behavioral dysfunction within the context of decision making. For instance, amphetamine dependent patients exhibit impaired behavioral performance (altered win-stay patterns of responding) and attenuated insular and PFC activation when making decisions during varying outcome contexts involving reward, uncertainty, and punishment (Paulus et al., 2002b, 2003, 2005). Moreover, amphetamine and/or cocaine dependent individuals show insular, ACC, PFC, and/or thalamic attenuations in paradigms involving reward evaluation (Goldstein et al., 2007; Monterosso et al., 2007; Hoffman et al., 2008), moral judgments (Verdejo-Garcia et al., 2012b), selective attention and working memory (Bolla et al., 2003; Kubler et al., 2005; Tomasi et al., 2007a,b; Clark et al., 2012), response conflict (Salo et al., 2009; Nestor et al., 2011) and behavioral inhibition (Kaufman et al., 2003; Connolly et al., 2012).

On the whole, neuroimaging studies of decision making indicate that the insular cortex, thalamus, PFC, and ACC subserve many functions that may be impaired in stimulant dependence. However, studies involving the intersection of decision making and interoception are still warranted to address the role of neural and behavioral function in stimulant users, particularly in response to aversive bodily signals that may drive stimulant use. Moreover, it is still unclear whether alterations in frontocingulate, insular, and thalamic regions are: (1) markers of the susceptibility to experiment with stimulants more generally; (2) present in the early stages of problem stimulant use (PSU; e.g., recent onset abuse and/or dependence); or (3) indicators of chronic stimulant use only. If neural mechanisms involved in decision making and interoception are impaired in the transition to problem use as well as in chronic use, neuroimaging can be utilized as an early detection tool to motivate more intensive interventions for high risk individuals.

To address these questions, the present study utilized functional magnetic resonance imaging (fMRI) to study decision making during an aversive interoceptive manipulation in a sample of young adults with varying levels of stimulant use. A twochoice prediction task with fluctuating error rates was employed to examine decision making in response to rewarding, uncertain, and punishing outcomes. In addition, an inspiratory breathing load shown to activate insula and PFC during decision making was used as an aversive interoceptive manipulation (Paulus et al., 2012) during the two-choice prediction task.

Five specific hypotheses were tested in this investigation. First, it was hypothesized that if attenuated insular and frontocingulate activations are markers of stimulant addiction, individuals who had recently transitioned to problems with stimulant use (abuse and/or dependence) would exhibit lower activation in these regions during decision making than past occasional stimulant users and stimulant naïve individuals, given research using the two-choice prediction task in amphetamine dependent patients (Paulus et al., 2002b, 2003, 2005). Second, with respect to the role of interoceptive processing alone and its interaction with decision making, it was predicted that current stimulant users will exhibit insular, thalamic, ACC, and PFC attenuation during aversive interoceptive stimuli, consistent with findings in stimulant dependent subjects during the experience of pleasant interoceptive stimuli (May et al., 2013). Third, we forecasted an additive effect of condition and error rate findings, wherein problem stimulant users would show the lowest insula, ACC, and PFC activation compared to former stimulant users and stimulantnaïve subjects during decisions made under uncertainty paired with the aversive interoceptive manipulation. Fourth, given the proposed role of aversive interoceptive stimuli in the maintenance and/or exacerbation of addiction (Paulus et al., 2009; Naqvi and Bechara, 2010; Verdejo-Garcia et al., 2012a), it was hypothesized that problem users would subjectively report higher unpleasantness ratings of the aversive interoceptive stimuli than the other two groups. Fifth, it was predicted that current problem stimulant users would exhibit higher win-stay behavioral responses consistent with previous research in chronic stimulant dependent individuals (Paulus et al., 2002b, 2003). In addition to analyses examining these a-priori hypotheses, given that personality traits linked to addiction such as impulsivity, sensation seeking, and depression are thought to moderate neural mechanisms involved in interoception and decision making (Leland et al., 2006; Paulus et al., 2008, 2009; Verdejo-Garcia et al., 2008; Brewer et al., 2010; Naqvi and Bechara, 2010), exploratory correlations were performed between brain regions of interest (insula, ACC, PFC, thalamus) and personality measures more highly endorsed in current problem stimulant users than past stimulant users and/or stimulant naïve individuals.

#### **MATERIALS AND METHODS**

#### **SUBJECT RECRUITMENT AND PROCEDURE**

The study protocol was approved by the local Human Subjects Review Board (University of California, San Diego) and was carried out in accordance with the Declaration of Helsinki. Individuals were informed that this study was aimed to examine brain functioning of people who use stimulants, and all subjects gave written informed consent. Recreational, non-dependent male and female stimulant users were recruited and defined by methods described in previous experiments involving this sample (Reske et al., 2011; Stewart et al., 2013). Among this original cohort of 184 subjects, these individuals were contacted 3 years after their initial lab visit, with an overall follow-up rate of 93% (171 followed up; 10 unreachable; 3 refused to participate). Each individual underwent a standardized interview during the followup assessment to examine the extent of drug use, allowing us to identify subjects in this cohort who developed problems associated with stimulant use and others who had desisted using stimulants. Thus, two stimulant user groups were formed for the present study, termed problem stimulant users (PSU) and desisted stimulant users (DSU).

Specifically, PSU were a priori defined by: (1) continued use of prescription and/or recreational stimulants (e.g., dextroamphetamine, cocaine, methylphenidate) since the initial visit and (2) endorsement of 2+ symptoms of DSM-IV-TR amphetamine and/or cocaine abuse or dependence criteria (American Psychiatric Association, 2000) as defined by the Semi Structured Assessment for the Genetics of Alcoholism II (SSAGA II) (Bucholz et al., 1994) occurring together during at least 6 contiguous months since the initial visit (*M* = 4*.*78 symptoms; *SD* = 2*.*24). In comparison, DSU were defined as having: (1) no 6-month periods of time with 3+ uses of reported prescription and/or recreational stimulants, and (2) no endorsement of symptoms of stimulant abuse or dependence (other than nicotine) in the interim as defined by SSAGA II. CTL were recruited from the general population and endorsed no lifetime history of substance or alcohol related problems as determined by SSAGA II (see **Figure 1** for schematic overview; see Section Clinical Interview Session for exclusion criteria for each group). Participants from all three groups were selected to be matched on gender, age, and education. No subjects from any group were regular nicotine smokers. The final cohort of the present study (see **Table 1**) consisted of 18 PSU, 15 DSU, and 14 CTL subjects, all right handed as assessed with the Edinburgh Handedness Inventory (Oldfield, 1971). Subjects then completed two sessions: (1) a clinical interview and questionnaire session; and (2) an fMRI session wherein they completed the two-choice prediction task with breathing load (described below).

#### **CLINICAL INTERVIEW SESSION**

Subjects were assessed by experienced interviewers using the SSAGA II and diagnoses were based on consensus meetings (accredited clinician Martin P. Paulus and trained study personnel). The following were exclusion criteria for all groups: (1) incorporated metal or any other factor that precludes use of fMRI; (2) head injuries or loss of consciousness for longer than 5 min; (3) prescription medication for attention deficit hyperactivity disorder (ADHD), depression, bipolar disorder, anxiety and other psychiatric disorders taken currently and/or within the

#### **Table 1 | Group differences in demographics, personality, and drug use.**


*(Continued)*

#### **Table 1 | Continued**


*aGroup variances are unequal.*

*bTest compared all three groups.*

*cTest compared PSU and DSU only.*

*†Group main effect from CO2 repeated measures analysis of variance. PSU, Problem Stimulant Users. DSU, Desisted Stimulant Users. CTL, Healthy Comparison Subjects. WTAR, Wechsler Test of Adult Reading. SSS-V, Sensation Seeking Scale. BIS-11, Barratt Impulsivity Scale. BDI-II, Beck Depression Inventory II. STAI, State-Trait Anxiety Inventory. BPQ, Body Perception Questionnaire. ANS, Autonomic Nervous System. Stimulant Craving Questionnaire, average of subjects' responses to the Cocaine Craving Questionnaire (CCQ) filled out twice, once with respect to amphetamine use and once with respect to cocaine use. CO2, carbon dioxide. DSM-IV, Diagnostic and Statistical Manual of Mental Disorders IV. The following number of subjects did not complete specific measures: WTAR: n* = *1 PSU and n* = *3 CTL; SSS-V, BPQ, and BIS-11: n* = *3 PSU, n* = *1 DSU, n* = *3 CTL; Stimulant Craving (CCQ): n* = *6 PSU, n* = *1 DSU. An additional CTL was an outlier (>3SD from mean) on BPQ ANS Reactivity and therefore was not included in analysis of that particular subscale.*

past 3 years; (4) any diagnosed neurological disorder (including ADHD); (5) evidence for lifetime psychosis (e.g., schizophrenia, bipolar disorder) or antisocial personality disorder; (6) current and/or past 6 month episodes of DSM-IV anxiety disorders or unipolar depression; and (7) a positive urine toxicology screen for any substance other than marijuana at the time of the fMRI session (given that marijuana can be present in urine as long as 6 weeks after use).

At the time of the clinical interview, several personality and symptom assessment questionnaires known to correlate with substance use disorders were administered, including the Sensation Seeking Scale (SSS) (Zuckerman, 2007), the Barratt Impulsiveness Scale (BIS-11) (Patton et al., 1995), the State-Trait Anxiety Inventory (STAI) (Spielberger et al., 1983), and the Beck Depression Inventory II (BDI-II) (Beck et al., 1996). To assess trait interoceptive responses to stress, subjects completed the Body Perception Questionnaire (BPQ) (Porges, 1993). In addition, PSU and DSU completed the Cocaine Craving Questionnaire (CCQ) (Tiffany et al., 1993) twice, once with respect to craving linked to amphetamine use and once with respect to craving linked to cocaine use.

#### **fMRI SESSION**

All subjects were required to abstain from drugs for 72 h prior to the fMRI session. Two subjects tested positive for marijuana on the pre-fMRI urine toxicology screen (*n* = 1 PSU; *n* = 1 DSU) but no subjects tested positive for any other substances.

#### *Breathing load apparatus*

During the entire fMRI session, subjects wore a nose clip and respired through a mouthpiece and non-rebreathing valve (2600 series, Hans Rudolph). The apparatus was attached to the fMRI scanner head coil to eliminate the need for the subject to contract mouth muscles while maintaining an airtight seal. The resistance load was a stainless steel screen mesh disk placed in a Plexiglas tube (loading manifold). Subjects were given a 40 cmH2O/L/s inspiratory load applied to only the inspiratory port of the nonrebreathing valve for 40 s at a time. Prior to scanning, subjects were given instructions about the task and experienced three 1-min segments of the breathing load. After the fMRI session, subjects completed Visual Analog Scale (VAS) questionnaires, on which they were asked to rate the breathing load experience on a 10 cm scale anchored from "not at all" (0) to "extremely" (10) on the following 16 dimensions: pleasant, unpleasant, intense, tingling, fear of losing control, faintness, fear of dying, unreality, hot/cold flushes, trembling, choking, abdominal distress, chest pain, palpitations, sweating, and dizziness, corresponding to items used in prior studies (Chan and Davenport, 2008; Davenport and Vovk, 2008).

#### *Two-choice prediction task with breathing load manipulation*

The two-choice prediction task has been utilized to determine the response characteristics in decision making situations with uncertain outcomes (Paulus et al., 2002b, 2003, 2005). The version of the two-choice prediction task employed in the present study also included an aversive interoceptive breathing load manipulation. **Figure 2** shows that for each trial (lasting for a fixed duration of 5000 ms), a house was shown in the center of the computer screen (variable duration: 416–797 ms), followed by an updated image of the house with two people: one to the left and one to the right (fixed duration of 1500 ms). Subjects were told that, as soon as they saw two people appear next to the house, their task was to predict whether a car would come by to pick up the person on the left or right side of the computer screen by pressing a left or right button, respectively. Subjects had 1500 ms to register a response. If subjects did not respond during a particular trial, they automatically received negative feedback following response timeout. Participants were given no predictive information and had to make a choice based on the history of preceding responses and outcomes. After the 1500 ms reaction time window ended, the car was presented on the far left or right side of the screen for the remainder of the trial (variable duration: 2703–3084 ms). If the selected response (left or right) matched the side where the car was presented, the person on the selected side met up with the car. Unbeknownst to the subject, the car was presented according to a predetermined schedule. Specifically, a computer algorithm, which took each subject's response into account, determined whether a response would be "correct" or

**FIGURE 2 | Illustration of two choice prediction task.** For each trial, a house was shown in the center with two people: one on the left and one on the right of the house. Subjects pressed a button to predict whether a car would come by on the left or right side to pick up the person. After the subject made a decision, the car was presented on the left or right side of the screen. If the selected response matched the side where the car was presented, the person on the selected side met up with the car. Although each trial lasted 5000 ms and subjects were allowed to respond with a button press during a fixed 1500 ms period at the point when they saw the two people on the screen, the length of the beginning and ends of each trial were designed to have variable interstimulus intervals (lSI). Brain activation consisting of time from trial onset to the button press was included in fMRI analysis as a decision regressor of interest, wherein brain activation during the remainder of the trial was incorporated into the overall baseline regressor.

"incorrect." Correct responses consisted of the word "YAY!" presented in the top center of the computer screen for the remainder of the trial duration, while incorrect responses consisted of the word "BOO!" presented in the same location. The time of trial onset to the subject's button press on each trial was considered the decision phase of interest during the task, whereas the remaining portion of the trial was incorporated into the overall baseline with which the decision phase was later compared.

**Figure 3** shows that the two-choice prediction task was divided into three types of trials with differing reinforcement, or error, rates: (1) 20% *error rate*, indexing response to reward, wherein "YAY" feedback is presented on the computer screen after 80% of each subject's responses and "BOO" feedback is presented on the computer screen for the remaining 20% of trials; (2) 50% *error rate*, indexing response to uncertainty, wherein "YAY" feedback is presented after 50% of each subject's responses and "BOO" feedback is presented for the remaining 50% of trials; and (3) 80% *error rate*, indexing response to punishment, wherein "YAY" feedback is presented after 20% of each subject's responses and "BOO" feedback is presented for the remaining 80% of trials. Two runs of 122 trials each were presented to subjects (total number of trials presented for each error rate: 20% = 80, 50% = 84, 80% = 80). Each error trial type was presented consecutively for 9–20 trials.

Within the context of each error rate, subjects also experienced an aversive interoceptive manipulation, involving three conditions: (1) *baseline* (ranging from 6 to 8 consecutive trials): no additional cues are presented on the screen; (2) *anticipation* (ranging from 3 to 5 consecutive trials): a yellow circle shown in the center of the house, warning the subject that there was a 25% chance that their breathing would be loaded in upcoming trials; and (3) *breathing load* (8 consecutive trials): a yellow sun shown in the center of the house, wherein subject experienced an inspiratory 40 cmH2O/L/s breathing load for 40 s duration. This paradigm was implemented using an event-related fMRI design, consisting of 2 runs with 306 repetition times (TR) each (*TR* = 2000 ms; 2.5 TR per trial). The total number of trials presented for baseline, anticipation, and breathing load interoception conditions were 112, 72, and 48, respectively, with a total of 24 anticipation and 16 breathing load trials presented for each of the three error rates. The remaining 12 trials were null trials distributed across the three error types. At least 3 consecutive trials were presented for baseline and anticipation conditions. The breathing load condition always consisted of 8 trials. Response latency and button choice (left, right) were recorded for each trial. The order of conditions and error rates (see **Figure 3**) were kept fixed across subjects, although the specific feedback given to each subject within each error rate context was contingent upon frequency of their responses in order to match up with the reinforcement determined by each error rate.

#### **NEUROIMAGING ACQUISITION AND ANALYSIS**

Images were acquired using a 3T GE CXK4 Magnet at the UCSD Center for Functional MRI, which is equipped with 8 highbandwidth receivers that allow for shorter readout times and reduced signal distortions and ventromedial signal dropout. Each 1-h session included: (1) a standard anatomical protocol consisting of a spoiled gradient recalled (SPGR) sequence (FOV 25.6 cm; 192 × 256 matrix; 172 sagittally acquired slices of 1 mm thickness; *TR*: 8 ms; *TE*: 3 ms; flip angle = 12) and (2) two runs of the two-choice prediction task (for each run: axial T2∗-weighted echo-planar images (EPI); FOV 24 cm; 64 × 64 matrix; 40 slices of 3 mm thickness; 1.4 mm gap; *TR* = 2000 ms; *TE* = 30 ms; flip angle = 90◦; first two TRs were discarded to allow for BOLD signal stabilization). During the two-choice prediction task, carbon dioxide (CO2) levels were also collected at a rate of 40 Hz for each subject via nasal cannula.

#### *First level analysis*

All subject-level structural and functional image processing was computed with the Analysis of Functional Neuroimages (AFNI) software package (Cox, 1996). The multivariate regressor approach detailed below was used to relate changes in EPI intensity to differences in task characteristics (Haxby et al., 2000). EPI images were co-registered using a 3D-coregistration algorithm (Eddy et al., 1996) that was developed to minimize the amount of image translation and rotation relative to all other images. Six motion parameters (dx, dy, dz, and roll, pitch, and yaw) were obtained across the time series for each subject and the latter three were used as regressors to adjust EPI intensity changes due to motion artifacts. This has been shown to increase power in detecting task-related activation (Skudlarski et al., 1999). Slice timing correction was then performed, followed

by automatic coregistration of the EPI to the high-resolution anatomical image. Each dataset was manually inspected to confirm successful alignment. New outliers were generated for the volume-registered dataset using AFNI's 3dToutcount. If *>* 10% voxels were marked as outliers within a particular TR that time point was then excluded from further analysis. Approximately 1% of TRs were censored (across subjects for entire task: *M* = 7*.*15, *SD* = 4*.*89, range = 0–21).

Nine decision regressors of interest were generated to delineate trials with differing error rates (20%, 50%, 80%) and conditions (baseline, anticipation, breathing load), with timing of the decision phase for each regressor based on individual subjects' reaction times during each trial (see **Figure 2**). These regressors were convolved with a gamma variate function for each subject using the AFNI waver program (Boynton et al., 1996) to model a prototypical hemodynamic response consisting of a 6–8 s delay (Friston et al., 1995) and account for temporal dynamics of the hemodynamic response (typically 12–16 s). All nine convolved time series were normalized. Three movement regressors (roll, pitch, yaw), an overall non-decision related task baseline regressor (see **Figure 3**), a linear drift regressor computed by AFNI, and the nine decision making regressors (20% error: baseline [*n* = 36 trials], anticipation [*n* = 24 trials], and breathing load [*n* = 16 trials]; 50% error: baseline [*n* = 40 trials], anticipation [*n* = 24 trials], breathing load [*n* = 16 trials], and 80% error: baseline [*n* = 36 trials], anticipation [*n* = 24 trials], and breathing load [*n* = 16 trials]) were included in a linear regression model in AFNI's 3dDeconvolve program to estimate the goodness of fit between model estimates and BOLD responses for each subject. Following the deconvolution, voxels were resampled into 4 <sup>×</sup> <sup>4</sup> <sup>×</sup> 4 mm3 space and whole-brain voxel-wise normalized percent signal change, the main dependent measure, was determined by dividing the beta coefficient for each of the nine decision predictors by the beta coefficient for the nondecision related overall baseline regressor and multiplying by 100. Next, a Gaussian spatial filter (4 mm full width half maximum) was used to spatially blur percent signal change values to account for anatomical differences and this output was then normalized to Talairach coordinates (40 × 48 × 38 voxel coverage) as defined by AFNI's built-in atlases. Finally, individual subject percent signal change scaled beta weight values for error rates and interoception conditions (baseline 20%, baseline 50%, baseline 80%, anticipation 20%, anticipation 50%, anticipation 80%, breathing load 20%, breathing load 50%, and breathing load 80%) were extracted for their use as dependent measures in group analyses.

#### *Second level analysis*

A linear mixed effects (LME) model (Pinheiro et al., 2013) was computed in R (R-Development-Core-Team, 2008) for each voxel, wherein group (PSU, DSU, CTL), error rate (20%, 50%, 80%), and interoception condition (baseline, anticipation, breathing load) were modeled as fixed factors, whereas subject was modeled as a random factor. Percent signal change scaled beta weight value was the dependent variable. For each voxel, degrees of freedom, *F*, and *p*-values were obtained for each main effect and interaction. Next, significant clusters of voxels were extracted using a threshold adjustment method based on 1000 Monte-Carlo simulations (AFNI's program Alpha Sim), which guarded against identifying false positive areas of activation (considering whole brain voxel size, 4 mm smoothness). For main effects and interactions involving group, AlphaSim identified a minimum cluster volume of 512μL (8 contiguous voxels) to result in a voxel-wise probability of *p <* 0*.*02 significance (*p <* 0*.*01 two tailed), corrected for multiple comparisons. The voxelwise threshold for effects of interest were based on the following LME degrees of freedom and *F* values: (1) Group main effect: *F(*2*,* <sup>44</sup>*)* = 4*.*28; (2) Error rate and interoception condition main effects: *F(*2*,* <sup>352</sup>*)* = 3*.*95; (3) Group by error rate, group by condition, and error rate by interoception condition interactions: *F(*4*,* <sup>352</sup>*)* = 2*.*96; and (4) Group by error rate by interoception condition interaction: *F(*8*,* <sup>352</sup>*)* = 2*.*31<sup>1</sup> .

#### **QUESTIONNAIRE/INTERVIEW ANALYSIS**

Group differences in demographics, personality measures, and state emotion were evaluated using Predictive Analytics Software (PASW) (SPSS, 2009) univariate analysis of variance (ANOVA) and Bonferroni *post-hoc* tests for significant results. In addition, *t*-tests were used to examine differences between PSU and DSU in stimulant craving and interim/lifetime drug use (number of distinct sessions used) and chi-square tests compared frequency of substance abuse/dependence diagnoses between groups.

#### **PHYSIOLOGICAL ANALYSIS**

CO2 data were visually inspected for artifacts and down sampled by 80 (40 Hz ∗ 2 s per TR) to obtain one value per TR per fMRI run. A total of 32/47 (68%) of subjects (*n* = 11 DSU and CTL; *n* = 10 PSU) had usable CO2 data for both runs. For these subjects, CO2 values during each error rate and interoception condition were extracted, averaged, and input into a repeated measures ANOVA with condition and error rate as within-subjects factors, and group as the between-subjects factor.

#### **BEHAVIORAL ANALYSIS**

A LME was computed in R for percentage of win-stay responses, wherein group (PSU, DSU, CTL) and error rate (20%, 50%, 80%) were fixed factors and subject was modeled as a random factor. Condition was not included as a factor because the anticipation condition consisted of too few trials within each error rate to extract reliable probability estimates.

#### **EXPLORATORY ANALYSIS**

Given that PSU endorsed higher BIS inattention-related impulsivity, BDI-II depression, and higher number of interim stimulant uses (the latter two natural log transformed due to nonnormality) than DSU and/or CTL (please see results below), within the PSU group correlations were computed between each of these three measures and variables of interest in order to assist in the explanation of results: (1) percentage of win-stay responses averaged across error rates; (2) VAS unpleasantness ratings; and (3) thalamus, PFC, ACC, and insula activation emerging as significant from LME results. Correlations were then corrected for multiple comparisons (*p* = 0*.*05*/*18 = 0*.*003).

#### **RESULTS**

#### **QUESTIONNAIRE/INTERVIEW ANALYSIS**

Demographic, personality, state emotion, and drug use information is presented in **Table 1**. Groups were comparable in age and education and endorsed similar levels of sensation seeking, anxiety, and levels of alcohol abuse. PSU endorsed higher levels of BDI-II depression than DSU (*post-hoc p* = 0*.*01) but neither group differed from CTL. PSU also reported higher BIS Inattention scores than CTL (*post-hoc p* = 0*.*02) but both groups did not differ from DSU. With respect to drug use, PSU and DSU reported similar levels of current stimulant craving, current marijuana abuse, current alcohol abuse and dependence, and lifetime marijuana use. However, PSU endorsed greater stimulant use in the 3-year interim period prior to the interview/fMRI sessions than DSU as well as higher levels of current stimulant abuse and stimulant and marijuana dependence.

#### **PHYSIOLOGICAL ANALYSIS**

A main effect of interoception condition emerged [*F(*2*,* <sup>58</sup>*)* = 26*.*82, *p <* 0*.*001], wherein breathing load was associated with lower CO2 (*M* = 1*.*22, *SE* = 0*.*05) than baseline (*M* = 1.40, *SE* = 0*.*05) and anticipation (*M* = 1.41, *SE* = 0*.*05) across subjects. No other main effects or interactions, including those with group, approached significance (all *p >* 0.29).

#### **VAS ANALYSIS**

Subjective ratings of the aversive interoceptive manipulation are presented in **Table 2**. Overall, across subjects the breathing load stimulus was rated low in pleasantness, moderately high in unpleasantness, and moderate in intensity. Although groups did not differ on any of the sixteen dimensions rated, exploratory analyses (**Figure 4A**) showed that within PSU, higher BIS Inattention scores were associated with greater unpleasantness ratings of the breathing load stimuli (*r* = 0*.*54, *p* = 0*.*04), although this correlation did not survive correction for multiple comparisons.

#### **BEHAVIORAL ANALYSIS**

Results indicated that group main effects or interactions with error rate did not emerge for win-stay responses (*p* = 0*.*43 and *p* = 0*.*45). However, an error rate main effect was evident for win-stay responses, which were more frequent across subjects for 20% error rate (54%) than 80% error rate (46%) [*F(*2*,* <sup>86</sup>*)* = 7*.*8, *p* = 0*.*001]. Despite no group differences in win-stay behavior, **Figure 4B** illustrates that within PSU higher interim stimulant use was associated with lower rates of win-stay behavior during the task (*r* = −0*.*49, *p* = 0*.*04), although this finding did not survive correction for multiple comparisons.

#### **fMRI ANALYSIS**

No significant results emerged for the group main effect, the error rate by interoception condition interaction, or the group by error

<sup>1</sup>fMRI data were also analyzed at the standard *<sup>p</sup>* <sup>=</sup> <sup>0</sup>*.*01 cutoff using the following LME degrees of freedom and F values: (1) Group main effect: *F(*2*,* 44*)* = 5*.*12; (2) Error rate and interoception condition main effects: *F(*2*,* 352*)* = 4*.*66; (3) Group by error rate, group by condition, and error rate by interoception condition interactions: *F(*4*,* 352*)* = 3*.*37; and (4) Group by error rate by interoception condition interaction: *F(*8*,* 352*)* = 2*.*56. Voxelwise extraction was corrected for multiple comparisons at *p* = 0*.*01 (minimum cluster size = 7 contiguous voxels).


**Table 2 | Post-fMRI Visual Analog Scale (VAS) ratings of aversive interoceptive stimulus (breathing load).**

*PSU, Problem Stimulant Users; DSU, Desisted Stimulant Users; CTL, Healthy Comparison Subjects; VAS scales range from 0–10.*

rate by interoception condition interaction using the voxelwise corrected *p* = 0*.*02 threshold<sup>2</sup> .

#### *Error rate main effect*

Across subjects, the 50% error rate was associated with greater left dorsal ACC, left thalamus, bilateral inferior frontal gyrus (IFG), and right inferior temporal gyrus activation than the 20% and 80% error rates (see **Figure 5A** and **Table 3**). In addition, the 80% error rate resulted in greater bilateral IFG activation than the 20% error rate. Exploratory analyses revealed that within PSU, higher BIS Inattention scores were associated with lower left thalamus and bilateral IFG activation in response to the 20% error rate (thalamus *r* = −0*.*53, *p* = 0*.*04; left IFG *r* = −0*.*55, *p* = 0*.*03; right IFG *r* = −0*.*57, *p* = 0*.*03), although both findings did not survive correction for multiple comparisons.

#### *Interoception condition main effect*

Given that conditions differed across a large portion of the cortex, (e.g., 3863 contiguous voxels emerged as significant for the wholebrain analysis), a restricted mask (threshold *p* = 0*.*02 pcorrected

**FIGURE 4 | (A)** Although problem stimulant users (PSU), desisted stimulant users (DSU), and healthy comparison subjects (CTL) did not differ on visual analogue scale (VAS) unpleasantness ratings of the breathing load stimuli, within PSU higher Barratt Impulsivity Scale (BlS) Inattention scores were associated with higher VAS unpleasantness ratings. Three PSU subjects did not have BIS scores and therefore were not included in scatterplots. **(B)** Although groups did not differ in percentage of win-stay responses across error-rates, within PSU a greater frequency of stimulant drug uses in the past 3 year interim period was associated with a lower percentage of win-stay responses. Error bars indicate ± 1 standard error.

for multiple comparisons) was used to examine differences as a function of breathing load for insula, ACC, striatum, and thalamus, brain regions implicated in the processing of interoception in response to pleasant stimuli (May et al., 2013). Across subjects, the breathing load condition was associated with greater activation in bilateral anterior/posterior insula and bilateral dorsal striatum (caudate) than the anticipation condition, which in turn elicited greater activation than the baseline condition (see **Figure 5B** and **Table 4**). Although groups did not differ in insula activation during breathing load, exploratory analyses showed that within PSU, higher BIS Inattention scores were associated with lower right insula activation during breathing load (*r* = − 0*.*52, *p* = 0*.*049) but this finding did not survive correction for multiple comparisons.

#### *Group by error rate interaction*

**Table 5** demonstrates that for the 20% error rate, PSU exhibited lower activation than CTL in left superior frontal gyrus (SFG). For the 50% error rate, PSU showed lower activation than CTL in bilateral temporal gyri, bilateral postcentral gyri, right supramarginal gyrus, and right IFG (see **Figure 6**). In contrast, for the 80% error rate, PSU exhibited greater bilateral IFG activation than DSU and CTL.

#### *Group by interoception condition interaction*

On the whole, similar patterns of activation emerged for baseline and anticipation conditions as a function of group. However, during the breathing load condition, PSU exhibited lower activations

<sup>2</sup>No findings emerged for the group main effect, error rate by condition interaction, or the group by error rate by condition interaction at *p* = 0*.*01. For the remaining main effects and interactions presented in **Tables 3**–**6**, footnotes under each table highlight which brain regions remained significant at the *p* = 0*.*01 threshold.

error rate) elicited greater anterior cingulate cortex (ACC), thalamus, and inferior frontal gyrus (IFG) activation than reward (20% error rate) and punishment (80% error rate). Within the problem stimulant user (PSU) group, higher Barratt Impulsivity (BIS) Inattention scores were associated with lower thalamus and IFG activation in response to 20% error rate. **(B)** The main

anterior and posterior insula and dorsal striatum (caudate) activation than baseline and anticipation conditions. Within the PSU group, higher BIS Attention scores were linked to lower right insula activation during breathing load. Three PSU subjects did not have BIS scores and therefore were not included in scatterplots. Error bars indicate ± 1 standard error.

#### **Table 3 | fMRI Results for the main effect of error rate.**


*L, left hemisphere; R, right hemisphere; BA, Brodmann Area; Coordinates reflect center of mass for each cluster. All clusters emerged as greater than F(2, 352)* = *3.95, <sup>p</sup>* <sup>=</sup> *0.02 corrected voxelwise for multiple comparisons (minimum significant cluster* <sup>=</sup> *8 voxels). †Regions that also remained significant at F(2, 352)* <sup>=</sup> *4.66, p* <sup>=</sup> *0.01 corrected.*

in bilateral subgenual ACC, right striatum, right middle frontal gyrus (MFG), right IFG, left cuneus, and left parahippocampal gyrus than DSU and CTL (see **Figure 7** and **Table 6**). Moreover, PSU and DSU showed lower bilateral thalamus, bilateral middle temporal gyrus, and bilateral cerebellum activation than CTL during the breathing load condition.

#### **DISCUSSION**

The present study examined how young adults with varying levels of stimulant use differed on neural, behavioral and selfreport indices of decision making while experiencing an aversive interoceptive stimulus. Young adults transitioning to PSU show attenuated frontal activations in response to aversive interoceptive


**Table 5 | fMRI results for the group by error rate interaction.**


*L, left hemisphere; R, right hemisphere; BA, Brodmann Area. Coordinates reflect center of mass for each cluster. All clusters emerged as greater than F(2, 352)* = *3.95, p* = *0.02 corrected voxelwise for multiple comparisons (minimum significant cluster* = *8 voxels). All regions remained significant at F(2, 352)* = *4.66, p* = *0.01 corrected.*


*L, left hemisphere; R, right hemisphere; BA, Brodmann Area; PSU, Problem Stimulant Users; DSU, Desisted Stimulant Users; CTL, Healthy Comparison Subjects. Other 2, remaining two groups. Coordinates reflect center of mass for each cluster. All clusters emerged as greater than F(4, 352)* = *2.96, p* = *0.02 corrected voxelwise for multiple comparisons (minimum significant cluster* <sup>=</sup> *8 voxels). †Regions that remained significant at F(4, 352)* <sup>=</sup> *3.37, p* <sup>=</sup> *0.01 corrected.*

stimuli (right IFG/MFG), rewarding outcomes (left SFG), and ambiguous outcomes (right IFG), consistent with hypotheses. In contrast to predictions, however, PSU exhibited heightened frontal activation to punishing outcomes (bilateral IFG), which may be due to the fact that PSU do not register ambiguous feedback as salient via IFG as healthy individuals do (Hampshire et al., 2010). Moreover, it is possible that PSU need to recruit greater IFG to override or ignore aversive non-interoceptive feedback (punishment) but are unable to do so within the context of aversive interoceptive feedback (breathing load) due to an altered homeostatic system (Paulus et al., 2009; Paulus and Stewart, 2014), characterized by reduced thalamic and ACC function. In summary, PSU exhibit impaired somatosensory input via the thalamus and under-recruitment of neural resources to motivate remediation of aversive perturbations via the ACC. In other words, problem users do not exert as many neural resources to process aversive body states and may not integrate these aversive states with ongoing controlled processing, suggesting a disconnect between how "feeling bad" affects a change in "acting."

In this investigation, five specific hypotheses were examined. First, it was predicted that if attenuated brain activation during decision making indexed current stimulant abuse/dependence, then PSU would exhibit lower neural activation than DSU and CTL while making choices during reward, uncertainty, and/or punishment feedback. PSU exhibited lower right IFG activation than DSU and CTL during uncertain outcomes as hypothesized, but in contrast showed greater bilateral IFG activation than both groups in response to punishing outcomes. Given that healthy individuals demonstrate heightened IFG activation in response to uncertainty (Paulus et al., 2002a; Huettel et al., 2005; Volz et al., 2005; Krain et al., 2006), reduced IFG responses suggest that neural valuation of stimuli is aberrant in PSU. With respect to reward processing, CTL exhibited a pattern of greater left SFG responses to reward than punishment, replicating recent research in healthy individuals (Linke et al., 2010). In contrast, PSU and DSU did not exhibit modulation in this region as a function of valenced feedback. Given that our task as a whole focused on aversive interoceptive manipulation superimposed on rewarding

**FIGURE 6 | The group by error rate interaction demonstrates that, compared to healthy comparison subjects (CTL), problem stimulant users (PSU) exhibited (1) lower left middle temporal gyrus (MTG) bilateral postcentral gyrus (PCG), and right supramarginal gyrus (SMG) activation in response to uncertainty**

**(50% error rate); (2) lower left superior frontal gyrus activation in response to reward (20% error rate); and (3) higher bilateral inferior frontal gyrus (IFG) to punishment.** Activation is reflected as percent signal change (%SC) from baseline. Error bars indicate ± 1 standard error.

standard error.

feedback, additional research is needed to determine the role of SFG in stimulant use in various contexts involving reward.

**stimulant users (DSU) and healthy comparison subjects (CTL)**

Our second prediction was that PSU would exhibit lower neural activation than DSU and CTL during an aversive interoceptive stimulus. This hypothesis was partially supported, wherein PSU showed lower right IFG/MFG and bilateral subgenual ACC activation than the other two groups. However, both PSU and DSU exhibited lower bilateral thalamic activation than CTL, suggesting that reduced processing of somatosensory inputs (Craig, 2003) may be characteristic of the propensity to try stimulants, rather than a marker of stimulant abuse/dependence. These results suggest that right frontocingulate attenuation, reflecting reduced resources devoted to goal maintenance and action selection in the presence of interoceptive perturbations (May et al., 2013) are indicators of PSU. However, somatosensory reductions via thalamic projections do not appear to be specific to problem use. Third, we hypothesized that lower frontocingulate and insular activation differences between PSU and the other groups would be strongest for the breathing load condition paired with uncertainty (50% error rate). We did not find group differences in insula activation as a function of the aversive interoceptive manipulation with or without uncertainty, suggesting that integration


*L, left hemisphere; R, right hemisphere; BA, Brodmann Area; PSU, Problem Stimulant Users; DSU, Desisted Stimulant Users; CTL, Healthy Comparison Subjects; Coordinates reflect center of mass for each cluster. All clusters emerged as greater than F(*4*,* <sup>352</sup>*)* = 2*.*96*, p* = *0.02 corrected voxelwise for multiple comparisons (minimum significant cluster* = *8 voxels). Brain activation in all regions above did not correlate with (1) log-transformed lifetime marijuana uses within PSU and DSU; (2) log-transformed interim stimulant uses within PSU; (3) log-transformed BDI-II scores within PSU; (4) BIS-11 Attention scores within PSU. †Regions that remained significant at F(*4*,* <sup>352</sup>*)* = 3*.*37*, p* = *0.01 corrected.*

and generation of bodily feeling states associated with insular function may only be impaired in chronic stimulant users.

Fourth, we hypothesized that PSU would report higher subjective unpleasantness ratings of breathing load than DSU and CTL, and overall this prediction was not supported. Although PSU with higher impulsive inattention reported higher ratings of breathing load unpleasantness, this finding was examined *post-hoc* and did not survive multiple comparison thresholds of significance. It could be that in early stages of PSU, individual differences in personality characteristics may play a role in potential aversive interoceptive dysfunction, with a disjointed relationship between subjective feeling states (heightened reported unpleasantness) and neural registration of these feeling states (attenuated insula during unpleasant stimuli). Our findings hint at this possibility, but lack power to robustly support it. Future research is warranted to determine whether larger samples of recent problem users as well as chronic stimulant users show this disjunction moderated by impulsivity.

Our fifth and final prediction was that PSU would employ greater use of a win-stay behavioral strategy than DSU and CTL, a hypothesis that was not supported. *Post-hoc* exploratory analysis suggested a link between greater frequency of stimulant use in the past 3 years and lower win-stay patterns of responding to feedback in the PSU group, a pattern also evident in a recent study of occasional stimulant users (Paulus et al., 2008). However, this correlation did not survive correction for multiple comparisons, limiting further interpretation. Results within our sample indicate that young adults transitioning to problem use do not show similar behavioral impairments as chronic stimulant dependent patients. Perhaps behavioral inflexibility is a result of chronic stimulant use.

Despite several strengths of this study, including use of a novel paradigm and recruitment of young adults at different stages of stimulant use, this investigation possesses several limitations. First, although we had aimed to examine the interaction between decision making and interoception, the two-way error rate by interoception interaction and the three-way group by error rate by interoception condition interaction did not produce significant findings. These null results may be due to an underpowered sample and/or not enough trials within each error rate and interoceptive condition to examine differences. Additional research is warranted to determine the influence of aversive interoceptive stimuli on decision making within the context of different types of valenced feedback in healthy individuals as well as substance users. Second, prior work demonstrates gender differences in neural responses to stress as a function of stimulant dependence (Duncan et al., 2007; Potenza et al., 2012). However, given the relatively small sample sizes for each of our three groups, we are underpowered to reliably examine the role of gender in the present paradigm. Third, although neural indices of interoception and decision making may be differentially altered as a function of the type of stimulant drug used, the modest sample size of groups in the present study did not allow us to examine differences in amphetamine vs. cocaine problem use. Fourth, results indicated that CO2 was altered by the aversive interoceptive manipulation across subjects but did not differ as a function of group membership. However, only 2/3 of subjects had usable CO2 recordings, which limits the conclusions that can be drawn from this analysis. Future investigations utilizing a large sample of healthy individuals might include CO2 levels as a regressor in the fMRI deconvolution analysis to determine its influence on the overall BOLD signal. However, usable CO2 data did not differ as a function of group, suggesting that group differences as a function of interoceptive condition cannot be reduced to differences in carbon dioxide levels during breathing load.

This investigation employed a novel task to examine neural and behavioral indicators of decision making and interoception in recent PSU, demonstrating that altered frontocingulate function characterizes young adults transitioning to stimulant dependence. Additional studies are needed to clarify the interoceptive contexts wherein recent and chronic stimulant users exhibit decision making dysfunction.

#### **AUTHOR CONTRIBUTIONS**

Jennifer L. Stewart assisted in data collection, analyzed final fMRI, behavioral, and self-report data, and wrote original draft of manuscript. Jason M. Parnass processed fMRI data up to final analysis, created figures, assisted in data presentation/interpretation, and edited draft of manuscript. April C. May assisted in data collection, created figures, assisted in data presentation/interpretation, and edited draft of manuscript. Paul W. Davenport designed aversive interoceptive manipulation, assisted in data interpretation, and edited draft of manuscript. Martin P. Paulus designed the two-choice prediction task as well as the entire grant-funded project, assisted in data interpretation, and edited draft of manuscript.

#### **ACKNOWLEDGMENTS**

Grant Support: This work was supported by grants from the National Institute on Drug Abuse (Grant Nos. R01-DA016663, P20-DA027834, R01-DA027797, and R01-DA018307 to Martin Paulus).

#### **REFERENCES**


abstinence. *Drug Alcohol Depend.* 121, 45–53. doi: 10.1016/j.drugalcdep. 2011.08.007


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

#### *Received: 06 August 2013; accepted: 29 October 2013; published online: 15 November 2013.*

*Citation: Stewart JL, Parnass JM, May AC, Davenport PW and Paulus MP (2013) Altered frontocingulate activation during aversive interoceptive processing in young adults transitioning to problem stimulant use. Front. Syst. Neurosci. 7:89. doi: 10.3389/ fnsys.2013.00089*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2013 Stewart, Parnass, May, Davenport and Paulus. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Mesocorticolimbic monoamine correlates of methamphetamine sensitization and motivation

*Kevin D. Lominac 1, Courtney L. McKenna1, Lisa M. Schwartz 1, Paige N. Ruiz 1, Melissa G. Wroten1, Bailey W. Miller 1, John J. Holloway1, Katherine O. Travis 1, Ganesh Rajasekar 1, Dan Maliniak1, Andrew B. Thompson1, Lawrence E. Urman1, Tamara J. Phillips <sup>2</sup> and Karen K. Szumlinski <sup>1</sup> \**

*<sup>1</sup> Department of Psychological and Brain Sciences, Neuroscience Research Institute, University of California at Santa Barbara, Santa Barbara, CA, USA <sup>2</sup> Behavioral Neuroscience, Methamphetamine Abuse Research Center, Veterans Affairs Medical Center, Oregon Health and Science University, Portland, OR, USA*

#### *Edited by:*

*Dave J. Hayes, University of Toronto, Canada*

*Reviewed by: Nicola B. Mercuri, University of Rome, Italy M. Foster Olive, Arizona State University, USA Marek Schwendt, University of Florida, USA*

#### *\*Correspondence:*

*Karen K. Szumlinski, Department of Psychological and Brain Sciences, University of California at Santa Barbara, Building 551, UCen Road, Santa Barbara, CA 93106-9660, USA e-mail: karen.szumlinski@ psych.ucsb.edu*

Methamphetamine (MA) is a highly addictive psychomotor stimulant, with life-time prevalence rates of abuse ranging from 5–10% world-wide. Yet, a paucity of research exists regarding MA addiction vulnerability/resiliency and neurobiological mediators of the transition to addiction that might occur upon repeated low-dose MA exposure, more characteristic of early drug use. As stimulant-elicited neuroplasticity within dopamine neurons innervating the nucleus accumbens (NAC) and prefrontal cortex (PFC) is theorized as central for addiction-related behavioral anomalies, we used a multi-disciplinary research approach in mice to examine the interactions between sub-toxic MA dosing, motivation for MA and mesocorticolimbic monoamines. Biochemical studies of C57BL/6J (B6) mice revealed short- (1 day), as well as longer-term (21 days), changes in extracellular dopamine, DAT and/or D2 receptors during withdrawal from 10, once daily, 2 mg/kg MA injections. Follow-up biochemical studies conducted in mice selectively bred for high vs. low MA drinking (respectively, MAHDR vs. MALDR mice), provided novel support for anomalies in mesocorticolimbic dopamine as a correlate of genetic vulnerability to high MA intake. Finally, neuropharmacological targeting of NAC dopamine in MA-treated B6 mice demonstrated a bi-directional regulation of MA-induced place-conditioning. These results extend extant literature for MA neurotoxicity by demonstrating that even subchronic exposure to relatively low MA doses are sufficient to elicit relatively long-lasting changes in mesocorticolimbic dopamine and that drug-induced or idiopathic anomalies in mesocorticolimbic dopamine may underpin vulnerability/resiliency to MA addiction.

**Keywords: methamphetamine, sensitization, nucleus accumbens, dopamine, prefrontal cortex, serotonin, addiction vulnerability**

#### **INTRODUCTION**

Methamphetamine (MA) is a potent, highly addictive, amphetamine derivative with intense psychomotor-activating properties. MA abuse is linked to pronounced cognitive, behavioral and emotional deficits, with a high relapse potential, constituting a major public health concern (e.g., Rusyniak, 2011; Dean et al., 2013). While MA ranks second highest as the most commonly abused illicit drug in the world (United Nations Office on Drugs and Crime, 2011), neurobiological research concerning genetic vulnerability to MA abuse/addiction and the impact of early MA experience on the brain to the development of early-stage addiction is limited.

MA is a substrate for plasma membrane monoamine transporters, including the dopamine (DA) transporter (DAT), as well as for the vesicular monoamine transporter, and is reported to also inhibit monoamine oxidase (e.g., Fleckenstein et al., 2007; Chen et al., 2009). Through these mechanisms, MA profoundly increases DA within forebrain terminals, notably nucleus accumbens (NAC), dorsal striatum and prefrontal cortex (PFC) (e.g., Sulzer et al., 2005). As such, the majority of neurobiological research pertaining to MA addiction has focused primarily on MA-forebrain DA interactions (e.g., McCann and Ricaurte, 2004; Yamamoto and Bankson, 2005; Espana and Jones, 2013). The majority of extant pre-clinical data has been derived using very high-dose MA treatment regimens (10–100 mg/kg acutely or binge administration of 4–10 mg/kg given multiple times within a day) that elicit neurotoxicity within dorsal striatal regions (for recent reviews on the subject: Kuhn et al., 2011; Carvalho et al., 2012; Ares-Santos et al., 2013; Halpin et al., 2014). While we have gained tremendous molecular and cellular insight into how high-dose MA experience produces forebrain damage of relevance to late-stage addiction, to the best of our knowledge, less than 15 reports exist pertaining to the interactions between forebrain dopamine systems and low-dose, subchronic exposure to MA (e.g., Zhang et al., 2001; Broom and Yamamoto, 2005; Ago et al., 2006, 2007, 2012; Segal and Kuczenski, 2006; Fukakusa et al., 2008; Schwendt et al., 2009; Lominac et al., 2012; Lacan ´ et al., 2013; Le Cozannet et al., 2013). Also, whereas it is generally held that repeated MA exposure elicits a sensitization of forebrain dopamine release that contributes to the development of behavioral sensitization and/or underpins this drug's positivereinforcing or rewarding properties (e.g., Ujike et al., 1989; Yang et al., 2008a,b), discrepancies exist regarding the relative roles played by DA within different forebrain terminal regions, most notably the NAC vs. mPFC (see Ago et al., 2006, 2007, 2012).

Moreover, in comparison to the extant data for cocaine and d-amphetamine (c.f., Robinson and Becker, 1986; Vanderschuren and Kalivas, 2000) and for high-dose MA exposure (c.f., Carvalho et al., 2012), we know relatively little regarding how MA exposure early during the use of the drug impacts the brain. As such an understanding has relevance to the transition from MA use to abuse/addiction, the first study presented here examined the effects short and longer-term withdrawal from repeated MA upon basal extracellular DA (DAEC) content, DAT and D2/3 receptor (D2/3R) expression, as well as MA-induced DA sensitization, experimenter-administered, MA injections after short-term and longer-term withdrawal in the NAC and PFC. Prior studies have indicated that the effects of contingent vs. non-contingent intravenous MA upon striatal indices of DAEC do not depend highly upon the behavioral contingency of MA delivery and are qualitatively similar (Lominac et al., 2012; Lacan et al., 2013 ´ ). Thus, we employed a repeated, experimenter-administered injection regimen for this study as this route of MA administration is technically facile in mice.

Another major question pertaining to the neurobiology of MA addiction is why only certain individuals come to repeatedly abuse MA in the first place? This question has also received very little direct experimental attention, until recently (Ikeda et al., 2007; Morita et al., 2008; Phillips et al., 2008; Uhl et al., 2008; Wheeler et al., 2009; Shabani et al., 2011, 2012a,b). In humans, moderate doses of amphetamine-type stimulants (e.g., 0.1–0.4 mg/kg) elicit euphoria and behavioral activation, which are typically considered appetitive/reinforcing; higher, subtoxic, MA doses (e.g., 1.0–4.0 mg/kg) can induce anxiety and elicit psychophysiological symptoms, which can be perceived as aversive (c.f., Cruickshank and Dyer, 2009). As for other drugs of abuse (e.g., Schuckit et al., 1997; Fergusson et al., 2003; Petrakis et al., 2004), individual differences in sensitivity to MA's rewarding vs. aversive effects might influence addiction risk. Thus, experimental attention regarding the neurobiological substrates of MA addiction vulnerability is critical to understand MA addiction etiology, identify potential biomarkers for MA addiction vulnerability/resiliency and develop treatment strategies for early intervention in the disease process. The advent of mice selectively bred to drink higher vs. lower amounts of MA (Methamphetamine High Drinking or MAHDR and Methamphetamine Low Drinking or MALDR) that not only differ in their MA intake and preference under free-access 2 bottle-choice procedures (Wheeler et al., 2009), but also exhibit divergent phenotypes under operant and place-conditioning procedures (Shabani et al., 2011, 2012a,b) provide the opportunity to identify biochemical correlates of high vs. low genetic risk for MA addiction-related behaviors. Thus, we also determined whether or not dopamine anomalies were correlates of genetic vulnerability/resiliency to MA addiction-related behavior and assayed, using neuropharmacological approaches, the role for forebrain DA in MA preference in inbred mice. As the results of an earlier study of MAH/LDR mice suggested anomalies in serotonin (5HT) as a biochemical correlate of high MA intake (Wheeler et al., 2009), we also examined for line differences in indices of forebrain 5HT function.

## **MATERIALS AND METHODS**

#### **SUBJECTS**

Most studies employed adult, male (8 weeks old) C57BL/6J (B6) mice (Jackson Laboratories, Sacramento, CA). For studies of the biochemical correlates of genetic vulnerability/resiliency for MA intake, adult female MA High Drinking and MA Low Drinking (MAH/LDR) mice on a mixed C57BL/6J and DBA2/J background were generated at the Portland VA Medical Center (see Wheeler et al., 2009) and shipped to UCSB Santa Barbara, where they were acclimatized for a minimum of 6 weeks. Mice were housed in groups under a regular 12-h light:dark cycle (lights off at 19:00 h), with food and water available *ad libitum*. All experimental protocols were consistent with the guidelines put forth by the NIH (NIH Publication No. 80–23, revised 1996) and were approved by the Institutional Animal Care and Use Committees of the University of California Santa Barbara and Oregon Health and Science University.

### **STEREOTAXIC SURGERY**

The surgical procedures for implanting stainless steel guide cannulae (10 mm, 20 gauge, Small Parts; Roanoke, VA) above the mPFC and NAC of mice were identical to those described recently (see Ary et al., 2013). For studies involving *in vivo* microdialysis or microinjection procedures (see below), mice were anesthetized using 1.5–2% isoflurane with 4% oxygen as a carrier gas. Mice were mounted in a stereotaxic device with tooth and ear bars adapted for mice. The animal's skull was exposed, leveled, and holes were drilled based on coordinates from Bregma for the mPFC (AP: +1.8 mm; ML: ± 0.5 mm; DV: −1.0 mm) or NAC (AP: +1.3 mm, ML: ±1 mm, DV: −2.3 mm), according to the mouse brain atlas of Paxinos and Franklin (2007). The guide cannulae were lowered bilaterally such that the tips of the cannulae were 3 mm above the mPFC or border region of the shell and core subregions of the NAC. The skull was then prepared for polymer resin application, the 2 guide cannulae occluded and post-operative care was conducted as described previously (e.g., Ary et al., 2013). Probe placements within the mPFC and NAC were verified prior to any statistical analyses using microscopic analysis of Nissl-stained sections. Only those mice exhibiting correct placement within the boundaries of the mPFC or NAC were included in the statistical analyses of the data (see e.g., **Figure 1**).

#### **MA TREATMENT AND EXPERIMENTAL DESIGN** *Studies of B6 mice*

Following either a minimum of 5 days recovery from surgery or following acclimation to the vivarium, B6 mice were randomly assigned to receive either repeated intraperitoneal injections of 2 mg/kg MA (Sigma Aldrich; St Louis, MO) or an equivalent volume of 0.09% saline (SAL; vol = 0.01 ml/kg). MA/SAL injections were administered once daily, for 10 consecutive days, as this regimen is reported to alter NAC DA in rats (Broom and Yamamoto, 2005). *In vivo* microdialysis procedures or sacrifice for immunoblotting were conducted at either 1 or 21 days withdrawal in B6 mice. Whenever possible (see below), the B6 mice

underwent 2 identical microdialysis sessions; the first session was conducted at 1 day withdrawal and the second session was conducted at 21 days withdrawal to d and separate groups of animals were used to assay for MA-induced changes in basal DA content, for basal D2R function, for basal DAT function and for MA-stimulated release (*n* = 10–12 at the outset of each assay), as described below.

#### *Studies of MAH/LDR mice*

Due to the relatively limited number of animals available, the MAH/LDR mice were randomly assigned to *in vivo* microdialysis or immunoblotting studies. Mice in the *in vivo* microdialysis studies were assayed in 2 distinct microdialysis sessions, separated by 2–3 days, and for these sessions, microdialysis probes were lowered into guide cannulae implanted on opposite hemispheres. In one session, we assayed for basal DA or 5HT content using no net-flux procedures (counterbalanced across animals within genotype). In the second session, mice were assayed for the basal content of the other neurotransmitter. Another group of animals only underwent 1 microdialysis session in which we assayed for the effects of an acute injection of MA (2 mg/kg) upon monoamine levels and thus, a microdialysis probe was inserted unilaterally, with the hemisphere counter-balanced across animals. For the immunoblotting studies, half of the MAH/LDR mice were administered an acute injection of 2 mg/kg MA to examine for potential drug effects upon protein expression; the remaining half were administered an acute injection of saline for comparison. At 3 h post-injection, tissue was collected for processing by immunoblotting to be consistent with the experimental design of an earlier report examining for genotype differences in mRNA expression (Wheeler et al., 2009).

#### *In vivo microdialysis*

Microdialysis was conducted in repeated MA/SAL-treated B6 mice at 1 and 21 days withdrawal or in MAH/LDR mice using procedures very similar to those described recently in Ary et al. (2013). For each microdialysis session, a probe was inserted unilaterally into the mPFC or the NAC, counterbalancing across hemispheres between groups. The animals were then connected a liquid swivel (Instech; Plymouth Meeting, PA) and perfused (2μl/min) with microdialysis buffer (NaCl, 147 mM, CaCl2, 1.2 mM, KCl, 2.7 mM, MgCl2, 1.2 mM, Na2HPO4, 0.5 mM; adjusted to pH = 7.4). Following a 3-h equilibration period, dialysate collection began and occurred in 20-min intervals into vials containing 10μl of preservative (4.76 mM citric acid, 150 mM NaH2PO4, 50μM EDTA, 3 mM sodium dodecyl sulfate, 10% methanol (v/v), 15% acetonitrile (v/v), pH = 5.6). The duration of testing ranged from 3–4 h, depending upon the experiment (see below). Upon completion of a session, animals were lightly restrained for probe removal and the dummy cannula reinserted. It should be noted that while each *in vivo* microdialysis study commenced with *n* = 10–12 mice (depending upon the study), we were not always able to obtain data from all animals from the first *in vivo* microdialysis sessions due to a loss of probe patency during sampling, misplaced guide cannula or HPLC malfunction. The final sample sizes indicated in the Results below reflect the final number of animals for which we successfully obtained all samples for the session and for which the probes were localized within the intended region. While not always possible for additional technical reasons (e.g., dislodged cranial implant; clogged cannulae; *n* = 2–4/replicate), a second microdialysis session was conducted by inserting a probe into the guide cannula implanted in the opposite hemisphere. This 2nd session occurred either at 21 days withdrawal from repeated MA/SAL (B6 mice) or a minimum of 3 days following the 1st session (MAH/LDR mice). Thus, of the total number of mice at the outset of each study, we successfully obtained data from both microdialysis sessions for approximately 70% of the subjects tested and this percentage ranged depending upon the study and the number of technical issues that were encountered at the time of sample collection and HPLC analyses.

All microdialysis studies commenced with a 1-h baseline sampling period. To examine the effects of MAH/LDR genotype and of repeated MA treatment on MA-stimulated dopamine release, mice were injected with 2 mg/kg MA and dialysate was collected for an additional 3 h. For the study of repeated MA effects or MAH/LDR genotype differences in basal extracellular dopamine content, we employed quantitative *in vivo* microdialysis procedures, in which increasing concentrations of DA (Sigma Aldrich) were diluted in aCSF to concentrations encapsulating biological levels (0, 2.5, 5, and 10 nM; e.g., Sam and Justice, 1996) and perfused in ascending order for 1 h each. The basal extracellular serotonin (5HT) content was also assayed in MAH/LDR mice (0–10 nM; Sigma Alrich). Linear regression analyses were employed to calculate the point of no net-flux (*y* = 0) and the slope of the plot (i.e., the extraction fraction or Ed; an index of neurotransmitter release and reuptake; e.g., Sam and Justice, 1996), which were analyzed using between-subjects ANOVAs. To relate the MA-induced changes in extracellular dopamine (DAEC) observed in B6 mice to drug effects on DAT and D2 autoreceptor function, we assayed the effects of reverse dialyzing the DAT inhibitor GBR-12909 (0, 1, 10, and 100μM; Pierce and Kalivas, 1997) or the D2-like DA receptor antagonist (±) sulpiride (0, 50, 100μM; Engleman et al., 2004). In the studies of D2-like receptor function, sulpiride was infused *in lieu* of a D2R agonist, as the results of published studies and unpublished pilot studies in our laboratory have failed to reliably detect changes in DAEC using this approach (e.g., Galloway et al., 1986; Santiago et al., 1993; Liu and Steketee, 2011), while the local infusion of D2R antagonists more reliably alter DAEC (Engleman et al., 2004; present study). All compounds were dissolved in microdialysis buffer, although sulpiride required initial dissolution in 50μL of acetic acid and the final pH ranged from 6.9–7.3, depending upon the replicate. Each drug concentration was infused for 1 h, akin to the quantitative microdialysis procedures above. Given the known differences in basal DA levels confirmed by no net-flux microdialysis and the susceptibility of these procedures to differences in probe recovery (Westerink and Cremers, 2007), all data for MA-induced neurotransmitter release and for reverse dialysis experiments were expressed as a percent change from baseline. The microdialysis data were analyzed using ANOVAs with repeated measures across the Time or Dose factors and interactions deconstructed for simple effect analyses, followed by *post-hoc* comparisons using *t*-tests.

### *HPLC Analysis*

Dopamine and serotonin in 27μl dialysate was measured using high pressure liquid chromatography (HPLC) with electrochemical detection using an ESA Coularray HPLC system (ESA Inc.; Bedford, MA). For this, MD-TM mobile phase was employed (Thermo Dionex), and monoamine neurotransmitters were separated using an MD-150 × 3*.*2 column (C18, 150 mm × 3.2 mm I.D.; Thermo Dionex). An ESA 5014B analytical cell with two electrodes was used for the electrochemical detection of monoamines—the reduction analytical electrode (E1, −150 mV), and an oxidation analytical electrode (E2, +220 mV). Dopamine and serotonin content in each sample was analyzed by peak height and compared to external standard curves (one for each neurotransmitter) for quantification (e.g., Szumlinski et al., 2007). Unfortunately, our HPLC conditions did not permit reliable detection of norepinephrine in the dialysate.

#### **IMMUNOBLOTTING**

Immunoblotting was conducted on whole-cell tissue homogenate collected from the mPFC, NAC shell and NAC core of B6 mice at either 1 or 21 days withdrawal from repeated MA treatment (10 × 2 mg/kg) or of MAH/LDR mice 3 h following an acute injection of SAL or 2 mg/kg MA (to be consistent with the experimental design of a prior assay of mRNA levels; Wheeler et al., 2009). To relate MA's effects upon indices of DAEC in B6 and MAH/LDR mice, we examined for the total protein expression of DAT and D2 dopamine receptors (D2Rs), the latter of which serves as an autoreceptor regulating DAEC. We also related MA's effect upon indices of 5HTEC in MAH/LDR mice to the total protein expression of SERT and 5HT1B receptors (5HT1BRs), the latter of which serves as an autoreceptor on 5HT terminals. The general procedures used to dissect and homogenize tissue, as well as to separate, transfer and visualize proteins were described recently (Ary et al., 2013). The PFC, NAC shell and NAC core were excised over ice and frozen at −80◦C until assay. Tissue was homogenized with a PTFE hand-held tissue grinder in homogenization medium consisting of 0.32 M sucrose, 2 mM EDTA, 1% sodium dodecyl sulfate, 50μM phenyl methyl sulfonyl fluoride, and 1μg/ml leupeptin (pH = 7.2) containing a protease inhibitor (Complete Mini, Roche; Indianapolis, IN) and phosphatase inhibitor cocktail (Sigma). Samples were then subjected to low-speed centrifugation (2000 g). Protein determinations were performed using the Bio-Rad DC protein assay (Bio-Rad; Hercules, CA). Samples (30μg total protein) were subjected to a sodium dodecyl sulfate-polyacrylamide gradient gel (4–12% Bis-Tris or 3–8% Tris-Acetate Invitrogen; Carlsbad, CA) electrophoresis, transferred via standard apparatus (Bio-Rad) to nitrocellulose membrane. Depending upon the study, the total protein content of DAT, D2Rs, SERT, and 5-HT1BRs were determined by immunoprobing using the following rabbit polyclonal antibodies: anti-D2R (1:333–1:500, Novus Biologicals), anti-DAT (1:333–1:1:500, Millipore), anti-SERT (1:500, Millipore), anti-5HT1BR (1:250–1:500; Lifespan Biosciences). Anti-Calnexin (1:1000, Millipore) was used to control for protein loading and to normalize the expression of the protein of interest. Immunolabeled proteins were detected using horseradish peroxidaseconjugated secondary IgGs (diluted 1:20,000–1:80,000) (Jackson Immuno Research) and visualized with enhanced chemiluminescence (Amersham Life Sciences). Immunoreactive levels were quantified by integrating band density × area using computerassisted densitometry (NIH ImageJ version 1.60). The density × area measurements were averaged over 3–4 control samples for each gel and all bands were expressed as percent of the control values (SAL-1 day withdrawal for the repeated MA study; SAL-F2B6D2 mice for the genetic study). The immunoblotting data were analyzed using ANOVAs, followed by analyses for main effects and *t*-tests for *post-hoc* comparisons, when appropriate.

#### **MA-INDUCED PLACE-CONDITIONING**

Groups of B6 mice were implanted with guide cannulae above the mPFC or NAC and then subjected to a MA place-conditioning regimen that was similar to that employed previously for stimulants in our laboratory (Ary et al., 2013). The study proceeded in five sequential phases: habituation, pre-conditioning test (Pre-Test), MA/SAL conditioning, post-conditioning test (Post-Test), and microinjection test (Microinjection Test). All sessions were 15 min in duration and animals received no systemic injections during the habituation, Pre-Test, Post-Test or Microinjection Test, when they had free-access to both compartments of the apparatus. For conditioning, mice received 4 alternating pairings of distinct compartments with either MA (2 mg/kg; vol = 0.01 ml/kg) or an equivalent volume of saline in an unbiased fashion. One compartment had black and white marble-patterned walls, with a textured floor, while the other compartment had wood-patterned walls, with a smooth Plexiglas floor. The difference in the time spent in the drug-paired vs. unpaired compartment (CPP Score) on the Post-Test served to index place-conditioning prior to any intracranial manipulation. Having established that MA elicited a conditioned place-preference (CPP), mice were assigned to receive an intracranial infusion (0.5 μl/side) of 100 nM GBR-12909, 100 nM of the D2/3R agonist quinpirole or an aCSF vehicle. The doses of GBR-12909 and quinpirole were selected to be maximally effective for raising and lowering, respectively, extracellular dopamine in on-going microdialysis studies in our laboratory. Microinfusions were delivered at a rate of 0.5 ul/min via 33-gauge stainless steel tubing (12 mm in length). Microinjectors were left in place for an additional 1 min prior to careful removal. Five min later, mice were placed into the place-conditioning apparatus for 15 min. Following testing, microinjector placements within the mPFC or NAC were verified in Nissl-stained coronal tissue sections (**Figure 5**). The data were analyzed using ANOVAs.

## **RESULTS**

#### **DA SENSITIZATION IN MA-TREATED B6 MICE**

We characterized the short- and longer-term effects of a history of subchronic MA upon MA-induced DA release within NAC and mPFC in B6 mice. As illustrated in **Figure 1**, acute MA (2 mg/kg) elicited a rise in extracellular dopamine that exhibited a clear sensitization in MA-treated animals (**Figure 1**; compare open vs. closed symbols in each panel) [for NAC, Repeated Treatment × Time: *F*(11*,* 297) = 4*.*40, *p <* 0*.*0001; for mPFC: Repeated Treatment × Time: *F*(11*,* 319) = 2*.*06, *p* = 0*.*02]. Although the magnitude and time-courses of this effect varied between regions, repeated MA sensitized DA release in both regions in a time-independent fashion as indicated by no main or interaction effects of the Withdrawal factor for either region (*p*'s *>* 0.20).

#### **BASAL DA CONTENT IN MA-TREATED B6 MICE**

When obtained by conventional microdialysis, the average basal DAEC levels within both the NAC and mPFC were moderately lower in MA-treated mice at 21 days withdrawal (**Table 1**). However, statistical analyses of the data failed to identify any significant main or interaction effects (all *p*'s *>* 0.06). As the results from conventional microdialysis procedures are subject to differences in probe recovery, we then employed quantitative microdialysis procedures to examine the data for MAinduced changes in basal DAEC and DA clearance from the probe. Using no net-flux procedures and linear regression analyses (**Figures 2A,D**), we observed time-dependent changes in basal DA*EC* content (*y* = 0) in MA-sensitized mice within both the NAC (**Figure 2B**) [Treatment × Withdrawal: *F*(1*,* 29) = 4*.*162, *p* = 0*.*05] and the mPFC (**Figure 2E**) [Treatment × Withdrawal: *F*(1*,* 21) = 4*.*767, *p* = 0*.*04]. SAL-MA differences in *y* = 0 were not present in either region at 1 day withdrawal (*t*-tests, p's *>* 0.05), but group differences were apparent at 21 days withdrawal [for NAC, *t*(11) = 3*.*08, *p* = 0*.*01; for mPFC, *t*(11) = 3*.*13, *p* = 0*.*01]. The MA-induced reduction in DAEC content was unrelated to changes in release/reuptake, as group differences were

**Table 1 | When assayed using conventional** *in vivo* **microdialysis procedures, we failed to detect an effect of repeated methamphetamine experience or withdrawal upon basal extracellular dopamine (fg/20**μ**l dialysate) within either the mPFC or the NAC.**


*The data represent the mean* ± *s.e.m. of the number of animals indicated in parentheses.*

not observed regarding Ed derived from the slopes of the linear regressions for either region (**Figures 2C,F**) [for NAC, Treatment effect: *F*(1*,* 29) = 7*.*48, *p* = 0*.*01; no Withdrawal main or interaction effect, p's *>* 0.15; for mPFC, Treatment: *F*(1*,* 21) = 4*.*81, *p* = 0*.*04; Withdrawal: *F*(1*,* 21) = 15*.*831, *p* = 0*.*001; interaction: *p* = 0*.*27].

#### **DAT EXPRESSION AND FUNCTION IN MA-TREATED B6 MICE**

We next related MA-induced changes in DAEC to the expression and function of DAT using microdialysis and immunoblotting approaches. MA withdrawal did not affect the capacity of GBR-12909 to elevate DAEC levels within either the NAC (**Figure 3A**) or the mPFC (**Figure 3C**) (for both regions, Dose effects: *p*'s *<* 0.001; no main or interaction effect of the Repeated Treatment factor: *p*'s *>* 0.05). MA treatment also did not alter DAT expression within the NAC shell (not shown) or mPFC (**Figure 3D**; Two-Way ANOVA's, *p*'s *>* 0.05). However, drug treatment reduced DAT expression within the NAC core (**Figure 3B**), although the results did not support a time-dependency to this effect [Repeated Treatment effect: *F*(1*,* 39) = 4*.*26, *p* = 0*.*05; no Withdrawal effect or interaction, *p >* 0*.*20].

#### **D2R EXPRESSION AND FUNCTION IN MA-TREATED B6 MICE**

We then related MA-induced changes in DAEC to the expression and function of D2/3Rs using microdialysis and immunoblotting approaches. MA withdrawal blunted D2/3R function within the NAC at both withdrawal time-points (**Figure 4A**) [1 day: Dose × Repeated Treatment: *F*(1*,* 14) = 11*.*18, *p* = 0*.*005; *t*-tests; 21 days: effects of Dose and Treatment: *p*'s *<* 0.006, but no interaction: *p* = 0*.*27]. MA animals also exhibited reduced receptor expression at the 21-day withdrawal time-point within the NAC core (**Figure 4B**) [Repeated Treatment × Withdrawal (α = 0*.*1 based on microdialysis results): *F*(1*,* 38) = 3*.*739, *p* = 0*.*06; *post-hoc*, 1 day WD, Repeated Treatment: *p* = 0*.*68; 21 days WD, *F*(1*,* 18) = 5*.*378, *p* = 0*.*03]. No change in D2R expression was observed within the NAC shell (Two-Way ANOVAs, α = 0*.*1, all *p*'s *>* 0.25). While MA history did not impact the sulpiride-induced rise in DA within the mPFC at 1-day withdrawal [Dose effect:

*F*(1*,* 13) = 19*.*12, *p <* 0*.*001; no main or interaction effect of the Repeated Treatment factor, *p*'s *>* 0.10], MA reduced responsiveness to the 100μM dose in the long-term (**Figure 4C**) [Dose × Repeated Treatment: *F*(1*,* 14) = 8*.*56, *p* = 0*.*01; *t*-tests: *p*'s *<* 0.05]. However, we failed to detect changes in D2R expression within the mPFC (**Figure 4D**; Two-Way ANOVA's, all *p*'s *>* 0.05).

#### **MA-INDUCED CPP IN B6 MICE**

The data presented above indicated that subchronic MA exposure was sufficient to produce enduring anomalies in DAEC within both the mPFC and NAC. As the role for DAEC in mediating MA preference has not been fully vetted, we examined the effects of raising (via site-directed infusions of the DAT inhibitor GBR-12909) or lowering (via site-directed infusions of the D2/3 autoreceptor agonist quinpirole) DAEC upon the expression of a MA-conditioned place-preference. Neuropharmacological manipulation of the NAC impacted the expression of a MA-induced place-conditioning in B6 mice (**Figure 5A**) [Side × Test × Drug: *F*(2*,* 21) = 18*.*60, *p <* 0*.*0001]. In the absence of intra-NAC infusion (Post-test), there were no group differences in CPP magnitude (Side × Treatment, *p >* 0*.*50). However, group differences emerged with respect to both the extent and direction of place-conditioning upon intra-NAC microinjection [Side × Treatment: *F*(2*,* 21) = 48*.*49, *p <* 0*.*0001]. Vehicle-infused animals exhibited a non-significant CPP [Side effect: *p* = 0*.*11], GBR12909 facilitated CPP expression [Side effect: *F*(1*,* 8) = 49*.*00, *p <* 0*.*0001] and quinpirole elicited a marked conditioned place-aversion (CPA) [Side effect: *F*(1*,* 7) = 48*.*52, *p <* 0*.*0001]. An analysis of CPP scores for the Microinjection Test confirmed greater CPP in GBR12909-infused animals vs. vehicle controls [*F*(2*,* 21) = 48*.*49, *p <* 0*.*0001; *t*-tests, *p*'s *<* 0.05]. In contrast, intra-mPFC DA manipulations failed to alter CPP expression across 2 replicates of study (**Figure 5B**) [Side effect: *F*(1*,* 39) = 32*.*72, *p <* 0*.*0001; Test effect: *F*(1*,* 39) = 11*.*85, *p <* 0*.*0001; no main or interaction effect of the Drug factor, *p*'s *>* 0.30].

#### **MONOAMINE CONTENT IN MAH/LDR MICE**

The data for MA-injected B6 mice indicate that a history of subchronic MA exposure is sufficient to produce enduring alterations in basal DAEC within both the NAC and mPFC (**Figure 2**). Moreover, the neuropharmacological results

supported an important role for DAEC, particularly within the NAC, in mediating MA-conditioned preference and aversion (**Figure 5**). Thus, we determined whether or not the divergent behavioral phenotypes of MAH/LDR mice might relate to differences in basal DAEC content. We also examined for differences in basal 5HTECcontent, as a prior examination for biochemical correlates of genetic vulnerability to high MA intake indicated higher expression of the mRNA encoding the serotonin transporter SERT within the NAC of MAHDR vs. MALDR mice (Wheeler et al., 2009). Using no net-flux approaches, MAHDR-MALDR differences were noted for NAC basal DAEC content (**Figure 6A**) [*t*(15) = 2*.*50, *p* = 0*.*02], with levels being lower in MAHDR vs. MALDR mice. mPFC DAEC content also varied with genotype (**Figure 6B**) [*t*(15) = 2*.*41, *p* = 0*.*03] and again, MAHDR animals exhibited lower DA content. While no genotypic differences were noted for mPFC 5HTEC content (**Figure 6D**; *t*-test, *p* = 0*.*92), NAC 5HTEC content varied with genotype (**Figure 6C**) [*t*(14) = 2*.*89, *p* = 0*.*01], with MAHDR mice exhibiting higher serotonin levels than MALDR animals. No genotypic differences

were observed for the Ed for either DA or 5HT within either brain region (**Table 2**; One-Way ANOVA's, all *p*'s *>* 0.35). Thus, the genotypic differences in extracellular neurotransmitter content were not obviously related to neurotransmitter clearance/release.

#### **MA-STIMULATED MONOAMINE RELEASE IN MAH/LDR MICE**

The results of the quantitative microdialysis studies indicated line differences for basal DAEC and 5HTEC content (**Figure 6**). Thus, we examined also for line differences in the capacity of an acute MA injection (2 mg/kg) to elevate DA and 5HT levels within the NAC and mPFC. A summary of the average baseline levels of DA and 5HT within the NAC and mPFC is provided in **Table 3**. As the absolute amount of neurotransmitter detected by conventional microdialysis procedures is subject to influences by individual probe recovery (Westerink and Cremers, 2007), the results obtained under conventional microdialysis procedures did not match exactly those obtained under quantitative microdialysis procedures. Notably, our conventional microdialysis procedures did not detect line differences in basal NAC neurotransmitter

levels (*t*-tests, *p*'s *>* 0.10). However, we did detect lower DAEC and higher 5HTEC within the mPFC of MAHDR vs. MALDR mice [for DA, *t*(14) = 2*.*18, *p* = 0*.*04; for 5HT, *t*(14) = 3*.*63, *p* = 0*.*003].

Surprisingly, no line differences were observed for MAstimulated DA release within the NAC (**Figure 7A**) [Time effect: *F*(11*,* 154) = 5*.*09, *p <* 0*.*0001; interaction: *p* = 0*.*85]. In contrast, marked differences were observed for MA-induced DA release in the mPFC (**Figure 7B**) [Genotype × Time: *F*(22*,* 154) = 3*.*12, *p* = 0*.*001]. As illustrated in **Figure 7B**, 2 mg/kg MA injection produced a robust (2–3-fold increase) in mPFC DA levels in MAHDR mice [F(11*,* 66) = 2*.*33, *p* = 0*.*02]. In contrast, the MA injection did not elevate mPFC DA levels at all in MALDR mice; rather, their DA levels dropped below baseline post-injection [*F*(11*,* 88) = 4*.*82, *p <* 0*.*0001]. Acute MA failed to alter 5HT levels in the NAC of either genotype (**Figure 7C**; Two-Way ANOVA, all *p*'s *>* 0.05). However, there was an overall genotypic difference for MA-induced increases in mPFC 5HT (**Figure 7D**) [Genotype effect: *F*(1*,* 14) = 5*.*49, *p* = 0*.*03; no main or interaction effects of Time: *p*'s *>* 0.20], with MALDR mice exhibiting a 2 to 2.5 fold elevation in mPFC 5HT levels post-injection that persisted throughout the microdialysis session and MAHDR mice exhibiting no sign of MA-induced 5HT release (**Figure 7D**).

#### **IMMUNOBLOTTING IN MAH/LDR MICE**

Given genotype differences in basal and MA-stimulated neurotransmitter release (**Figures 6**, **7**), we next employed immunoblotting to index the expression of DAT, SERT, D2R, and 5HT1BR in the selected lines (**Figure 8**). The results of the statistical analyses for all proteins failed to indicate any main or interaction effects of the Treatment factor (all *p*'s *>* 0.05). Thus, the data were collapsed across treatments for clarity of presentation. In the NAC shell (**Figure 8A**), we observed genotypic differences in D2R [Genotype effect: *F*(1*,* 37) = 10*.*63, *p* = 0*.*003], DAT [Genotype effect: *F*(1*,* 37) = 13*.*53, *p* = 0*.*001]

**Table 2 |** *In vivo* **extraction fractions (***Ed* **) for dopamine and serotonin in the medial prefrontal cortex (mPFC) and the nucleus accumbens core-shell interface (NAC) of mice selectively bred for high vs. low methamphetamine drinking (respectively, MAHDR, and MALDR), as determined using quantitative microdialysis approaches.**

animals indicated in parentheses in each panel. ∗*p <* 0*.*05 vs. MALDR.


*The data represent the mean* ± *s.e.m. of the number of animals indicated in parentheses.*

and SERT [Genotype effect: *F*(1*,* 37) = 4*.*90, *p* = 0*.*03] expression, with MAHDR mice exhibiting lower D2R levels, but higher transporter levels, vs. MALDR animals. Genotypic differences were not observed for NAC shell 5HT1BR expression. In contrast MAHDR mice exhibited higher DAT and SERT levels relative to MALDRalso varied with genotype within the NAC shell, but this effect did not reach statistical significance [Genotype effect: *p* = 0*.*06]. SERT levels varied significantly with genotype in the NAC shell [Genotype effect: *F*(2*,* 57) = 4*.*79, *p* = 0*.*01], with MAHDR mice exhibiting higher expression vs. the other genotypes (LSD *post-hoc* tests, *p*'s *<* 0.02). No genotypic difference in 5HT1BR was observed in the NAC shell (Genotype effect, *p >* 0*.*35). In the NAC core (**Figure 8B**), MAHDR mice exhibited **Table 3 | Basal extracellular levels of dopamine and serotonin (in fg/27 µl sample) within the medial prefrontal cortex (mPFC) and the nucleus accumbens (NAC) of mice selectively bred for high vs. low methamphetamine drinking (MAHDR and MALDR, respectively), as determined using conventional microdialysis approaches.**


*The data represent the mean* ± *s.e.m. of 3 dialysate samples, collected during the hour prior to a methamphetamine injection. The number of animals employed in the statistical analyses of the data is indicated in parentheses. \*p < 0.05 (t-test).*

**and serotonin (C,D) exhibited within the NAC core-shell interface (left) and mPFC (right) of genetically heterogeneous B6 × D2 mice (F2B6D2) and mice selectively bred for high vs. low MA intake (MAHDR and MALDR, respectively) administered an acute injection of 2 mg/kg MA.** The data are expressed as a percent change from the average baseline values and represent the means ± s.e.m's of the number of animals indicated in parentheses in each panel. ∗*p <* 0*.*05 vs. average baseline.

lower 5HT1BR expression but higher DAT expression than MALDR animals [for 5HT1BR, Genotype effect: *F*(1*,* 38) = 6*.*13, *p* = 0*.*02; for DAT, Genotype effect: *F*(1*,* 38) = 5*.*46, *p* = 0*.*03]. Genotypic differences were not noted for D2R or for SERT expression within the NAC core (Genotype effects, *p*'s *>* 0.10). In the mPFC (**Figure 8C**), we observed no line differences for the D2R (Genotype effect: *p* = 0*.*29). Overall, MAHDR exhibited higher 5HT1B levels than MALDR mice, but this difference was shy of statistical significance (Genotype effect: *p* = 0*.*07).

**receptor, the dopamine transporter (DAT), the 5HT1B receptor and the serotonin transporter (SERT) of genetically heterogeneous B6 × D2 mice (F2B6D2) and mice selectively bred for high vs. low MA intake (MAHDR**

clarity of presentation of genotypic differences. The data represent the means ± s.e.m's of the number of animals indicated in parentheses in each panel. ∗*p <* 0*.*05 vs. MALDR.

and the rise in 5HT1BR expression observed in MAHDR. there was a moderate genotypic difference in 5HT1BRs that reflected lower levels in both selected lines, compared to F2B6D2 mice [Genotype effect: *F*(2*,* 59) = 2*.*99, *p* = 0*.*06]. A similar pattern of genotypic differences were observed for the D2 receptor [Genotype effect: *F*(2*,* 59) = 4*.*68, *p* = 0*.*01], with F2B6D2 mice exhibiting significantly higher receptor levels, compared to both selected lines (LSD *post-hoc* tests, *p*'s *<* 0.05). We could not detect DAT within the mPFC of the mice in this study, and no differences were noted for SERT expression (Genotype effect: *p >* 0*.*45).

#### **DISCUSSION**

An understanding of the neurobiological substrates of MA addiction vulnerability and the effects of subchronic, subtoxic, MA experience upon the brain is critical to understanding the etiology of MA addiction, identifying potential biomarkers for MA addiction vulnerability/resiliency and developing treatment strategies for early intervention in the disease process. The present studies were conducted to further extend knowledge regarding the interactions between subchronic MA exposure and forebrain DA and to relate anomalies in forebrain DA and 5HT to genetic vulnerability/resiliency to high MA intake.

#### **SUBCHRONIC MA ELICITS A TIME-INDEPENDENT SENSITIZATION OF CORTICOLIMBIC DA**

Regional differences existed in MA's ability to elicit DA release within the 2 major terminal regions of the mesocorticolimbic DA system and regional differences were apparent with respect to the time-course of DA release in both acute and repeated MA-treated animals (**Figure 1**). As reported previously (Shoblock et al., 2003), MA elicited a markedly larger rise in DAEC within the NAC than within the mPFC, although the MA-induced elevation in DAEC produced within the mPFC was more persistent than that observed within the NAC. This regional distinction in the DA response to MA might relate to regional differences in DAT expression, which is higher in striatal vs. frontal cortical structures (Sesack et al., 1998). Higher DAT expression within the NAC could account for the larger magnitude of MA effect and the faster rate of decline in DAEC observed in the NAC, relative to the mPFC. Regional differences exist also in the relative roles played by metabolizing enzymes, monoamine oxidase (MAOs) and catechol-O-methyltransferase (COMT), in determining DAEC (e.g., Karoum et al., 1994). MAO can be inhibited by MA (e.g., Fleckenstein et al., 2007; Chen et al., 2009) and by virtue of the fact that the majority of DA released within striatal structures is removed into neuron terminals by DAT (Sesack et al., 1998), MAOs play a more critical role in regulating DAEC within striatum than they do within frontal cortex, where DAT expression is relatively low (Karoum et al., 1994). However, the fact that the MA-induced rise in DA*EC* was less persistent within NAC than within mPFC (**Figure 1**), argues less in favor of a role for MA inhibition of MAO as a major contributing factor to the rise in NAC DA observed in acute MA-injected animals.

Regardless of regional differences in the time-course of MAstimulated DA release, a subchronic history of MA was sufficient to elicit DA sensitization within both the NAC and the mPFC of male B6 mice. It is notable that in both the cases of the NAC and the mPFC, two features of the time-course of MA-induced DA release varied as a function of MA experience: the magnitude of the initial rise and the persistence of the rise, particularly during the last hour post-injection. Markedly apparent for both regions, MA-experienced animals exhibited a higher initial rise in DAEC post-injection than did animals acutely administered drug. Such findings suggest the repeated MA experience may increase the amount or function of plasma membrane or vesicular transporters or in the availability of vesicular DA for release. While we did not assay for changes in the levels of the vesicular transporter, we did not detect any obvious relation between sensitized DA release and the protein levels of DAT within either the NAC or mPFC. Moreover, we failed to detect SAL-MA differences in the rise in DAEC produced by infusion of GBR-12909, which depends upon both the availability of DAT for binding and the integrity of impulse-dependent DA release mechanisms. Thus, at the present time, it would not appear that alterations in DAT function/expression or in mechanisms regulating the releasability of DA contribute significantly to the sensitization of the initial rise in DAEC observed in MA-experienced mice, although these mechansims were cannot be vetted thoroughly using *in vivo* microdialysis and conventional immunoblotting methods. The fact that the rise in DAEC elicited by the MA challenge injection was more persistent in MA-experienced vs. acutely treated animals (**Figure 1**) suggests that perhaps repeated drug experience lowered DA catabolism. As mentioned above, MA inhibits MAO (Fleckenstein et al., 2007) and the possibility exists that with repeated drug experience, this mechanism may contribute to this drug's capacity to promote higher DAEC levels, particularly within the NAC (Popova et al., 2004). In contrast to striatum, frontal cortical structures exhibit low DAT expression (Sesack et al., 1998). As such, DAT and MAOs play less of a role in DA catabolism within frontal cortex than they do within subcortical regions (e.g., Karoum et al., 1994) and DA catabolism is mediated in large part by COMT, particularly under conditions of elevated DAEC such as those produced by MA treatment (e.g., Karoum et al., 1994; Huotari et al., 2002a,b; Matsumoto et al., 2003). While speculative at this time, the possibility exists that repeated MA treatment reduces also the function of COMT, via some indirect mechanism, that may promote the amount of DAEC. Given the reliance of mPFC DAEC upon COMT, this mechanism would be predicted to impact the duration of the DA response to MA more so in this region than within NAC. However, arguing against a major role for drug-induced deficits in COMT in mediating the sensitization of MA-induced DA release is evidence that neither pharmacological inhibition of COMT nor null COMT mutation significantly impact amphetamine-induced increases in DAEC within striatum or frontal cortex (Törnwall and Männistö, 1993; Törnwall et al., 1993; Tuomainen et al., 1996; Gogos et al., 1998; Männistö and Kaakkola, 1999; Huotari et al., 2002a,b).

Irrespective of the mechanisms at play, the fact that subchronic dosing with subtoxic MA elicited sensitization within both the NAC and mPFC is a finding in line with earlier reports for MA-experienced rodents (e.g., Stephans and Yamamoto, 1995; Zhang et al., 2001; Broom and Yamamoto, 2005; Lominac et al., 2012; Lacan et al., 2013; Le Cozannet ´ et al., 2013; but see Ago et al., 2006, 2007, 2012). Moreover, the DA sensitization was time-independent, manifesting at 1 day post-treatment and persisting, unchanged, for at least 21 days (**Figure 1**). This finding distinguishes MA-induced DA sensitization from that produced by repeated cocaine or amphetamine, the latter two of which tends to grow with the passage of time during withdrawal (e.g., Paulson and Robinson, 1995; Vanderschuren and Kalivas, 2000). Nevertheless, the present results for MA-injected B6 mice are qualitatively similar to results from relatively recent studies, in which rats with a history of behavior-contingent vs. non-contingent intravenous MA exposure displayed MA-induced DA sensitization that manifested early in withdrawal (Lominac et al., 2012; Lacan et al., 2013; ´ Le Cozannet et al., 2013). While requiring further study, particularly with respect to MA pharmacokinetics (see Segal and Kuczenski, 2006 for Discussion), the capacity of repeated MA to elicit time-independent DA sensitization within the NAC (and perhaps also within the mPFC) is qualitatively similar across rodent species, routes of delivery and contingency of delivery, which renders non-contingent models of MA administration well-suited for the study of the psychobiological consequences of subchronic MA exposure of relevance to MA abuse and the development of addiction (see also Lacan et al., ´ 2013).

Peculiarly, the expression of MA-sensitized DA release within neither the NAC nor the mPFC of male B6 mice was not obviously related to drug effects upon basal DAEC (**Table 4**). First, changes in DAEC were only apparent in the long-term (**Figure 2**), while DA sensitization was manifest in early withdrawal. Second, MA history oppositely affected DAEC in the mPFC (increase) and NAC (decrease), despite eliciting DA sensitization within both regions (**Table 4**). DA transmission within the NAC is highly implicated in mediating the incentive motivational properties of drugs and conditioned stimuli, as well as those for natural reinforcers, (e.g., Berridge and Robinson, 1998; Di Chiara, 1999; Robinson and Berridge, 2008; Wise, 2008; Blum et al., 2012). Indeed, the data from our neuropharmacological study of the NAC supports a bi-directional role for NAC DA in regulating the motivational valence of MA-conditioned environments, with elevated DA promoting the expression of conditioned approach and reduced DA eliciting conditioned avoidance (**Figure 5A**). In contrast, neuropharmacological manipulations of mPFC DA did not impact the magnitude or direction of conditioned behavior in our place-conditioning paradigm (**Figure 5B**). Such findings indicate that mPFC DAEC, particularly that within the prelimbic cortex (see histology in **Figure 5B**), does not actively regulate the recall of a drug-context associations or the conditioned incentive motivational properties of a drug-paired environment. This contrasts with the role for mPFC DA role in the acquisition of place-conditioning reported previously (e.g., Wilkinson et al., 1998; Hayen et al., 2014). The divergent effects of subchronic dosing with MA upon basal DAEC within the NAC and mPFC indicate that distinct cellular or molecular mechanism(s) underpin the changes in basal vs. stimulated DA release in suchronic MA-treated animals within corticolimbic DA terminals. Whatever these mechanism(s) are that operate within the NAC and mPFC to impact DAEC in the drugged vs. undrugged state (see below), their dysregulation by a suchronic history of MA injections is regionally selective and temporally distinct.

#### **NAC DA, MA PREFERENCE AND GENETIC VULNERABILITY TO MA INTAKE**

High-dose MA injection regimens are well-characterized to induce neurotoxicity within DA neurons in dorsal striatum, while sparing DA neurons within the NAC (c.f., Carvalho et al., 2012). However, evidence from studies using more moderate MA treatment regimens, including that herein (**Figure 2B**), indicate that subchronic, subtoxic MA can also lower NAC basal DAEC. The reduction in NAC basal DAEC observed in male B6 mice herein is akin to that reported previously in male rats subjected to an identical MA injection regimen as that employed in the present study (Broom and Yamamoto, 2005). However, in contrast to this earlier study of rat, reduced basal DAEC was apparent in our male mice only in protracted withdrawal and was paralleled by reduced DAT expression, but no discernable change in DAT function (as assessed by either Ed or the DA response to GBR-12909 infusion) (**Table 4**). Reduced striatal DAT binding is observed consistently in imaging studies of human MA addicts and MA-experienced non-human primates, even during protracted withdrawal (e.g., McCann et al., 1998; Sekine et al., 2001, 2003; Volkow et al., 2001a,b; Johanson et al., 2006; Groman et al., 2012). However, in MA-injected rats, reduced NAC basal DAEC was reported to cooccur with *increased* DAT function and expression (Broom and Yamamoto, 2005). As this prior rat study did not examine for long-term changes in NAC DA levels or DAT expression/function, it is not known if MA-induced changes in DAT expression within the rat is biphasic with respect to time in withdrawal or if species differences exist for the long-term effects of repeated MA exposure upon NAC DA. Moreover, we and others have failed to detect pronounced changes in indices of NAC basal DA in rats with histories of intravenous MA (Schwendt et al., 2009; Lominac et al., 2012; Lacan et al., 2013; Le Cozannet et al., 2013 ´ ). Thus, the role played by route of administration in the manifestation of MAinduced anomalies in DAEC and DAT requires more systematic preclinical investigation.

In contrast to a recent study in non-human primates which failed to detect changes in D2R levels within the more ventral aspects of the striatum in MA-experienced subjects (Groman et al., 2012), our MA treatment regimen administered to male B6 mice reduced NAC D2R expression in long-term withdrawal (**Figure 4**). It is interesting to note that while this change in protein expression was late to manifest, an impairment in NAC D2/3R function was apparent very early in withdrawal prior to detectable changes in protein levels (**Figure 4**). Such data indicate a cause-effect relation between subchronic MA exposure and functional anomalies in D2/3Rs that are not necessarily related to gross alterations in total D2R protein expression, but could reflect changes in D3Rs, the latter of which we could not reliably detect

**Table 4 | Comparison of the dopamine effects of subchronic, subtoxic methamphetamine (MA) vs. saline (SAL) treatment and the dopamine phenotype of MA-naïve MAHDR vs. MALDR mice.**


*WD, withdrawal.* ↑ *denotes relative increase;* ↓ *denotes relative decrease;—denotes no change; n.d., denotes not determined.*

with our immunoblotting procedures. Alternatively, as we measured total protein expression, the possibility existed that drugelicited changes in the surface expression of proteins may have been masked by the study of whole tissue homogenate. D2/3Rs operate presynaptically as autoreceptors to inhibit both basal and impulse-dependent DA release (c.f. Ford, 2014). As such, the temporal relation between lowered D2/3R function and reduced DAEC is not obvious at the present time. Nevertheless, these data extend the results of non-human primate studies (e.g., Groman et al., 2012) by demonstrating a cause-effect relation between subchronic MA history and NAC D2R expression/function that is not afforded in imaging studies of humans. This demonstration is of high clinical relevance given the purported link between polymorphisms in the gene encoding D2R, D2R hypofunctioning and MA addiction vulnerability, as well as addiction severity, in human and non-human primates (c.f., Blum et al., 2012; e.g., Lee et al., 2009; Groman et al., 2012).

Related to this latter point, female MAHDR mice exhibited lower NAC D2R expression, compared to their MALDR counterparts and the reduced NAC D2R expression was accompanied by lower basal DAEC, but higher DAT levels within this region (**Table 4**). The former two observations are consistent with the effects of subchronic MA upon the NAC of male B6 mice observed herein (**Table 4**) and are more, rather than less, consistent with recent work in non-human primates correlating low basal D2R availability within striatum to subsequent MA-taking (Groman et al., 2012). The higher DAT expression observed in MAHDR vs. MALDR female mice is consistent with earlier results for MA-injected male rats, which was suggested to contribute to the low DAEC observed in MA-experienced animals (Broom and Yamamoto, 2005). Thus, drug-naïve or acute MA-injected female mice with a genetic predisposition to consume high amounts of MA exhibit DA anomalies within the NAC that are similar to, but not identical with, those reported in MA-experienced male rodents (**Table 4**). This later finding is interesting as earlier research clearly indicates that the striatum of female mice are much less sensitive to the neurotoxic effects of high-dose MA injection regimens, compared to males (Wagner et al., 1993; Yu and Wagner, 1994). Moreover, both female rodents (Morissette and Di Paolo, 1993a,b; Rivest et al., 1995; Bhatt and Dluzen, 2005; Ji and Dluzen, 2008) and humans (Lavalaye et al., 2000; Mozley et al., 2001; Staley et al., 2001) are reported to exhibit higher DAT expression or function than males. As we were limited in the total number and the sex of MAH/LDR animals available to study, it remains to be determined whether or not (1) MAHDR and MALDR mice differ in terms of DAT or DA autoreceptor function within the NAC (or other regions for that matter) and (2) the divergent D2R, DAT and DAEC observed in MAH/LDR females mice occur also in males. Moreover, we could not determine how differences in basal DAEC, DAT, and D2R contribute to the divergent behavioral phenotypes of MAH/LDR mice nor could we determine the extent to which line differences in indices of DA function interact with sex to influence behavior. Nevertheless, from the similarities in NAC DA exhibited by suchronic MA-treated male and genetically vulnerable female mice (see **Table 4**), our findings herein resonate with results of recent studies for non-human primates indicating that either drug-induced or idiopathic reductions in NAC D2R expression and basal DAECmay be critical biochemical triggers and/or predictors of subsequent high MA preference and intake (see Groman and Jentsch, 2013 for more detailed discussion). Extrapolating to the human condition, reduced striatal D2R binding reported in MA-addicted individuals (e.g., Volkow et al., 2001a; Lee et al., 2009) could very well reflect a combination of a pre-existing and drug-elicited hypo-DAergic state and further preclinical research is required in order to determine how idiopathies in ventral striatal D2R or basal DAEC predict individual variation in MA preference and intake of relevance to the development of an addicted state.

#### **PFC DA AND VULNERABILITY TO MA ADDICTION**

A subchronic history of subtoxic MA exposure increased mPFC DAEC in B6 mice (**Figure 2E**), a finding in line with reduced tissue DOPAC reported in rats with a history of intravenous MA self-administration under short-access, operant procedures (Schwendt et al., 2009). Also consistent with this prior work, the MA-induced rise in mPFC DAEC observed herein was found to be unrelated to indices of DAT function as we failed to detect saline-MA differences in the Ed for mPFC DA (an index of basal DA release/reuptake; Sam and Justice, 1996), total DAT expression or the capacity of intra-mPFC GBR-12909 infusions to elevate DAEC (**Figures 3C,D**). Such data argue that the rise in mPFC DAEC that manifests during protracted withdrawal from subchronic MA is not likely mediated by drug effects upon DAT and may very well reflect drug-elicited changes in other monoamine transporters, most notably NET. While we attempted to measure NET within the mPFC of our MA-injected mice to begin to address this possibility, the experiment was fraught with technical difficulties related to signal reliability of the available anti-NET primary antibodies, which prevented any firm conclusions on the matter. Future studies should assay the functional involvement of NET via the local infusion of selective reuptake inhibitors to probe the role for MA-induced changes in transporter function in the regulation of DAEC within mPFC.

Consistent with the earlier report for MA self-administering rats (Schwendt et al., 2009), as well as rats treated noncontingently with drug (Liu et al., 2009), our subchronic MA injection regimen did not alter mPFC D2R expression (**Figure 4D**), supporting the notion that drug-elicited changes in frontal cortical DAT and D2R binding reported in MA addicts (e.g., Volkow et al., 2001a,b; Sekine et al., 2003), likely require extensive MA treatment histories in order to manifest (see Lacan ´ et al., 2013 for Discussion). However, it is notable that MA-treated animals exhibited a reduced capacity of intra-mPFC sulpiride to elevate DAEC and this effect manifested only in protracted withdrawal (**Figure 4C**). Thus, the possibility exists that the elevated DAEC observed in the mPFC of MA-injected mice may reflect, at least in part, a progressive impairment in autoreceptor function within mPFC. To the best of our knowledge, these data are the first to characterize the effects of MA experience and withdrawal upon mPFC D2R function *in vivo* and we observe a cause-effect relation. Thus, while anomalies in frontal cortex function in MA addiction have been associated with low D2R availability in caudate (e.g., Volkow et al., 2001a), the present data indicate that subchronic drug experience is sufficient to produce local changes in D2/3R function and DAEC within mPFC (primarily prelimbic cortex) that are predicted to impact cognitive control over drug-taking and –seeking early in the addiction process. As discussed above for the NAC, D2R hypofunctioning is highly associated with addiction, as well as other disorders characterized by motivational anomalies (c.f., Blum et al., 2012; Jentsch and Pennington, 2014). Moreover, reducing D2R function in both drug-naïve humans and laboratory animals elicits inhibitory control deficits that are akin to those observed in the clinical condition (e.g., Lee et al., 2007, 2009; Herold, 2010; Groman et al., 2011, 2012). Thus, while our attempt to probe the functional relevance of MA-induced changes in mPFC DAEC and D2Rs failed to support a critical role for the expression of MA-conditioned approach/avoidance during early withdrawal (**Figure 5B**). Preliminary work from our laboratory indicates that the expression of MA-induced CPP in mice is highly resistant to extinction, persisting for weeks, even in the face of daily extinction training (Cohen, Barrett and Szumlinski, unpublished data). This raises the possibility that the impairment in mPFC D2/3R function produced by subchronic MA experience might underpin a deficiency in learning to inhibit forward approach behavior toward stimuli previously associated with MA and this possibility will be a topic of future research in our laboratory.

While low D2R availability within striatal structures is highly implicated in MA addiction vulnerability (c.f., Blum et al., 2012; Groman and Jentsch, 2013), there is little data supporting baseline D2R availability in frontal cortex with predisposition to addiction. In the present study, we did not detect any obvious relation between total D2R expression within mPFC and genetic vulnerability to high MA intake/preference (**Figure 8C**) and we were unsuccessful in our attempts to reliably detect DAT expression in the selected lines. Nevertheless, we did observe line differences in both basal and MA-stimulated DAEC within the mPFC, which may of relevance to their divergent phenotypes. Opposite to MA-experienced B6 mice, MAHDR mice exhibited lower mPFC basal DAEC; however, both MA-experienced and MAHDR animals exhibited a more pronounced rise in MA-induced DA release within mPFC than their respective controls (**Table 4**). As DA release within mPFC contributes to the acquisition of Pavlovian and instrumental associations (e.g., Wilkinson et al., 1998), line differences in the DA responsiveness of the mPFC to acute MA might account, at least in part, for their divergent phenotypes when assessed in MA-induced place-preference and operant self-administration paradigms (Wheeler et al., 2009; Shabani et al., 2011, 2012a,b). That MAHDR mice exhibit "normal" MAinduced DA release within the NAC and greater MA-induced DA release within mPFC, in the face of lower basal content, suggests that the high MA-drinking phenotype of these animals (Wheeler et al., 2009) may reflect an attempt to supersede an allostatic state. Indeed, drug-naïve MAHDR animals exhibit signs of anhedonia, in that they exhibit lower instrumental responding for a palatable sweet solution despite exhibiting greater responding for MA reinforcement, compared to MALDR animals (Shabani et al., 2012a). Thus, the low baseline DAEC (c.f. Wise, 2008), low basal D2R expression within the NAC (Blum et al., 2012), and/or dysregulated DA-5HT interactions within both the NAC and mPFC (e.g., Shirayama and Chaki, 2006) may all contribute to this presumed allostatic state in MAHDR mice that is theorized to underpin their addiction vulnerable phenotype.

#### **ANOMALIES IN FOREBRAIN 5HT ARE CORRELATES WITH HIGH GENETIC VULNERABILITY TO MA INTAKE**

While the majority of this study focused on DA, MAHDR mice were reported to exhibit higher NAC *Slc6a4* mRNA expression than MALDR mice (Wheeler et al., 2009) and thus, we investigated also for genotype differences in indices of 5HT neurotransmission within the NAC and mPFC. Extending earlier results (Wheeler et al., 2009), MAHDR mice exhibited higher SERT expression within the NAC, but lower SERT expression within the mPFC, relative to MALDR mice (**Table 5**). Interestingly, the MAHDR-MALDR differences in SERT expression were inversely related to genotype differences in 5HT1BR within these two regions (**Table 5**). Thus, as observed for corticolimbic DA, genotype differences in indices of 5HT neurotransmission depended upon the forebrain region investigated.

In the NAC, higher basal 5HTEC (**Figure 6C**) was coincident with higher SERT levels (**Figures 8A,B**) in MAHDR mice. Although we did not assay SERT function directly due to limited animal availability, we failed to detect line differences in the Ed for NAC 5HT using quantitative microdialysis procedures (**Table 2**), indicating no difference in 5HT uptake within this region (Sam and Justice, 1996). Thus, while the possibility may still exist that SERT is functioning sub-optimally within the NAC of MAHDR mice, a more parsimonious explanation is that the rise in SERT expression is merely a compensatory response to elevated 5HTEC, the latter of which results from 5HT1BR hypo-functioning (**Table 5**). Indeed, MA-stimulated monoamine release is primarily impulse-independent (e.g., Fleckenstein et al., 2007; Chen et al., 2009). Therefore, a perturbation in terminal 5HT1B autoreceptor function in MAHDR animals could underpin their elevated basal 5HTEC levels, without necessarily influencing the capacity of MA to raise 5HTEC. As acute treatment with 2 mg/kg MA failed to elevate NAC 5HTEC levels in either genotype (**Figure 7C**), it is difficult to discern a relation between the observed changes in 5HTEC, SERT and/or 5HT1BR expression to MA-induced 5HT release within the NAC.

**Table 5 | Comparison of the protein expression of serotonin-related proteins within the nucleus accumbens (NAC) and medial prefrontal cortex (mPFC) of the MAHDR and MALDR selected lines.**


*< denotes expression is less than; > denotes expression is greater than;* = *denotes no difference. n.s. indicates a strong trend for a genotypic difference that failed to reach statistical significance (i.e., p* = *0.05–0.09).*

Nevertheless, the present data for MAH/LDR mice indicate that higher basal 5HTEC and SERT, as well as lower 5HT1BR, expression within the NAC are correlates of high genetic vulnerability to MA intake, preference and reinforcement (see Wheeler et al., 2009; Shabani et al., 2011, 2012a,b) that are worthy of further exploration.

As was observed for forebrain DA, there were marked regional differences in the 5HT correlates of high vs. low genetic vulnerability to self-administer MA (**Table 5**). While MAHDR-MALDR differences were noted for NAC basal 5HTEC content, no line differences were noted for mPFC basal 5HTEC content or Ed. However, as observed for mPFC DAEC (**Table 4**) marked genotype differences were apparent regarding MA-stimulated 5HT release in this region; however, in the latter case, MALDR mice were considerably more sensitive to MA than MAHDR animals (**Figure 7D**). In fact, the genotype difference in MA-induced 5HT release within mPFC was polar opposite that observed for DA release in this region (**Figures 7B** vs. **7D**). This suggests a reciprocal interplay between these two monoamines systems, the basis of which cannot be discerned from the results of the present study. Nevertheless, the line differences in MA-induced 5HT release within the mPFC was associated with significant genotype differences in SERT and more moderate differences in 5HT1BR expression (**Table 5**). Notably, MAHDR mice displayed lower SERT and higher terminal autoreceptor expression, relative to MAHDR animals. Thus, the failure of MA to elevate 5HTEC within the mPFC of MAHDR mice might relate to their lower levels of SERT, although the possibility that higher 5HT1BR autoreceptor tone might influence the amount of 5HT release cannot be negated at this time. Together, the above data for 5HT in MAH/LDR mice implicate anomalies in both basal and MA-induced changes in corticolimbic 5HT transmission in the propensity to develop a MA-addicted phenotype and research into individual variation in indices of mesocorticolimbic 5HT transmission and MA addiction, as well as a more systematic characterization of the effects of MA history upon forebrain 5HT, are warranted at both the clinical and preclinical levels in order to better understand the interrelation between MA addiction vulnerability, addiction severity and 5HT.

#### **CONCLUSIONS**

While a number of questions still remain, the results of the present study indicate that a history of subchronic, subtoxic MA is sufficient to produce a persistent dysregulation of indices of DA neurotransmission within the NAC and mPFC of mice. Moreover, we demonstrated that DA within the NAC, but not mPFC, actively regulates the expression of MA preference and mice with genetic vulnerability for high MA intake exhibit DA anomalies within the NAC, many of which are akin to those produced by subchronic MA experience. As the MA injection regimen employed herein attempted to model early drug experience, these data suggest an important role for idiopathic or drug-elicited anomalies in NAC DA for MA preference/intake. Moreover, as mice with a genetic vulnerability to high MA intake exhibit also anomalies in 5HT, particularly within the mPFC, implicates also mPFC 5HT neurotransmission in the etiology of MA addiction.

## **ACKNOWLEDGMENTS**

This work was funded, in part, by NIH grant DA024038 to Karen K. Szumlinski, as well as funding from the Department of Veterans Affairs and NIDA grant P50 DA018165 to Tamara J. Phillips.

## **REFERENCES**


FP-CIT SPET in healthy volunteers. *Eur. J. Nucl. Med.* 27, 867–869. doi: 10.1007/s002590000279


processes within the mesolimbic dopamine system. *Addict. Biol.* 13, 287–294. doi: 10.1111/j.1369-1600.2007.00090.x


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 10 March 2014; accepted: 10 April 2014; published online: 07 May 2014. Citation: Lominac KD, McKenna CL, Schwartz LM, Ruiz PN, Wroten MG, Miller BW, Holloway JJ, Travis KO, Rajasekar G, Maliniak D, Thompson AB, Urman LE, Phillips TJ and Szumlinski KK (2014) Mesocorticolimbic monoamine correlates of methamphetamine sensitization and motivation. Front. Syst. Neurosci. 8:70. doi: 10.3389/ fnsys.2014.00070*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2014 Lominac, McKenna, Schwartz, Ruiz, Wroten, Miller, Holloway, Travis, Rajasekar, Maliniak, Thompson, Urman, Phillips and Szumlinski. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## **Gregory B. Bissonette\*, Ronny N. Gentry, Srikanth Padmala, Luiz Pessoa and Matthew R. Roesch**

Department of Psychology, University of Maryland, College Park, MD, USA

#### **Edited by:**

Dave J. Hayes, University of Toronto, Canada

#### **Reviewed by:**

John D. Salamone, University of Connecticut, USA Thilo Womelsdorf, Robarts Research Institute London, Canada

#### **\*Correspondence:**

Gregory B. Bissonette, Department of Psychology, University of Maryland, 1148 Biology-Psychology Building, College Park, MD 20742, USA e-mail: bissonette@gmail.com

Decision-making is motivated by the possibility of obtaining reward and/or avoiding punishment. Though many have investigated behavior associated with appetitive or aversive outcomes, few have examined behaviors that rely on both. Fewer still have addressed questions related to how anticipated appetitive and aversive outcomes interact to alter neural signals related to expected value, motivation, and salience. Here we review recent rodent, monkey, and human research that address these issues. Further development of this area will be fundamental to understanding the etiology behind human psychiatric diseases and cultivating more effective treatments.

**Keywords: reward processing, salience, value encoding, appetitive and aversive outcomes, neural encoding**

#### **INTRODUCTION**

Decision making is a complex process by which an organism must weigh multiple possible outcomes against current and long term goals before deciding on a course of action. Possible outcomes can be grouped into the probability of obtaining something rewarding or avoiding an outcome that is negative or punishing. Although an established body of literature has extensively studied neural systems involved in both of these functions, very few have set out to explicitly study how these neural systems directly reconcile both appetitive and aversive neural signals in a single task. Even fewer have addressed questions related to how anticipated appetitive and aversive outcomes interact to alter neural signals related to expected value, motivation, and salience. Here, we review studies that have addressed this issue in a number of key brain areas, all of which have been shown to exhibit neural activity modulated by expectation of appetitive and aversive stimuli when studied independently. We first review the non-human animal literature and then review studies performed in humans.

We know from a vast literature that neural activity of several regions throughout the brain are modulated by expected outcome, whether it be appetitive or aversive. It is widely assumed that this activity corresponds to an internal representation of how appetitive or aversive that expected outcome is. However, in many cases, change in responses relating to the expectation of emotionally charged outcomes alters other functions related to motivation, salience, arousal and attention that serve to facilitate response mechanisms to approach or avoid.

For example, an association of a particular odor that previously predicted the presence of a predator would be highly salient though lead to a negative association with that odor, while another odor may predict a salient rewarding stimulus, like ripe fruit, but with a positive valence. So while the general idea that appetitive and aversive systems oppose each other in the brain seems logical and useful (Solomon and Corbit, 1974; Daw et al., 2002), stimuli of either valence may drive arousal or enhance attention to stimuli of learned associations (Anderson, 2005; Lang and Davis, 2006; Phelps et al., 2006; Anderson et al., 2011).

Thus, a key problem is how to dissociate value from these other co-varying factors. A common approach is to vary appetitive and aversive stimuli in the same task. In these types of studies there are typically three basic trial types, such that: (1) a conditioned stimulus (CS) predicts a large reward; (2) a CS predicts a neutral condition or a small (or no) reward; and (3) a CS predicts a small (or no) reward with the threat of an aversive outcome. In animal studies, the aversive outcome may range in quality from time-outs, delivery of a bitter quinine solution, electric shock, or air-puff to the eye (Rolls et al., 1989; Roesch and Olson, 2004; Anstrom et al., 2009; Brischoux et al., 2009; Matsumoto and Hikosaka, 2009; Roesch et al., 2010a; Lammel et al., 2011; Bissonette et al., 2013). In human studies, the aversive outcome may be loss of money, an unpleasant liquid, shock, or an unpleasant odor (Delgado et al., 2000; Anderson et al., 2003; Small et al., 2003; Cooper and Knutson, 2008; Carter et al., 2009; Litt et al., 2011; Choi et al., 2013). If neurons encode *value*, activity should show a decreasing relationship during appetitive, neutral, and aversive trials (**Figure 1**). If appetitive and aversive stimuli are encoded by independent populations, then neurons should be modulated by either appetitive or aversive stimuli but not both. Finally, if activity is modulated by factors that vary with the strength of appetitive and aversive stimuli, neurons should respond with the same "sign" for appetitive and aversive trials compared to neutral trials (**Figure 1**). Although the relationship between neuronal responses and blood-oxygenationrelated activity obtained via functional MRI is complex (Goense and Logothetis, 2008), it is typically assumed that the same

pattern of activity applies to them both—thus, the same type of relationships should be observed at the levels of voxels or regions.

Below we first focus on animal studies that have used similar paradigms to try to dissociate value from motivation, salience, arousal and intensity. We then turn to functional MRI studies in humans.

#### **ORBITOFRONTAL CORTEX (OFC)**

Orbitofrontal cortex (OFC) encodes expectations about future appetitive and aversive outcomes that are critical for guiding learning and decision-making (Schoenbaum et al., 1998; Roesch and Olson, 2004; Schoenbaum and Roesch, 2005; Plassmann et al., 2010; Morrison et al., 2011; Morrison and Salzman, 2011). For example, neurons in OFC are modulated by cues that predict different appetitive outcomes, such as different food stuffs and magnitudes of reward; other OFC neurons signal when an aversive stimulus is anticipated, such as quinine or airpuff. However, since motivation and value were hard to disentangle in most of these experiments, it was unknown whether neural signals genuinely represented the value of the predicted outcome, or the motivational level associated with obtaining reward or avoiding aversive outcomes. For example, neurons in OFC fire strongly when monkeys anticipate a desirable outcome (Schoenbaum et al., 1998, 1999), but if that outcome is paired with a chance for another, more preferable outcome (Wallis and Miller, 2003), or is devalued through satiation (Rolls et al., 1989), then the rate of firing decreases. This activity modulation might reflect the decrease in value, but it might also reflect changes in motivation. A similar situation holds true for OFC neurons that predict aversive outcomes; activity might reflect how aversive the stimulus is or how motivated the animal is to avoid it.

To address these issues Roesch and Olson (2004) designed a task to dissociate value from motivation by simultaneously manipulating reward and punishment. The monkeys performed a memory-guided saccade task during which two cues presented at the beginning of each trial indicated the size of the reward the monkey would receive in the event of success (one or three drops of juice) and the size of a penalty that would be incurred in the event of failure (a 1 s or 8 s timeout). Behavioral measures indicated that the monkeys found the large reward appetitive and the punishment aversive; monkeys chose a large reward more often than a small one and avoided a large penalty more often than a small one. More importantly, monkeys were more motivated by large rewards and penalties as compared to smaller ones. Under both the largereward and large-penalty conditions, the monkeys broke fixation less often, made fewer errors, and were faster to respond, relative to neutral conditions (**Figure 2A**). Thus dissociation of value and motivation was achieved via simultaneous manipulation of appetitive and aversive outcomes (Roesch and Olson, 2004).

With this task it was predicted that neurons sensitive to the degree of motivation should respond with similar changes in firing rate to increasing the size of either the promised reward or the threatened penalty, thus paralleling the behavior. Indeed, premotor (PM) neurons that are strongly associated with motor output fired continuously during the delay between predictive cues and the behavioral response at a higher rate when either a large reward or a large penalty was expected (**Figures 2D, E**). Since activity persisted throughout the delay into the time when the monkey was making the behavioral response and because enhancement was observed in neurons with response direction selectivity, changes in firing were interpreted as reflecting motivational enhancement of motor output, as opposed to general arousal or salience. Accordingly, activity in PM reflected the motivational impact of the trial being performed, not its overall value, demonstrating a dissociation between motivation from value at the neural level.

Since OFC is more associated with more emotional/evaluative functions than motor areas, like PM, we expected that OFC neurons would better reflect the value associated with cues and reward delivery. Indeed, in stark contrast to PM neurons, OFC neurons fired most strongly for cues that predicted large reward (with small penalty) and least strongly for cues that predicted large-penalty (with small reward) relative to neutral conditions (small reward and small penalty; **Figures 2B–C**). Thus the strength of responding in OFC reflected the value conveyed by the combination of reward and penalty cues. Other studies have replicated these results and have further shown that other populations of OFC neurons do not represent the overall value associated with a given situation, but the actual offers being made and the option eventually chosen during performance of a choice task (Padoa-Schioppa and Assad, 2006, 2008; Hosokawa et al., 2007; Morrison et al., 2011). Collectively these studies have shown that OFC has all the signals necessary, at the single unit level, to make reward-guided decisions, as opposed to facilitating behavior through general motivational mechanisms under the influence of predictive appetitive and aversive events.

#### **BASOLATERAL AMYGDALA (ABL)**

Although the mainstream view holds that amygdala is important for acquiring and storing associative information related to both appetitive and aversive outcomes (LeDoux, 2000), there have been hints in the literature that amygdala also supports other functions related to associative learning, such as signaling of attention, uncertainty, and intensity (Saddoris et al., 2005; Belova et al., 2007; Tye and Janak, 2007; Tye et al., 2010; Morrison et al., 2011).

At the single neuron level, basolateral amygdala (ABL) is modulated by the predictability of both appetitive and aversive events, specifically when expectancies are violated (Belova et al., 2007; Roesch et al., 2010a,b; Tye et al., 2010). In other words, ABL neurons increase firing when outcomes are unexpectedly delivered or omitted, events that are highly salient and attention grabbing. We have shown that activity in ABL increases when rewards are unexpectedly delivered (appetitive) or omitted (aversive) in a task in which expected reward varies in size and time to delivery (Roesch et al., 2010a). Others have reported increased activity in ABL when rats were expecting reward, but not delivered during extinction (Tye et al., 2010). In primates, unexpected delivery of appetitive and aversive (airpuff) outcomes during performance of a trace conditioning task with reversals caused amygdala neurons to fire more strongly than when the outcome was totally predictable (Belova et al., 2007). Additionally, it appears that the same populations of ABL neurons which represent appetitive stimuli were also activated by aversive stimuli, regardless of the particular sensory modality from which the experience comes (Shabel and Janak, 2009). This suggests a larger role for ABL in signaling the need for attention in the presence of cues, rather than signaling the associated value of those cues. All of this suggests that amygdala does more than just signal appetitive and aversive stimuli.

Together, these reports suggest that ABL integrates information about appetitive and aversive events and their intensity or salience, possibly in the service of modifying behavior via signaling errors in predictions or recruitment of attentional/executive functions. However, these reports tend to focus on modulation of activity during delivery of appetitive and aversive outcomes. Much less is known about modulation by salience during sampling of cues that predict outcomes. Notably, modulation of amygdala firing for cues that predict appetitive and aversive outcomes appears to occur in separate neurons, suggesting that activity during this period is more related to the valence of the expected outcome. Likewise, cues presented after unexpected events or during response conflict when enhanced attention is necessary do not elicit changes in activity as do errors in reward prediction or commission as observed in other areas, such as anterior cingulate. Thus, ABL might be critical for reporting attentional need, arousal or intensity during sampling of unconditioned stimuli in the service of learning to predict the appetitive and aversive nature of the outcomes during sampling of conditioned stimuli. This is consistent with the finding that ABL interference disrupts development of cue selectivity in other areas, such as OFC and ventral striatum (VS; Hatfield et al., 1996; Schoenbaum et al., 2003a,b; Stalnaker et al., 2007, 2009).

#### **VENTRAL STRIATUM (VS)**

The connectivity of VS with OFC and ABL suggests that it might also represent the value of expected outcomes. This would be consistent with its proposed role as the "critic" in actor-critic models, where VS generates value predictions about future outcomes, which are used by dopamine (DA) neurons to compute prediction errors (PEs) necessary for updating actions polices in the "actor", namely dorsal striatum (Houk et al., 1995; Sutton, 1998; Haber et al., 2000; Joel et al., 2002; Redish, 2004; Ikemoto, 2007; Niv and Schoenbaum, 2008; Takahashi et al., 2008; Padoa-Schioppa, 2011; van der Meer and Redish, 2011). However, VS has traditionally been thought to be the "limbic-motor" interface critical for motivating behaviors. Under this framework, one might predict that VS is critical for motivating or facilitating behaviors in response to both appetitive and aversive stimuli and not for representing value *per se*. Consistent with both of these theories, pharmacological manipulations of VS impact motivated behaviors dependent on value expectations during a variety of tasks (Wadenberg et al., 1990; Berridge and Robinson, 1998; Blokland, 1998; Ikemoto and Panksepp, 1999; Di Ciano et al., 2001; Cardinal et al., 2002a,b; Di Chiara, 2002; Salamone and Correa, 2002; Giertler et al., 2003; Wakabayashi et al., 2004; Yun et al., 2004; Floresco et al., 2008; Gruber et al., 2009; Ghods-Sharifi and Floresco, 2010; Stopper and Floresco, 2011), including reward seeking (Ikemoto and Panksepp, 1999), cost-benefit analysis (Floresco et al., 2008; Stopper and Floresco, 2011), and delay/effort discounting (Cardinal et al., 2001; Ghods-Sharifi and Floresco, 2010).

Motivation is a complex psychological feature, likely arising from assessments of physiological states, understanding and attending to current environmental cues, past reinforcement history, and assessing expected value associated with current contexts. In this light, pharmacological manipulations of the VS will only likely uncover a portion of the story, while single unit recording may uncover separate yet concurrent roles for a brain region, difficult to piece apart with pharmacological work.

Previous single unit work has clearly demonstrated that firing in VS is modulated by the value associated with cues that predict reward in rats (Carelli and Deadwyler, 1994; Setlow et al., 2003; Janak et al., 2004; Nicola et al., 2004; Ito and Doya, 2009; van der Meer and Redish, 2009; Kalenscher et al., 2010; Lansink et al., 2010; van der Meer et al., 2010; Day et al., 2011; Goldstein et al., 2012) and monkeys (Schultz et al., 1992; Shidara and Richmond, 2004; Cromwell et al., 2005; Kim et al., 2009; Nakamura et al., 2012) performing a variety of instrumental tasks, including go/nogo (Schultz et al., 1992; Setlow et al., 2003), lever pressing (Carelli and Deadwyler, 1994; Janak et al., 2004; Shidara and Richmond, 2004; Cromwell et al., 2005; Day et al., 2011), discrimination (Nicola et al., 2004; van der Meer et al., 2010; Goldstein et al., 2012), maze running (van der Meer and Redish, 2009; Kalenscher et al., 2010; Lansink et al., 2010), and eye movement paradigms (Kim et al., 2009; Nakamura et al., 2012). However, from these studies, it was unclear what exactly VS responses represented because none of these studies had independently manipulated value from motivation. To address this issue we

penalty. Fixation break rate: trials terminated by a fixation break expressed as a percentage of all trials. Reaction time: average interval between fixation spot offset and saccade initiation on all trials in which the monkey made a saccade in the rewarded direction. Asterisks (all planned

penalty expressed as a fraction of all trials on which they chose reward or

three incentive conditions for all 176 OFC neurons. **(D, E)** Neuronal activity in premotor (PM) reflects the motivational impact of the incentive cues. **(D)** Shown are data from a single neuron firing throughout the trial at a rate that was high for large reward and large penalty. **(E)** Mean firing rate as a function of time under the three incentive conditions for all 135 PM neurons. Adapted from Roesch and Olson (2004).

adopted a similar behavioral strategy in rats as we did in primates, varying expected reward and punishment so that value and motivation signals could be dissociated (Bissonette et al., 2013).

Rats were trained on a task in which illumination of a left or right light indicated the location of reward. Prior to the spatial cue, an odor informed the rat of the size of reward and punishment that would result upon correct and incorrect performance, respectively. On two trial-types, there was no risk of punishment, just the potential of a large or small reward for a correct response. On a third trial-type, a small reward was promised for accurate performance, but there was also a risk of punishment (quinine) if the rat performed the task incorrectly. Rats were more accurate and faster to move down to the fluid well in response to the lights on large reward and quinine risk trials compared to small reward trials, demonstrating higher motivation on these trials relative to small reward trials (**Figure 3A**).

Remarkably, we found that single units in VS encoded both value and motivation. An example of the former is illustrated in **Figure 3B**. During odor sampling this neuron fired the most and the least for cues that predicted reward and punishment, respectively. This same neuron also fired during delivery of reward, but did not merely encode reward consumption, as evidenced by elevated firing when reward was omitted on error trials (**Figure 3D**). These results suggest that neurons in VS reflect the expected value of the reward during cue presentation and after the behavioral response. Thus, neurons in VS carry predictive value signals during odor sampling.

This relationship with value was mostly present in the activity of neurons that increased firing to both odor cues and reward delivery. Cue-responsive neurons that showed decreases in firing to reward delivery better reflected the degree of motivation associated conditioned stimuli, as illustrated in **Figures 3C, E**. For this neuron, activity was stronger for odor cues that predicted large reward and the risk of quinine punishment relative to small reward trials, consistent with representations of enhanced motivation.

Our results suggest that VS fulfills both evaluative and motivational functions, likely via separate neuronal populations, and might be required for integrating both types of information that are central to actor-critic models, as well frameworks that view the VS as a "limbic-motor" interface (Bissonette et al., 2013). All of this work features VS as a common junction point to act, possibly concurrently, to signal value and motivation which leads to the invigoration of particular behavioral actions over others. This idea is consistent with pharmacological studies suggested that DA in the VS had more to do with encoding incentive salience and motivation, rather than evaluative functions (Salamone, 1986; Salamone et al., 1991; McCullough and Salamone, 1992; Salamone, 1994; Koch et al., 2000; Berridge, 2007; Lex and Hauber, 2010; Salamone et al., 2012; Salamone and Correa, 2012; Nunes et al., 2013) and others that show that VS lesions disrupt rats ability to choice between differently valued rewards and to update behavior after devaluation of expected outcomes (Singh et al., 2010; Burton et al., 2013).

**FIGURE 3 | Ventral striatal neurons encode both value and motivation.** Rats performed a task during which two odors indicated the size (large or small) of the reward to be delivered at the end of the trial. If an error was committed on large and small reward trials, no reward was delivered. A third odor indicated that a small reward would be delivered on correct trials and that quinine would be delivered when rats responded to the wrong well. **(A)** Average lick rate over time during recording sessions. Black = delivery of large reward; Dark gray = delivery of small reward when there was no risk; Light gray = delivery of small reward when there was a risk of quinine. Dashed gray = delivery of quinine on risk trials during which rats went to the wrong fluid well. Average percent correct for the three trial types. Average time taken to move from the odor port to the fluid well in response to the spatial cue lights. **(B–C)** Single cell example of neurons that exhibited firing patterns consistent with value and motivation encoding on (Continued)

#### **FIGURE 3 | Continued**

correct trials for the 3 trial-types: large reward, small reward, and punishment. Activity is aligned to odor onset (left of dashed box) and reward delivery (right of dashed box). Inset: average waveform (not inverted). **(D–E)** Average normalized firing over all neurons that showed significant increases to both odor cues and reward delivery and those neurons that showed significant increased and decreased firing to cues and rewards, respectively. Firing rates were normalized by subtracting the baseline and dividing by the standard deviation. Ribbons represent standard error of the mean (SEM). Blue asterisks indicate significant differences between average firing during the odor epoch (gray bar) between large reward and small reward trials (blue versus yellow; t-test; p < 0.05). Red asterisks are the comparison between quinine punishment and small reward trials (red versus yellow; t-test; p < 0.05). The odor epoch did not include time when lights were on. Gray dashed = onset of odors. Black dashed = earliest possible time lights could turn on. Black arrow marks the average time of reward delivery. Adapted from Bissonette et al. (2013).

#### **DOPAMINE**

Signals from midbrain DA neurons play a critical role in reinforcement learning by providing a physiological correlate to the well-studied PE. This PE signal guides goal-directed behavior by informing the system which aspects of the environment are appetitive or aversive and initiating actions in order to obtain the good and avoid the bad (Schultz, 1997). Phasic bursts or pauses in neuronal activity, together with resulting neurotransmitter release, encodes this PE signal. The PE signal measures the difference between an expected outcome and the actual outcome in order to inform future behavior. A better-than-expected outcome activates dopaminergic neurons (positive PE) resulting in neurotransmitter release, while a worse-than-expected outcome (negative PE) induces a pause in dopaminergic firing. A fully predictable outcome elicits no change in firing of DA neurons. The same firing pattern applies for sensory cues that come to predict or give information about future rewards. Thus, DA firing and release tends to shift away from the delivery of primary rewards as they come to be predicted by cues during learning, resulting in more or less firing for cues that predict appetitive and aversive outcomes, respectively.

Based on the mismatch of expectation and consequence, the DA signal acts as a teaching mechanism, updating expectations and potential behavioral responses based on feedback received from the environment. DA neurons that fire synchronously and release DA as a result, are reinforced and are more likely to be activated in the future, promoting paired behaviors. The synchronized firing of dopaminergic neurons follows Hebb's idea that "neurons that fire together, wire together", but DA must be released in order for reinforcement learning to occur and the synaptic connection between neurons to be strengthened (Montague et al., 1996; Schultz, 1998; Bromberg-Martin et al., 2010). Most of the value signaling described in the brain areas above likely relies on DA to form associations between stimuli and outcomes during learning and decision-making.

Although PE signaling is often studied under paradigms that require animals to approach appetitive stimuli, PE theory holds true for DA signals related to avoiding aversive stimuli, such as air puff and shock. In primates, neurons that encode reward PEs are depressed by unexpected air puff and visual cues that predict them (Bromberg-Martin et al., 2010). Furthermore, DA firing increases when an expected air puff is omitted, an event that is more appetitive or better than expected. A similar story is true in rats performing an instrumental escape-avoidance paradigm. Oleson et al. (2012) showed that phasic DA activity to cue presentation can predict if rats will successfully avoid an upcoming foot shock. Successful avoidance behavior was contingent upon DA release time-locked to the warning light. DA was released at the time of the avoided shock and during cue presentation of successful avoidance trials. Thus, as with appetitive paradigms, DA signals adhere to the general rule of firing more or less strongly for cues and outcomes that are better or worse than expected, respectively (Oleson et al., 2012). Importantly, this signal is dependent on input from OFC (Takahashi et al., 2011).

Notably, not all DA neurons transmit reward PE signals. Other, anatomically discrete, DA neurons appear to be more concerned about the motivational salience of appetitive and aversive stimuli (Matsumoto and Hikosaka, 2009). These DA neurons are triggered by both appetitive and aversive outcomes and the cues that predict them. In experiments where visual stimuli predict either reward or air-puff, these DA neurons fire more strongly for delivery of these outcomes and the cues that predict them, relative to neutral trials where there is no reward or air-puff. Interestingly, these two types of DA neurons, referred to as value and salience encoding neurons, are somewhat segregated in evaluative VTA and SNc, with value encoding cells mostly located in VTA and motivational salience DA neurons in SNc. There exists additional support for the idea that a subset of VTA DA neurons fire preferentially for aversive stimuli in rats, including social defeat, aversive foot shock (Anstrom et al., 2009; Brischoux et al., 2009) and pain inducing plantar injection of formalin to mice (Lammel et al., 2011). Evidence supports the notion that aversive-preferring or salience DA neurons project preferentially to the prefrontal cortex (PFC) and the core of nucleus accumbens (NAc), while reward-preferring or PE DA neurons project preferentially to the ventromedial PFC and the shell of NAc (Bromberg-Martin et al., 2010). Given this data, and the aforementioned idea that these DA neurons may be encoding salience, it seems likely that such a signal would be critical for driving attention/motivation to salient (appetitive or aversive) events promoting learning in regions that these neurons project to, whereas PEs signal might be critical for specifically updating representations of associations between events and their respective outcomes. Thus, DA signals value in the form of PEs, supporting functions related to approach, evaluation and value learning, and also motivational salience, supporting functions related to orienting, attention, and arousal (Bromberg-Martin et al., 2010).

## **PARIETAL CORTEX AND ANTERIOR CINGULATE CORTEX (ACC)**

The most recent debate about value versus salience has focused on the parietal cortex. Parietal neurons have been shown to fire at a rate, dependent on the value of expected actions (actionvalue) and this signal is critical for making economic decisions about which action produces a better reward. Recently, Leathers and Olson (2012) reported that primate lateral intraparietal (LIP) neurons fire most strongly when a saccade is associated with a large versus small reward. Importantly, they also showed that the same neurons fired more strongly for cues that predicted a large versus small penalty. They suggest that the activity of LIP neurons encode the motivational salience of a cue, rather than the value necessary for decision-making.

In a rebuttal paper, Newsome et al. (2013) suggested that Leathers and Olson (2012) did not replicate delay-period activity as observed in previous experiments, calling into question the population of parietal neurons studied and the ability of the task to tap into these functions that capture action-value (Newsome et al., 2013). Subsequently, Leathers and Olson (2013) replied by pointing out that the key findings of their initial study, namely, that stronger activity was correlated with larger, rather than smaller penalty cues and that neurons signaled salience earlier in the trial during the decision process, were not in question, and that these correlates were found in cells that fired across delays the preceded the response. They suggest that the fact that salience, not value, is encoded by parietal cortex in this task suggests that value encoding is not a general function of parietal cortex. Further work is necessary to determine in what contexts parietal neurons might reflect salience versus value.

Other cortical areas thought to be involved in attention have been recently discussed in the realm of reward-related decisionmaking and reinforcement learning. Single neurons in macaque ACC show correlates related to unsigned PEs (Hayden et al., 2011), potentially signaling the necessity for additional resources in the face of signaling a need for behavioral modification. Using a variable size/delay task, Bryden et al. (2011) demonstrated rat ACC signaled errors and signaled the need for additional attentional resources during unexpected shifts in value in the same task used to investigate error signing in ABL (Bryden et al., 2011). Unlike activity in ABL, ACC firing was significantly stronger after both unexpected appetitive and aversive events during and before sampling of cues on subsequent trials. This signal likely reflects the salience or attention that is drawn to conditioned stimuli so that contingencies can be updated during learning.

These data are contrasted a bit by work in rhesus monkeys demonstrating ACC encoding of value as it relates to integrating previous outcomes with current choices (Kennerley et al., 2011). Indeed, additional research has suggested that medial PFC (which included parts of ACC) in rhesus monkeys signal both positive and negative PEs of action values (Matsumoto et al., 2007). Others have reported that distinct regions in ventromedial PFC encode rewards and punishments, with ventral and dorsal aspects being more active for appetitive and aversive trial-types, respectively (Monosov and Hikosaka, 2012). The fact that value and salience signals in ACC and parietal cortex appear to go hand in hand are consistent with the need for attentional control to ensure neural processes are prioritized depending on expected events and current behavioral strategy. Indeed neural correlates related to value predictions and spatial attention have been shown to be integrated in clusters of neurons in primate PFC (Kaping et al., 2011). Further research will need to be done to fully separate prefrontal and parietal contributions to signaling value, salience or both using a novel tasks that varies both appetitive and aversive outcomes.

#### **HUMAN STUDIES**

In parallel with the animal literature, human studies have implicated midbrain dopaminergic regions and their projection sites in the striatum and OFC during appetitive processing (Schultz et al., 2000; O'Doherty, 2004; Delgado, 2007; Haber and Knutson, 2010) and regions such as the amygdala and anterior insula during aversive processing (Adolphs and Tranel, 2000; LeDoux, 2000; Craig, 2002, 2009; Davis et al., 2010) Importantly, ventral and dorsal striatal regions are also involved during aversive processing (Jensen et al., 2003; Pruessner et al., 2004; Delgado et al., 2008), while there is some evidence for amygdala and anterior insula activity during appetitive processing (Everitt et al., 2003; Liu et al., 2011). Findings such as these question frameworks that promote appetitive and aversive processing purely in terms of distinct brain regions. Instead, they demonstrate that some of these regions encode factors such as salience and motivational "activation" not simply value.

Human studies have also attempted to dissociate the processing of value from factors such as salience, intensity, or arousal. These studies have used a wide range of tasks and focused on decision making and PE signals, as well as responses at different task phases, including cue and outcome-related activity. The overall logic used in human studies to attempt to dissociate value from other factors is similar to the one used in the animal literature (**Figure 1**). As before, three trial types are typically used: (1) appetitive, (2) aversive, and (3) neutral. If activity in a region represents "value" signals, then activation in that region for appetitive stimuli is expected to be greater than aversive stimuli, with responses to neutral somewhere in between. However, if activity represents salience or intensity, activation during both appetitive and aversive stimuli would be greater than that observed to neutral stimuli.

#### **DECISION MAKING**

Rangel and colleagues used a simple yet elegant decision-making task to disentangle fMRI signals related to value and salience (Litt et al., 2011). Participants were shown pictures of food items that ranged from being highly disliked to highly liked and were asked to make a choice whether or not they would like to eat the item after the experiment (participants in fact consumed these items following scanning). Consistent with previous animal work, value signals were observed in medial OFC (**Figure 4A**; as well as rostral ACC and PCC). Areas such as dorsal ACC, SMA, and insula generated salience type signals as they produced stronger responses for both "highly disliked" and "highly liked" items. Interestingly, signals in VS exhibited both value and salience type components consistent with the animal literature. As illustrated in **Figure 4B**, such signals in fact demonstrate that "hybrid" representations that code for both value and salience are also possible (Litt et al., 2011).

#### **REWARD CUE PROCESSING**

Adcock and colleagues utilized a simple cue followed by response task to dissociate value and salience signals in the VS and midbrain (Carter et al., 2009). In the experiment, cues signaled the chance to win monetary rewards ("gain") or the chance to avoid monetary losses ("loss") based on fast and accurate

experiment (participants in fact consumed these items following scanning). For each picture, participants entered their response on one of the four choices: "Strong No (St. No)", "No", "Yes" or "Strong Yes (St. Yes)". These four types of responses were used to define value and salience signals. The value regressor was defined based on the parametric weights [-2 -1 1 2] and the salience regressor was defined based on the parametric weights [2 1 1 2] corresponding to the four choices above (in that order). **(A)** Evidence for value type signals found in the medial OFC. **(B)** Evidence for both value and salience type signals found in the VS. Adapted from Litt et al. (2011).

performance; baseline conditions involving no gain or loss ("nogain"/ "no-loss") were also employed. Cue-related activity in both NAc and VTA increased for both gain and loss trials, thus providing evidence for salience signals in both structures. Furthermore, in both regions, cue-related activity during gain and loss trials was positively correlated across participants providing further evidence for the salience account (Carter et al., 2009).

Cooper and Knutson, 2008 also found similar "salience" type responses in the NAc while participants processed cues that signal performance-dependent monetary gains or losses (Cooper and Knutson, 2008; but see Knutson et al., 2001; Breiter et al., 2001). Interestingly, when the outcomes were certain (i.e., independent

**FIGURE 5 | Dissociation of value and salience signals during a reward processing task with humans.** Each trial started with one of the six cue types: two levels of certainty ("certain"/"uncertain") crossed with three levels of reward ("gain"/"neutral"/"loss"). After a variable delay period, a visual target appeared and participants pressed a button while the target was on the screen. The duration of the target was adjusted dynamically in each condition separately to maintain approximately 67% task performance. During "gain" trials participants could earn monetary reward; during "loss" trials participants could lose money; during "neutral" trials no win/loss occurred. During "certain" trials, outcomes were independent of performance, whereas during "uncertain" trials outcomes were based on performance. Value signals were found in the NAc when outcomes were "certain" (i.e., independent of performance) and evidence for salience signals were found when outcomes were "uncertain" (i.e., based on performance). Adapted from Cooper and Knutson (2008).

of performance), they observed increased activity for gain compared to loss cues revealing value type signals in the NAc (**Figure 5**).

#### **REWARD OUTCOME PROCESSING**

Delgado and colleagues used a simple card-guessing task to investigate the neural responses related to reward and punishment *feedback* (Delgado et al., 2000). Participants were asked to guess whether the value of the unknown card would be greater or smaller than 5. If they guessed correctly, they received monetary reward; for incorrect guesses monetary punishment was incurred. On neutral trials, where the value of the card turned out to be exactly 5, there was no reward or punishment. They observed value type signals in dorsal and VS during feedback, such that responses were greatest for reward, weaker for neutral and weakest for punishment trials (Delgado et al., 2000). In a follow-up study, they observed that value responses in dorsal striatum were present only when rewards were contingent upon behavior; they were absent when feedback was independent of the behavior (Tricomi et al., 2004).

#### **RESPONSES TO UNCONDITIONED STIMULI (US)**

Another class of experiment has investigated responses to pleasant or unpleasant sensory stimuli themselves. In one case, Anderson and colleagues independently varied the intensity and valence of olfactory stimuli by using pleasant and unpleasant odorants of high and low intensity (Anderson et al., 2003). Responses in amygdala reflected the intensity of the odor, not the valence. In contrast, the OFC revealed value type responses. Specifically, responses in medial OFC were stronger for pleasant compared to unpleasant odors whereas responses in lateral OFC were stronger for unpleasant compared to pleasant odors (Anderson et al., 2003). In a similar study with gustatory stimuli, Parrish and colleagues independently varied the intensity and valence of liquids and found similar evidence for salience signals in the amygdala and value signals in the OFC (Small et al., 2003).

These two studies suggested a general role for the amygdala in the coding of stimulus intensity. Yet, a follow-up study by Dolan and colleagues using olfactory stimuli demonstrated that the activity in the amygdala is best conceptualized in terms of an *interaction* between intensity and valence—that is, an interaction between salience and value (Winston et al., 2005). The authors used high/low concentrations of pleasant/unpleasant/neutral odors and reported that activity in the amygdala was increased for high (versus low) intensity odors *only* when they were pleasant or unpleasant, but not when the odor was neutral. Related valence by intensity interactions have also been observed in the amygdala in the animal literature (Paton et al., 2006).

#### **PREDICTION ERROR SIGNALS**

Several functional MRI studies have used Pavlovian conditioning paradigms to attempt to dissociate value and salience encoding based on the pattern of PE signals.

The logic of these experiments is that regions encoding value would exhibit opposite PE signals for appetitive and aversive stimuli, where a positive PE response would be observed when an appetitive US is delivered or when an aversive US is omitted, and a negative PE response would be observed when an aversive US is delivered or when an appetitive US is omitted. In contrast, regions encoding salience would exhibit similar PE signals for both appetitive and aversive stimuli, where a positive PE would be observed for reinforced outcomes and a negative PE for unreinforced outcomes. Using this logic, Jensen et al. (2007) reported salience type PE signals in the VS, bilateral anterior insula and medial OFC. Similarly, Dreher and colleagues reported salience type PE signals in the striatum (bilateral putamen) and amygdala (as well as anterior insula and ACC) (Metereau and Dreher, 2013). Notably, these studies did not find evidence for *value* type PE signals in the human brain.

#### **Salience signals or analysis confound?**

A challenge with functional MRI studies of PEs is that the PE signal is confounded with that of US delivery (Niv, 2009). Specifically, the PE is positive when the US is delivered and negative when the US is withheld. As a consequence, a traditional multiple regression analysis could implicate regions in the generation of PE signals when they are actually responding simply to US delivery. To control for this confound, researchers typically include an additional US regressor (i.e., covariate) for each trial type along with a "parametric" regressor to capture variance related to the PE. **Figure 6** illustrates this situation. Unfortunately, this strategy could itself spuriously lead to PE-related activity. For instance, imagine a region that simply responds to the US (e.g., insula activated by electric shock) but has no role in encoding PEs. When a single US regressor tries to account for variance during both reinforced and unreinforced shock outcomes as typically done, the estimated regression coefficient would be somewhere midway between the activity evoked by reinforced and unreinforced outcomes. Hence, the unaccounted variance in this region would have a positive value (i.e., residual) during reinforced outcomes and a negative value (i.e., residual) during unreinforced outcomes. This overall pattern qualitatively matches the shape of the PE regressor. Therefore, one could spuriously detect PE type signals in regions that simply respond to US delivery. Some functional MRI studies have avoided this problem (McClure et al., 2003; D'Ardenne et al., 2008).

#### **SIMULTANEOUS MANIPULATION OF APPETITIVE AND AVERSIVE STIMULI**

The work that we have discussed so far has considered appetitive and aversive information in isolation. A few recent studies have used stimuli that *simultaneously* incorporate appetitive and aversive information to further understand the role of different brain regions in processing value and/or salience type signals.

In a decision making paradigm, Tobler and colleagues investigated two kinds of salience signals that can only be distinguished in decisions that involve simultaneous costs and benefits (Kahnt and Tobler, 2013). When appetitive and aversive stimuli are presented in *isolation*, salience can be captured by the absolute value of the stimulus (i.e., |App| or |Aver|). But when appetitive and aversive stimuli are presented *simultaneously*, salience could be of two types: one based on the absolute value of the "total" (i.e., |App + Aver|), another based on the sum of the absolute values (i.e., |App| + |Aver|). Tobler and colleagues found evidence for the latter type of salience signal in a site in the temporo-parietal junction (TPJ). Consistent with previous studies, they also found evidence for value based signals in the VS (though they did not detect salience-related signals in the VS). A handful of additional decision making studies have used simultaneous appetitive and aversive stimuli to the same effect (Talmi et al., 2009; Park et al., 2011).

In a recent study, we were also interested in characterizing responses to stimuli containing both appetitive and aversive information. In the study, we investigated the interactions between the anticipation of reward and/or threat (Choi et al., 2013). Participants were presented with four advance cues to alert them of the possibility of: (1) reward/no shock, (2) reward/shock, (3) no reward/no shock, and (4) no reward/shock. Reward was contingent on performance whereas shock was independent of performance. This procedure juxtaposed two competing ideas. One, in line with what we have been discussing, for conditions involving simultaneous reward and threat, enhanced activity would reflect a type of salience signal (given the presence of both dimensions); the other predicted that the presence of both appetitive and aversive stimuli would lead to a "competition" between them. Skin conductance data acquired during scanning demonstrated an interaction between reward and threat processing, such that reward and threat effects were reduced by threat and reward, respectively. In terms of brain responses, several brain

**FIGURE 6 | Prediction error (PE) signal analysis and potential confounds in fMRI analysis. (A)** In a typical appetitive Pavlovian conditioning paradigm, one visual cue (CSneutral; not shown) is associated with no-reward (100% probability) whereas a second visual cue (CSreward) is associated with 50% probability of receiving reward. PE (i.e., actual minus expected outcomes) measured at the outcome phase of CSreward trials. **(B)** Simulated fMRI time series data (blue) generated using 10 reinforced and 10 unreinforced outcome events of CSreward trials in a pseudorandom order with 15 s separation between events at a typical TR of 2.5 s. For the sake of simplicity, we have not considered CSneutral trials and the cue phase of CSreward trials (which are typically modeled as separate regressors) and no noise was added to the simulated data. **(C)** When a single outcome phase regressor is used to account for variance during both reinforced and unreinforced outcomes of

CSreward trials as typically done, the estimated regression coefficient would be somewhere midway between the activity evoked by reinforced and unreinforced outcomes, as demonstrated by the estimated data fit (red). **(D)** Hence, the residual time series data (green) will show positive values during reinforced outcome events and negative values during unreinforced outcome events. **(E)** Parametric regressor based on trial-by-trial fluctuations of PE values at the outcome phase of CSreward trials calculated using the Rescorla-Wagner rule (Rescorla and Wagner, 1972) (a learning rate of 0.25 was used as often used in fMRI studies). **(F)** The residual time series and the PE regressors are overlaid to show the high correlation between them. Because of this, unaccounted variance during the outcome phase related activity of CSreward trials could be "spuriously" accounted by the PE regressor.

areas exhibited this type of reward-threat trade-off, including midbrain, caudate, putamen, and anterior insula.

#### **LIMITATIONS OF FUNCTIONAL MRI STUDIES**

Single unit recordings in midbrain and VS have identified separate populations of neurons coding for value and salience (Matsumoto and Hikosaka, 2009; Bissonette et al., 2013). The coarse spatial resolution of typical functional MRI studies prevents them from measuring separate signals for the separate populations. Indeed, in some cases, the measured fMRI response could be based on the combined activity of underlying value and salience processing neurons. Consider also single-unit studies revealing separate populations of neurons coding for appetitive and aversive stimuli (e.g., Ungless et al., 2004). In such cases, if a region shows salience type fMRI responses, it could be due to the contribution from separate underlying neuronal populations, which would be engaged by appetitive and aversive stimuli. But here it is worthwhile noting that some single-unit studies in humans (Laxton et al., 2013) and monkeys (Amemori and Graybiel, 2012; Monosov and Hikosaka, 2012) have revealed neurons coding for appetitive and aversive stimuli within the *same* population. These studies, together with human studies that revealed the dependence of valence signals on their salience (e.g., during active versus passive task processing) are consistent, in broad terms, with meta-analytic findings reporting little evidence for processing of discrete emotion categories in distinct brain regions (Lindquist et al., 2012).

A second issue is that the sluggish nature of hemodynamic responses makes it challenging to unambiguously disambiguate responses to different task phases, for example, "cue", "anticipation", and "outcome" phases. In contrast, the high temporal resolution of electrophysiology provides rich information to investigate the dynamics of value and salience representations (e.g., Matsumoto and Hikosaka, 2009). Importantly, high temporal resolution in single unit studies also allows the investigation of responses at the outcome phase independent from short-latency responses linked to the sensory properties of US (Fiorillo et al., 2013).

#### **DISCUSSION**

In this paper, we reviewed how the brain encodes appetitive and aversive events in both non-human animals and humans. This line of research is important, as understanding the neural processing behind appetitive and aversive stimuli is critical to understand what drives different behavioral responses. Much of the literature has approached these problems by studying how an animal associates a particular odor with a potential predator, or by investigating how a visual cue is associated with a tasty ripe fruit. The behaviors enacted in each of those scenarios would be, naturally, very different (alertly avoid, or boldly engage). However, few situations in real life are as cut and dry. Often, predators prowl near locations and objects that prey animals enjoy (near watering holes, food sources, migratory routes, following the mating calls of animals), and attaining rewards may require dealing with cues that signal aversive events (extracting honey from wild bees, picking fruit from thorny plants).

As reviewed here, despite differences in the species investigated and the techniques utilized, some consensus has started to emerge regarding the encoding of both value and other related motivational signals. Yet, both apparent discrepancies and unresolved issues remain and need to be addressed in future work. The combined evidence reveals that the OFC has a representation of value that is relatively "pure". The VS carries both value and salience signals that appear to be generated by different neuronal populations. Amygdala responses are modulated by stimulus intensity, though the signal is clearly moderated by the valence (i.e., value) of the stimulus.

Taken together, the work described here suggests a circuit by which OFC represents value expectancies necessary for guiding decision-making and learning. These signals depend on ABL, which not only encodes associative appetitive and aversive information during sampling of conditioned stimuli and across states, but integrates value and intensity/salience during delivery of appetitive and aversive outcomes. OFC and ABL both broadcast this information to VS and DA neurons, which carry both evaluative (VTA) and motivational salience (SNc) signals in separate populations of neurons (**Figure 7**). PE signals generated by VTA DA neurons provide feed-forward information to more dorsalmedial and dorsal-lateral regions in striatum, which are critical for goal-directed and habitual behaviors, respectively. Parietal and ACC likely increase attentional control to ensure that neural processes are prioritized depending on expected actions and unsigned errors in reward prediction. From this research it is clear that we have to continue to compare and contrast how neural systems reconcile both appetitive and aversive stimuli, and continue to disambiguate the meaning of signals modulated by valence and how they relate to subsequent behavior.

In terms of issues that will drive future research, we can highlight at least three. The first concerns the types of representation

Anterior Cingulate Cortex—ACC, Parietal Cortex—Parietal, Dorsal Medial Striatum—DMS, Dorsal Lateral Striatum—DLS, Ventral Tegmental Area—VTA, Substantia Nigra compacta—SNc, Superior Colliculus—SC, GP—Globus Pallidus, Thalamus—Thal, Substantia Nigra reticulata—SNr, Premotor Cortex—PM.

in parietal cortex. Are they closer to value based or are they better conceptualized in terms of salience? The second concerns the study of PEs in the human brain with functional MRI. As illustrated, it can be challenging to separate "true" PEs from responses to US delivery. Consequently, it is unclear at the moment if PE signals reflect salience representations across a wider set of regions of the brain as suggested by the human work (e.g., insula, dorsal striatum, amygdala, ACC), or if in some cases they may have resulted from responses to the US itself. This is an area that we believe future work is clearly needed, both non-human work investigating a wider group of regions, and human work that more effectively deals with potential confounds. A third issue is related to functional MRI as a methodology. Both issues of spatial and temporal resolution pose important challenges to being able to investigate value signals in the brain. These clearly need to be addressed more effectively; perhaps with newer techniques that allow finer temporal sampling (every 500 ms or less) of hemodynamic responses and finer spatial resolution (less than 1 mm) can go some way toward mitigating current issues (though higher temporal sampling can only go so far given the low-pass nature of the hemodynamic response). In any case, we anticipate exciting times ahead as the field advances the understanding of how the brain encodes value and other motivational variables.

#### **ACKNOWLEDGMENTS**

Support for this work was provided in part by the National Institute of Mental Health (R01MH071589 to Luiz Pessoa) and the National Institute on Drug Abuse (R01DA031695, Matthew R. Roesch).

#### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 29 October 2013; accepted: 04 February 2014; published online: 04 March 2014*.

*Citation: Bissonette GB, Gentry RN, Padmala S, Pessoa L and Roesch MR (2014) Impact of appetitive and aversive outcomes on brain responses: linking the animal and human literatures. Front. Syst. Neurosci. 8:24. doi: 10.3389/fnsys.2014.00024*

*This article was submitted to the journal Frontiers in Systems Neuroscience*. *Copyright © 2014 Bissonette, Gentry, Padmala, Pessoa and Roesch. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## Translational studies of goal-directed action as a framework for classifying deficits across psychiatric disorders

## **Kristi R. Griffiths , Richard W. Morris and Bernard W. Balleine \***

Behavioural Neuroscience Laboratory, Brain and Mind Research Institute, University of Sydney, Camperdown, Sydney, NSW, Australia

#### **Edited by:**

Dave J. Hayes, University of Toronto, Canada

#### **Reviewed by:**

M. Gustavo Murer, Universidad de Buenos Aires, Argentina Lesley K. Fellows, University of Oxford, UK

#### **\*Correspondence:**

Bernard W. Balleine, Behavioural Neuroscience Laboratory, Brain and Mind Research Institute, University of Sydney, Level 6, 94 Mallett Street, Camperdown, Sydney, NSW 2050, Australia e-mail: bernard.balleine@sydney.edu.au

The ability to learn contingencies between actions and outcomes in a dynamic environment is critical for flexible, adaptive behavior. Goal-directed actions adapt to changes in action-outcome contingencies as well as to changes in the reward-value of the outcome. When networks involved in reward processing and contingency learning are maladaptive, this fundamental ability can be lost, with detrimental consequences for decision-making. Impaired decision-making is a core feature in a number of psychiatric disorders, ranging from depression to schizophrenia. The argument can be developed, therefore, that seemingly disparate symptoms across psychiatric disorders can be explained by dysfunction within common decision-making circuitry. From this perspective, gaining a better understanding of the neural processes involved in goal-directed action, will allow a comparison of deficits observed across traditional diagnostic boundaries within a unified theoretical framework. This review describes the key processes and neural circuits involved in goal-directed decision-making using evidence from animal studies and human neuroimaging. Select studies are discussed to outline what we currently know about causal judgments regarding actions and their consequences, action-related reward evaluation, and, most importantly, how these processes are integrated in goal-directed learning and performance. Finally, we look at how adaptive decision-making is impaired across a range of psychiatric disorders and how deepening our understanding of this circuitry may offer insights into phenotypes and more targeted interventions.

**Keywords: goal-directed action, basal ganglia, amygdala, schizophrenia, ADHD, depression**

## **GOAL-DIRECTED ACTION AND ITS RELEVANCE TO PSYCHIATRY**

Flexible behavior is fundamental for adapting to a changing environment. In this context, learning the consequences of an action and the value of those consequences are critical precursors for choosing the best course of action. Impairment in either process, or a failure to integrate them with action selection, leads to aberrant decision-making, with detrimental consequences for achieving goals and real-world functioning. Dysfunctional decision-making is common across a range of psychiatric disorders, and indeed, it has been argued that many psychiatric symptoms are associated with dysfunction in either learning or reward circuitry (cf. Nestler and Carlezon, 2006; Martin-Soelch et al., 2007). Determining how the brain supports each step in achieving flexible, goal-directed behavior is, therefore, not only a major goal of decision neuroscience, but may also provide valuable insight into the neurobiology and attendant functional disabilities associated with psychiatric illness.

Decades of research in associative learning have provided key insights into the behavioral and biological processes that mediate goal-directed action. One advantage of this approach has been the development of testable structural and functional hypotheses, and the invention of critical behavioral paradigms specifically to assess predictions from these hypotheses. We argue that this approach provides a unique opportunity to systemically explore the decision-making deficits commonly observed in clinical populations, and allows for the classification of a variety of decisionmaking impairments within a common framework. In this review, we first describe the psychological determinants of goal-directed behavior, and the evidence for how these processes map onto specific neural circuits. We will then use this framework to assess how these processes may be affected in common symptoms within three clinical disorders: schizophrenia, attention-deficit hyperactivity disorder (ADHD), and depression. Behavioral and neurobiological heterogeneities exist within traditional disorder classifications, as well as commonalities across diagnostic boundaries. We argue that knowledge of specific decision-making processes and their neural bases may provide a unifying framework, using which we can classify deficits across psychiatric disorders to produce a functionally–and biologically -driven understanding of psychopathology.

#### **WHAT IS GOAL-DIRECTED ACTION?**

Formally, goal-directed action reflects the integration of two sources of information: (1) knowledge of the causal consequences or outcome of an action; and (2) the value of the outcome (Dickinson and Balleine, 1994; Balleine and Dickinson, 1998). The integration of both of these features, causal knowledge and reward value, is essential in producing goal-directed actions. Impairments in such actions can arise through a deficiency in either process, or through an inability to integrate them appropriately to guide decision-making. We will first discuss each of these features in turn and the key neural substrates that current research suggests are involved in these processes. We will then turn to potential deficits in these processes using examples of specific psychopathology, and in particular, how they are related to symptoms common to depression, schizophrenia and ADHD.

#### **CAUSAL LEARNING AND ACTION-OUTCOME ENCODING**

Knowledge regarding the causal consequences of specific actions emerges from the experienced contingency. Such contingencies can be positive, promoting performance of an action, or inhibitory; i.e., in some situations actions may prevent a desired outcome and, in these situation, actions should be withheld (Dickinson, 1994). Considerable research using tasks such as the Iowa Gambling Task (IGT; Bechara et al., 1994) and the Wisconsin Card Sorting Task (WCST; Grant and Berg, 1948) suggests that humans and rats are exquisitely sensitive to feedback contingent on their actions, and can flexibly update their choices based on that feedback. However, because specific choice problems are signaled using unique discriminative or localized cues in these tasks, choice performance could reflect knowledge of the actionoutcome contingency or associations between the action or the outcome with these task-related cues. This is a non-trivial distinction; as we shall review below, research has shown that different psychological processes and neural circuits exert control when actions are guided by environmental stimuli or by the actionoutcome contingency (see Balleine and Ostlund, 2007; Balleine and O'Doherty, 2010 for reviews).

Experimentally, we are able to determine the degree to which choice is guided by the action-outcome contingency using contingency degradation tests. In such tests a specific action-outcome contingency is degraded by introducing an outcome in the absence of its associated action, thereby reducing the causal relationship between them. This treatment decreases the performance of the degraded action in goal-directed agents (Hammond, 1980; Balleine and Dickinson, 1998). For example, Balleine and Dickinson trained rats to perform two actions, lever pressing and chain pulling, with one action earning sucrose and the other, food pellets. They subsequently delivered one of the two outcomes noncontingently, such that the probability of receiving that outcome was the same whether the rat performed its associated action or not. This produced a selective decrease in the performance of the degraded action. Similarly, it has been demonstrated in healthy humans that the degree of contingency degradation is negatively correlated with the rate of performance and with judgments regarding how causal an action is with respect to its outcome (Shanks and Dickinson, 1991; Liljeholm et al., 2011).

#### **A SPECIFIC CORTICOSTRIATAL CIRCUIT MEDIATES THE CAUSAL EFFECTS OF ACTIONS**

Systematic use of contingency degradation tasks in rodent studies has identified specific regions of prefrontal cortex and dorsomedial striatum necessary for encoding the action-outcome contingency (Corbit and Balleine, 2003; Yin et al., 2005; Lex and Hauber, 2010). In humans, there is evidence that homologous regions to those in rodents, i.e., the medial prefrontal cortex (mPFC) and anterior caudate, play a similar role in contingency sensitivity (cf. Balleine and O'Doherty, 2010). Tanaka et al. (2008) and Liljeholm et al. (2011) manipulated experienced action-outcome contingencies, and observed positive modulation of blood oxygenation level dependent (BOLD) activity in the human mPFC, and anterior caudate nucleus (aCN). Furthermore, mPFC activity reflected the local experienced correlation between responding and reward delivery, consistent with a role in the online computation of contingency (Tanaka et al., 2008). Activation of the aCN can also occur even fictively, in cases where a contingency between action and outcome is perceived where one does not actually exist (Tricomi et al., 2004), whereas subjective causality judgments have been shown to correlate with activity in the mPFC, along with the dorsolateral prefrontal cortex (dlPFC), a region implicated in top-down cognitive control (Tanaka et al., 2008). As shown in the green in **Figure 1A**, these data suggest that signals produced in the mPFC may be relayed to the aCN, where changes in contingency can be assimilated with evaluative information from other cortical regions.

Further evidence for the importance of the caudate in contingency sensitivity and in guiding action selection comes from studies in non-human primates. Samejima et al. (2005) recorded from striatal neurons during a choice task in which monkeys made left or right actions to obtain reward. Importantly, on some trials, action-outcome contingencies were similar whereas on others they differed so that activity related to the action value (in this instance, the strength of the action-outcome contingency) could be dissociated from the motor choice. They found that a large number of striatal neurons encoded action values, which subsequently influenced the probability of selecting a particular action. Lau and Glimcher (2007) also found populations of neurons in the caudate that encoded actions and outcome post-choice. The temporal correlation of neuronal firing rates with behavior suggested that the caudate not only represents the contingency of potential options, but might also update this information once the outcome has been received.

#### **THE ROLE OF VALUE IN GOAL-DIRECTED DECISION-MAKING**

In addition to causal knowledge, determining the current value of available outcomes in the context of current internal states or contexts is also critical for adaptive decision-making. For example, a state of hunger increases the desirability or incentive value of food relative to a satiated state, and increases its motivational impact. Outcome revaluation procedures exploit these variations in value. A common means of changing the value of a specific food is using sensory-specific satiety (Rolls et al., 1981). For example, in studies in which rats were trained to perform two actions for distinct outcomes, giving them an extended opportunity to eat one or other outcome altered the desirability of that outcome without affecting the value of the other uneaten outcome (Balleine and Dickinson, 1998). When given the opportunity to choose between the two actions in the

absence of any reward delivery (to prevent learning about the association between the action and the new outcome value during the test) the rats clearly preferred the action that had previously earned the outcome they had not eaten. Selective decreases in the performance of actions associated with a devalued outcome provide clear evidence that, in conjunction with knowledge of the action-outcome contingency, action selection is governed by the current value of the outcome.

An alternate means of revaluing the outcome used in animal research is conditioned taste aversion whereby an outcome is paired with a mild toxin such as lithium chloride that induces gastric malaise. In humans disgust can also be a useful tool for devaluing outcomes. For instance, food desirability ratings can be decreased considerably when an otherwise preferred outcome has been paired with an aversive taste (e.g., Baeyens et al., 1990).

#### **THE OFC AND vmPFC PLAY A ROLE IN ENCODING VALUE RELATIVE TO THE CURRENT MOTIVATIONAL STATE**

The OFC and, more broadly, the vmPFC, illustrated in red in **Figure 1A**, have long been argued to be critical for signaling the current value of an outcome. Single unit recording studies in hungry non-human primates found unit responses in the caudolateral OFC during presentation of a pleasant odor or taste, which decreased to baseline when the monkey were satiated (Rolls et al., 1989). Similarly, when humans were presented with food outcomes, the degree of hunger and pleasantness caused graded OFC/vmPFC BOLD activity (Morris and Dolan, 2001; Kringelbach et al., 2003) that was reduced after satiation with the presented food (O'doherty et al., 2000; Small et al., 2001; Valentin et al., 2007). Interestingly, this reduction in activity was evident even when using instructed devaluation, where participants were simply told via a red X over a predictive stimulus that the outcome was no longer valuable (de Wit et al., 2009) suggesting that revaluation, whether through visceral or cognitive treatments, affects value via a common neural pathway. These data advance the idea that the OFC undertakes simple economic valuation and emphasize its role in determining outcome value in the context of the *current* motivational state. Jones et al. (2012) have further developed this idea, arguing that the OFC is required when value is inferred from associative structures (i.e., value is computed based on the current state), but not when relying on pre-computed values stored from previous experience.

It is important to note that BOLD activation during evaluation has been reported within both the lateral and medial portions of the OFC. There is, however, evidence for cytoarchitectural and functional heterogeneity within the OFC (Carmichael and Price, 1995; Elliott et al., 2000; Kahnt et al., 2012), suggesting that studies using reward-predictive cues are utilizing alternate or additional learning processes. Though there is still considerable debate on this topic, a converging view is that the mOFC is involved in updating the expected values of different experienced outcomes, whereas the lateral OFC is responsible for the formation and updating of values derived from Pavlovian stimulus-outcome associations (Walton et al., 2010; cf Balleine et al., 2011; Fellows, 2011; Noonan et al., 2011, 2012; Rudebeck and Murray, 2011; Klein-Flügge et al., 2013). Both the predicted value of an outcome based on the presence of a Pavlovian cue, and the experienced value of an instrumental outcome, are incentive processes that play an important role in motivating behavior. Due to the differing circuitry and learning processes (instrumental vs. Pavlovian) however, paradigms that disentangle these processes provide clearer information.

#### **THE INFLUENCE OF A LIMBIC CORTICO-STRIATAL CIRCUIT ON THE VALUE OF OUTCOMES AND CUES THAT PREDICT OUTCOME DELIVERY**

Whereas the mOFC is computing current outcome value, the basolateral amygdala (BLA) plays a more fundamental role, linking value information with the sensory features of the reward or reward-related cues (see **Figure 1A**). A series of studies by Balleine et al. (2003) found that lesions of the BLA attenuated the sensitivity of rats to outcome devaluation, both when tested in extinction and with the outcome present. Furthermore, BLA lesions have been found to abolish the selective excitatory effects of reward-related cues whilst sparing the general motivational effects that such cues exert over responding (Corbit and Balleine, 2005). In humans, Jenison et al. (2011) acquired single neuron recordings from the BLA whilst subjects made monetary bids on food items that were presented to them as pictorial stimuli. Firing rates were linearly related to the monetary value assigned to food item stimuli, supporting a role for the BLA in assigning value to stimulus events. The strength of association between incentive value (either positive or negative) and both the features of outcomes and predictive cues not only determines their valence but also the magnitude of evaluative judgments, in keeping with a range of human imaging studies that have concluded the amygdala provides an overall magnitude signal for value judgments, or the interaction between intensity and valence (Anderson et al., 2003; Arana et al., 2003; Small et al., 2003; Winston et al., 2005).

Extensive anatomical connectivity exists between the OFC and BLA (see **Figure 1A**; Stefanacci and Amaral, 2002; Ghashghaei et al., 2007) allowing them to work closely together in encoding and retrieving value information (see Holland and Gallagher, 2004, for a review). Indeed, damage to the BLA can produce similar deficits to those observed from damage to the OFC (Hatfield et al., 1996; Baxter et al., 2000). However, no brain region acts in isolation, something clearly demonstrated when brain structures are left intact and only their anatomical connections with other structures are severed. Using OFC-BLA contralateral disconnection lesions, Zeeb and Winstanley (2013) found that rats were unable to update their choice preference following reward devaluation. This effect occurred both when the reward was delivered during test and also during extinction when rats needed to rely on stored representations of the outcome. The rats with disconnected OFC and BLA, however, did not differ from controls in their press rates or response latencies, suggesting an impairment specific to altering the value of a particular reward rather than a general reduction in motivation. Similar effects have been observed in humans where structural and functional connectivity between the OFC and BLA was found to correlate with rate of acquisition on a reversal learning task (Cohen et al., 2008).

The nucleus accumbens (NAc) also receives excitatory afferents from the OFC and BLA (amongst other regions), and selectively gates information projecting to basal ganglia output nuclei (**Figure 1A**; Alheid and Heimer, 1988; Groenewegen et al., 1999). It is often described as the limbic-motor interface, mediating the effect of reward value on action selection (Mogenson and Yim, 1991). Lesions of the NAc core impair the ability of rats to selectively reduce responding after outcome devaluation, demonstrating reduced sensitivity of instrumental performance to changes in outcome value (Corbit et al., 2001; Corbit and Balleine, 2011; Laurent et al., 2012) Importantly, lesions of the NAc also cause a reduction in the vigor of performance, indicating that this region may be involved in how the general motivating properties of reward-related stimuli affect performance (Balleine and Killcross, 1994; Corbit et al., 2001). Interestingly, NAc lesions do not impair sensitivity to selective contingency degradation, revealing that this region does not itself encode the action-outcome contingency but, rather, brings changes in reward value to bear on performance (Corbit et al., 2001). These key evaluative circuits are represented by the red connections in **Figure 1A**.

#### **ACTION VALUES: THE INTEGRATION OF CONTINGENCY AND VALUE**

The value of an action is a product of its contingency with a particular outcome and the desirability of that outcome. As a consequence, interest has grown in the analysis of the neural circuits involved in computing these action values. Studies using trial-and-error action-based learning tasks have reported action value-related signals in the supplementary motor area, where actions are presumably planned before execution. In contrast, BOLD activity in the vmPFC was modulated by the expected reward signal of the chosen action, suggesting that this region provides the agent with feedback about the consequences of their actions to guide future choices (Gläscher et al., 2009; Wunderlich et al., 2009; FitzGerald et al., 2012; Hunt et al., 2013). Camille et al. (2011) found that humans with dorsal anterior cingulate cortex (dACC) damage were unable to maintain the correct choice between actions after positive feedback, suggesting that this region is critically involved in updating action values, perhaps passing feedback from the vmPFC to the action planning areas in the supplementary motor areas via the aCN.

Top-down cognitive control exerted by such structures as the dlPFC and dACC may also modulate the integration of value and contingency, and its conversion into performance. Kim and Shadlen (1999) and Wallis and Miller (2003) found dlPFC neurons that encoded both reward value and the forthcoming response, whereas Kim et al. (2008) found neurons that ramped up or down in their firing rate with increasing or decreasing action values until a choice was made. In the ACC, neural signals resembling the difference between action values, or a combination of movement intention and reward expectation, have been reported (Matsumoto et al., 2007; Seo and Lee, 2007; Wunderlich et al., 2009). Furthermore, lesions of this area in non-human primates and humans produces deficits in action-based choice (Kennerley et al., 2006; Camille et al., 2011). Although there is less agreement about the distinctions in function of the dlPFC and ACC, it is clear that disturbances within these regions radically alter goal-directed choice.

We do know however that the anterior caudate, a part of the associative striatum, is a critical node in the goal-directed network, receiving evaluative input from the BLA and OFC, as well as contingency input from the dlPFC and mPFC. This is supported by data showing that the integration of dopamine and glutamate neurotransmission within this region enables learning and action control by shaping synaptic plasticity and cellular excitability (Shiflett and Balleine, 2011a). In particular, the extracellular signal-regulated kinase (ERK) is particularly important for goaldirected action control due to its sensitivity to combined DA and glutamate receptor activation (Shiflett et al., 2010; Shiflett and Balleine, 2011b). Thus, perturbation of ERK activation associated with various forms of psychopathology and/or drug abuse may produce deficits in goal-directed control. Nevertheless, the role of this region in mediating information from limbic and cortical networks has only relatively recently been recognized in other forms of psychopathology such as that involved in schizophrenia (Howes et al., 2009; Kegeles et al., 2010; Simpson et al., 2010).

#### **SUMMARY OF NEUROBIOLOGY OF GOAL-DIRECTED LEARNING**

In summary, the vmPFC is a functionally complex region critically involved in networks that compute and update outcome values based on feedback or changes in state. The BLA assists in this process by associating incentive value with the sensory information that informs the agent of the reward properties of outcomes, whilst the NAc brings this evaluative information to bear on performance. Simultaneously, the associative striatum and mPFC are also involved in the learning of action-outcome associations, providing information on how to obtain desired outcomes. Together, these processes are integrated in the associative striatum to produce goal-directed behavior. For the purpose of brevity, we have focused on what we believe are the key neural regions involved in goal-directed learning. It must be acknowledged, however, that many other regions likely contribute to these processes in ways that are not yet fully understood.

#### **STIMULUS-DRIVEN EFFECTS ON INSTRUMENTAL BEHAVIOR**

Multiple learning systems are involved in the production of healthy everyday behavior. So far we have focused on behavior guided by goals rather than cues. Goal-directed processes allow for flexible choices in the face of changing environmental contexts and conditions. Under stable conditions however, the consequences of actions need not be continually assessed. In these instances, habitual actions, established by the formation of stimulus-response associations, allow reflexive, cue-driven responses to occur at higher speeds and with lower cognitive load (see **Figure 1B**). The associative systems mediating goaldirected actions and habits are thought to coexist and compete for behavioral control in adaptive decision-making (Dickinson and Balleine, 1993). Another major learning process influencing behavior is the formation of Pavlovian stimulus-outcome associations and conditioned responding (see **Figure 1C**). Cues associated with reward are able to evoke reward anticipation, which may subsequently guide or bias instrumental choices. Both rewardpredictive cues and the experienced value of an instrumental outcome are important incentive processes that play an essential role in motivated behavior. Importantly however, although both may be able to induce reward approach behavior, Pavlovian cues exert their effects on actions through stimulus, rather than outcome value, control.

As depicted in **Figure 1**, these learning systems are situated in functionally organized cortico-basal ganglia loops. The cortical regions of each system send topographically organized inputs to the striatum—motivational or limbic input to the ventral striatum, associative input to the aCN and anterior putamen, and sensorimotor input to the posterior putamen (Nakano, 2000). From the striatum, GABA-ergic medium spiny neurons (MSNs) project to the principle striatal output nuclei, the substantia nigra pars reticulata (SNr) either directly or indirectly via the globus pallidus pars externa (GPe) and subthalamic nucleus (STN). Whereas MSNs in the direct pathway predominantly express dopamine D1 receptors and activate behavioral functions, those in the indirect pathway express dopamine D2 receptors and tend to suppress behavior (Albin et al., 1989). The ascending dopaminergic system, projecting to the striatum from the substantia nigra pars compacta (SNc) and ventral tegmental area (VTA), plays an important role in modulating activity within these pathways due to their differential expression of D1 and D2 receptors. These modulate the activity of the MSNs bidirectionally; whereas dopamine increases the activity in D1 expressing MSNs, it reduces the activity of D2 expressing MSNs (Gerfen and Surmeier, 2011).

### **THE BREAKDOWN OF GOAL-DIRECTED PROCESSES IN PSYCHIATRIC AND NEURODEVELOPMENTAL DISORDERS**

The nature of the interaction and cooperation between goaldirected and habitual control processes during decision-making has particular implications should problems arise in the cognitively demanding goal-directed system. Under such conditions, behavioral control may become dominated by dysregulated habitual control, resulting in the loss of flexibility of thought, and the increased stereotypy and behavioral disinhibition characteristic of many psychiatric conditions. Deficits in incentive processes may also produce a range of motivational dysfunctions. Having outlined these processes and their interaction in healthy decision-makers, together with the key neural systems involved above, we turn to consider whether deficits in goal-directed decision-making in psychiatric disorders map onto a common framework. Here we review select evidence for patterns of deficits in outcome sensitivity, action-outcome contingency awareness, and in the integration of these features with action selection in three disorders known for their motivational and cognitive deficits: schizophrenia, ADHD and depression.

#### **SCHIZOPHRENIA**

Motivational and associative learning dysfunction have long been noted in schizophrenia, and have been implicated in positive, negative and cognitive symptomology (Gold et al., 2008). It is often noted that individuals with schizophrenia experience difficulties using emotional states, prior rewards and goals to drive goal-directed action (Barch and Dowd, 2010); i.e, the relationship between value representations and action selection appears to be lost (Heerey and Gold, 2007; Gold et al., 2008; Heerey et al., 2008). We propose that this is due to what amount to functional disconnections within the cortico-striatal loops responsible for integrating evaluative and contingency learning for goal-directed action selection.

#### **Reduced sensitivity to changes in reward value**

Negative symptoms such as anhedonia (an inability to experience pleasure) and avolition (a reduced motivation to engage in motivated goal-directed behavior) seem to suggest valuation and action selection deficits are primary in this disease. Anhedonia may be produced by a breakdown in the evaluative circuits responsible for the actual consummatory pleasure experienced from the reward (i.e., the red circuit in **Figure 1A**). Recently however, a number of studies have shown that, on experiencing or consuming rewards, hedonic ratings are often not significantly reduced compared to controls (Burbridge and Barch, 2007; Gard et al., 2007; Heerey and Gold, 2007) and we have found similar effects in the lab. If evaluative learning is intact, then the critical deficit may lie in anticipating hedonic consequences (reward value) or in using experienced reward values to guide actionselection. Numerous behavioral and neuroimaging studies have focused on whether patients can anticipate reward values. For example, patients with severe avolition fail to choose stimuli associated with monetary reward over a stimulus indicating the avoidance of monetary loss (i.e., no reward) (Gold et al., 2012). This deficit in reward anticipation is consistent with neuroimaging evidence that ventral striatal responses to cues predicting reward are dulled in schizophrenia (Juckel et al., 2006a), including amongst unmedicated patients (Juckel et al., 2006b). Patients also have aberrant neural responses to rewards themselves, including predicted and unpredicted rewards (Waltz et al., 2009; Morris et al., 2012). However no study to date has tested whether patients can adjust their actions solely on the basis of experienced reward values. In a recent study, we tested whether patients with schizophrenia could use the anticipated or experienced reward value to select actions. Patients were able to learn action-outcome associations, and subjectively reported reductions in outcome value after an outcome devaluation procedure, however they did not use this updated outcome knowledge to effectively guide their choices, suggesting that the ability of patients to integrate the values of rewards with action selection processes is deficient. Importantly, BOLD activity in the caudate nucleus during the test requiring this integration was also deficient in patients. Moreover, reduced neural responses in the head of the caudate predicted more severe negative symptoms. This is consistent with recent evidence that neuropathology in schizophrenia, including upregulation of striatal D2 receptor density and occupancy, is most prevalent in the associative regions of the striatum (Buchsbaum and Hazlett, 1998; Abi-Dargham et al., 2000; Howes et al., 2009; Kegeles et al., 2010). On the other hand, patients were able to select actions on the basis of the anticipated reward value, when a cue predicting the availability of reward was presented, albeit not to the same extent as healthy adults (Balleine and Morris, 2013). Thus, the integration of reward values with action selection appears to be impaired in schizophrenia. This particularly affects goal-directed actions when cues are not present to indicate the consequences of action.

The caudate is a critical site for goal-directed actions but it does not function in isolation. In addition to aberrant regional activity in schizophrenia, there is also evidence for functional disconnection of the caudate from its cortical afferents, which can also be found during the prodromal state (Buchsbaum et al., 2006; Yan et al., 2012; Fornito et al., 2013; Quan et al., 2013; Quidé et al., 2013; Wadehra et al., 2013). Thus, the caudatecortical disconnection in schizophrenia is a critical target for understanding the deficit in goal-directed behavior and predicting functional outcomes associated with the disease.

#### **Changes in contingency awareness**

Cognitive deficits are the most pervasive and difficult to treat aspects of schizophrenia (Green, 1996). In particular, any deficit in the ability to form and use A-O associations appropriately and learn about the consequences of our everyday choices is likely to have a large impact on social and occupational functioning. Multiple studies have suggested that the initial acquisition of probabilistic contingencies is relatively unimpaired in schizophrenia, with the exception of some reports of slower rates of acquisition (Weickert et al., 2002; Kéri et al., 2005; Waltz and Gold, 2007). When contingencies are reversed many studies have shown schizophrenic patients do show significant impairments (Waltz and Gold, 2007; Murray et al., 2008), suggesting patients are insensitive to changes in action-outcome contingency. However, distinguishing this impairment in reversal learning from slower acquisition more generally has not been convincingly demonstrated. Using cognitive modeling, however, Strauss et al. (2011) found that patients with schizophrenia have a reduced tendency to explore alternative actions in an uncertain environment. This perseverative style of responding during uncertainty is consistent with greater habitual control of actions. A weakened sensitivity to the action-reward correlation and the predominant use of an S-R learning strategy is also consistent with the fact that rapid learning from trial-by-trial feedback is often impaired but more gradual learning remains intact (Kéri et al., 2005; Gold et al., 2008).

At a neural level, the associative striatum plays an integral role in acquiring A-O contingencies, detecting contingency changes and flexibly using this information during the process of action selection. As reviewed above, functional deficits in the associative striatum as well as pathology in cortical afferents appear early in the pathogenesis of schizophrenia and may be a risk factor for the disease. In this case, a deficit in learning action-outcome contingencies, which critically depends on this circuit, may stand in as an important marker of brain function. However, at present the status of contingency learning deficits in schizophrenia is unclear. Reversal learning tasks such as the IGT or the WCST are generally controlled by reward-related stimuli rather than by the relationship between action and outcome, which makes it difficult to discern whether any deficits are due to altered Pavlovian or instrumental learning. In addition, in reversal learning tasks, it is difficult to establish whether changes in outcome value or in contingency are driving choices. Thus, the use of contingency degradation tasks within this cohort will be critical to provide convincing evidence regarding the level of impairment in contingency awareness and the functional status of the related circuits.

#### **Schizophrenia summary**

In summary, during goal-directed learning, patients with schizophrenia are only mildly or are unimpaired in their subjective valuation assessments, and in the activation of prefrontal regions that support them. Dysfunction in the associative striatum and its cortical afferents, however, may interfere with the ability to modulate action selection using value information. Evidence also suggests that patients with schizophrenia are able to encode initial A-O associations, but they may be impaired at updating associations for flexible use in action selection. Taken together, these impairments in integrating the key components of goal-directed behavior suggest that patients with schizophrenia may over rely on habit learning and habitual strategies, predicting relatively intact functioning of the circuitry mediating habitual control but not goal-directed performance.

#### **ADHD**

Altered sensitivity to reinforcement is acknowledged as an important etiological factor in a number of theoretical frameworks of ADHD (Barkley, 1997; Sergeant et al., 1999; Castellanos and Tannock, 2002; Sagvolden et al., 2005; Frank et al., 2007; Tripp and Wickens, 2008; Sonuga-Barke and Fairchild, 2012). ADHD is characterized by symptoms of inattention, hyperactivity and impulsivity, consistent with dysregulation of top-down control processes modulating goal-directed control. A number of researchers have argued that ADHD is a motivational problem, whereby individuals are unable to use intrinsic motivation to guide choice performance (Douglas, 1989; Sergeant et al., 1999). This is supported by evidence that children with ADHD perform well on continuous reinforcement schedules, whereas their performance deteriorates on partial reinforcement schedules where the consistent extrinsic motivation of reward is not provided (Parry and Douglas, 1983; Luman et al., 2008).

Dopaminergic dysfunction clearly plays a key role in ADHD symptomology. The primary treatment for ADHD, Methylphenidate, preferentially blocks the reuptake of DA in the striatum (Schiffer et al., 2006), and studies have demonstrated its effectiveness in normalizing reinforcement sensitivity in ADHD relative to placebo (Tripp and Alsop, 1999; Frank and Claus, 2006). Furthermore, Volkow et al. (2012) has proposed that disruption of D2/D3 receptors is associated with the motivation deficits observed in ADHD, which may in turn contribute to attention deficits. Attention was found to be negatively correlated with D2/D3 receptor availability in the left NAc and caudate (Volkow et al., 2009), regions key to reward valuation and contingency awareness in goal-directed action. We hypothesize that motivational problems stem primarily from an inability to predict the rewarding consequences of cues or actions. As a consequence actions may be poorly controlled or regulated resulting in inappropriate responses to the situation and undesirable consequences.

#### **The dopamine transfer deficit theory**

The Dopamine Transfer Deficit theory of ADHD (Tripp and Wickens, 2008, 2009) proposes that altered phasic dopamine responses to reward-predictive cues results in blunted stimulusoutcome associations, and hence blunted reward anticipation. In this sense, motivational deficiencies may be derived from a lack of stimulus-outcome contingency awareness (i.e., an impairment within the circuitry detailed in **Figure 1C**). The relatively consistent finding of hypo-activation in the ventral striatum during reward anticipation supports this idea (Scheres et al., 2007; Ströhle et al., 2008; Plichta et al., 2009; Hoogman et al., 2011; Carmona et al., 2012; Edel et al., 2013; Plichta and Scheres, 2013). Wilbertz et al. (2012) found increased OFC activation during outcome delivery consistent with increased excitation to reward; however, as reward-related stimuli were generally less successful at inducing reward anticipation, it may also reflect an aberrant prediction error-like response. Overall, rather than suggesting that reward sensitivity is impaired, the evidence seems to support the notion that an inability to anticipate reward may reduce motivation or impair the ability to select the relevant action.

In comparison to schizophrenia, both patient groups have intact reward sensitivity, however the pathologies can be dissociated by the role of *predicted* reward-values and *experienced* reward-values on action-selection. In ADHD, we expect to see impairment in selecting actions on the basis of predicted reward (e.g., a deficit in outcome specific Pavlovian-to-instrumental transfer); whereas in schizophrenia the deficit is related to using experienced reward values to guide action selection (e.g., a deficit in outcome specific devaluation). The amount of overlap between these two groups should, therefore, be predicted to depend on the extent to which both share neuropathology in the ventral striatum, which will disrupt dopamine signaling due to hyper- or hypodopaminergia, regardless.

#### **Incentive learning deficits, response inhibition and impulsivity**

Response inhibition and impulsivity are key deficits exhibited in ADHD even when executive function demands are low (Wodka et al., 2007); both children and adult subjects are slower to inhibit responses during the go/no-go or stop-signal reaction time (SSRT) tasks, and make more errors than age-matched controls (Schachar et al., 1995; Purvis and Tannock, 2000; see Solanto, 2002 for a review). Lesions of the BLA and NAc both increase impulsive choice on a delay-discounting task in rats (Winstanley et al., 2004), and measures of impulsivity are generally negatively correlated with white matter integrity in right OFC fiber tracts in adults with ADHD. Thus impulsivity may be induced by dysfunction in key incentive processing regions, or alternatively, these regions may be underutilized due to an over reliance on reflexive actions that are not based on the value of consequences.

#### **Changes in contingency awareness**

Tripp and Wickens (2008) postulate that stimulus-outcome associations are disturbed in ADHD due to a lack of transfer of dopamine firing from reward receipt to reward-predictive cues. To date, however, there have not been any comparable studies assessing whether this is also the case for action-outcome learning. We predict that due to dopamine dysregulation within the associative striatum, contingency awareness will be deficient perhaps for both cue and action–based associations with specific outcomes. Firstly, reduced salience or attention allocation due to dysfunction in DA firing may inhibit the formation of action-outcome associations. Furthermore, when a temporal delay occurs between an action and its outcome, DA dysfunction may generate difficulties in "credit assignment"—deciding to which recent action one should attribute the outcome (Johansen et al., 2009). This difficulty could contribute to the delay aversion often documented in ADHD (Sonuga-Barke, 2002), and the easy distraction by extraneous stimuli. For instance, Carlson et al. (2000) found that, relative to controls, ADHD children were more likely to attribute success on an arithmetic task to luck, which seems to support reduced awareness of action-outcome causality. The dopamine transfer deficit theory also predicts that in ADHD, smaller anticipatory dopamine signals relative to the response to actual reinforcers would result in a greater influence of the most recent contingency than longer-term reinforcement history (Tripp and Wickens, 2008). This could result in faster extinction under partial reinforcement, or increases in the performance of occasionally rewarded, but overall suboptimal, actions.

#### **Caudate impairments and action selection in ADHD**

Meta-analyses have shown that the most consistent gray matter reductions in ADHD occur in the caudate, a region critical for goal-directed behavior. This morphological deficit was worse in samples with lower levels of stimulant medication, suggesting that dopamine normalization may counteract caudate atrophy (Valera et al., 2007; Nakao et al., 2011). Impairments in the striatum likely affect both contingency awareness and their integration with action selection processes. Reduced structural connectivity may also hinder this integration; indeed, ADHD patients have been shown to have anomalous white matter integrity in fronto-striatal and premotor (PM) regions relative to age matched controls (Ashtari et al., 2005; Silk et al., 2009; Konrad and Eickhoff, 2010).

#### **ADHD summary**

In summary, we hypothesize, with others, that motivational impairments in ADHD arise due to an inability to accurately predict the occurrence of rewarding outcomes. This in turn reduces the salience of reward predictive cues and optimal actions potentially contributing to attentional deficits. Dopamine dysfunction within the striatum seems to be a key factor in this contingency awareness impairment. Furthermore, a greater reliance on recent rather than longer-term reinforcement history could explain the rapid extinction of learnt associations, and why patients with ADHD respond better to continuous reinforcement schedules.

## **DEPRESSION**

The major diagnostic guidelines state that individuals experiencing depressive episodes often have difficulty making decisions (DSM IV, APA, 2000; ICD-10, WHO, 1992). Traditionally, it has been assumed that this was due to primary motivational impairments, however cognitive deficits associated with the disorder are becomingly increasingly well documented (Lee et al., 2012). We predict that whereas outcome valuation will be strongly affected in those experiencing anhedonia, contingency sensitivity impairments may also be detected in a subset of cognitivelyimpaired patients. Further, reward learning and cognitive deficits may persist during periods of euthymia, predisposing individuals to future depressive episodes.

### **Deficits in reward sensitivity**

Depression is commonly characterized by blunted reward responsiveness (Henriques and Davidson, 2000; Pizzagalli et al., 2008; McFarland and Klein, 2009) and behavioral neglect of positive stimuli (Clark et al., 2009), which is reflected in the symptoms of anhedonia, social withdrawal and reduced activity level. As experienced rewards are no longer pleasurable, it is easy to envisage how action control could become biased away from goal-directed actions toward habits, which require only the preservation of a sufficient reinforcement signal to form stimulusresponse associations.

During both reward and punished responding in depressed subjects, blunted responses are observed in the medial caudate and ventromedial OFC (Elliott et al., 1998). This supports behavioral accounts of blunted reward sensitivity. Interestingly, McCabe et al. (2009) found that, in remitted depressed patients, there were decreased reward responses in the ventral striatum, caudate and anterior cingulate, despite subjective ratings being the same as controls, suggesting that altered reward sensitivity occurs independent of mood symptoms, and may actually be a predisposing factor in the etiology of depressive episodes.

One prominent theory proposes that a defect in the topdown inhibition of the amygdala by the vmPFC may underlie depression symptoms (Myers-Schulz and Koenigs, 2011). For instance, Friedel et al. (2009) reported a negative correlation between depressive symptom severity and connectivity between the mOFC and the amygdala. As discussed earlier, the amygdala and OFC and their connectivity are required for the encoding and use of value-based information. Therefore impairment in either region, or reduced connectivity between them, will likely hamper the updating of value and its integration to mediate goal-directed choice. Due to reduced OFC-BLA connectivity, we predict that individuals with severe anhedonia will be unable to alter their choices appropriately after outcome devaluation.

Significantly reduced ventral striatal activity to positive stimuli has also been observed in depressed patients (Epstein et al., 2006; Robinson et al., 2012; Stoy et al., 2012), which may reflect a deficit in using value information to guide action selection. These studies employed predominantly Pavlovian learning processes and, therefore, the focus was generally on assessing anticipation of reward rather than how value knowledge was used to guide instrumental choices. Nevertheless, Stoy et al. (2012) discovered that treatment with the common antidepressant, escitalopram, normalized anticipatory reward signals in the ventral striatum, highlighting how medications affecting reward circuitry could be effective in improving depressive symptoms. In addition, deep brain stimulation to the bilateral NAc in refractory depression has shown promising results for reduction of the symptoms of anhedonia (Schlaepfer et al., 2008; Malone et al., 2009).

#### **Deficits in contingency awareness**

Although it is evident that anhedonia diminishes the impact of reward processes in goal-directed action, there is significantly more debate about how causal awareness is affected in depression. Depressed individuals often experience symptoms of learned helplessness, which may reflect dysfunction in causal knowledge. Learned helplessness is essentially an error in attribution of control (Miller and Seligman, 1975) in the sense that a depressed person may have aberrant beliefs about the causality of their actions in achieving a goal, or the lack thereof, and so not initiate an action. Using Bayesian modeling, Lieder et al. (2013) argued that generalization of action-outcome contingencies is able to account for a range of learned helplessness phenomena. By this account, individuals attribute outcomes to their current situation or state rather than to the chosen action; they generalize across available actions, with the belief that the state will determine the outcome, irrespective of their actions.

Paradoxically however, a large body of research has also supported the idea that dysphoric or depressive individuals often have *greater* causal sensitivity, an effect referred to as depressive realism (Alloy and Abramson, 1979; Martin et al., 1984; Benassi and Mahler, 1985; Ackermann and DeRubeis, 1991; Allan et al., 2007; Msetfi et al., 2012). Indeed, Alloy and Abramson (1979) found that, during a task incorporating both contingent and noncontingent outcomes non-depressed people were more likely to believe that their actions were causal of the outcome whereas depressed people did not show this illusion of control, and tended to rate their actions in this task as less causal.

These contradictory findings in depressed people might be reconciled by considering the role of competition between actions and cues for causal learning. There are two major predictors of outcomes in our environment: our own instrumental actions and situational stimuli such as Pavlovian cues. These two classes of events will compete as causes for outcomes of interest during causal learning tasks, like those described above. In such tasks, when non-contingent outcomes are provided, situational stimuli can become better predictors of those outcomes than actions. So the illusion of control could reflect a disposition to assign causal status to ones own actions over situational stimuli, even when situational stimuli are better predictors. In contrast, if actionoutcome contingency awareness is impaired, then situational stimuli should be predicted to outcompete actions for association with specific outcomes and in their attribution as causes of those outcomes. This should be anticipated to produce more accurate causal judgments of actions, consistent with depressive realism. Furthermore, the deficit in action-outcome contingency awareness will still produce learned helplessness.

An implication of this argument, derived from the distinct neural regions responsible for action-outcome vs. stimulusoutcome contingency awareness, is that pathology in depression should be restricted to those medial prefrontal cortical regions that are critical for A-O learning. Conversely, the lateral PFC regions implicated in S-O learning should be relatively intact on this view. In fact, considerable research has explored the role of mPFC in behavioral control over the effects of chronic stress (Amat et al., 2005; Maier and Watkins, 2010). Resistance to environmental stressors, and as such, resilience against feelings of helplessness, is thought to rely on inhibitory control exerted by the vmPFC over limbic structures. Without this inhibition, it is argued, stressors could cause sensitization of serotonergic neurons in the dorsal raphe, changing how the organism responds to subsequent aversive stimuli (Maier and Watkins, 2005).

Serotonin is a neuromodulator thought to play a key role in the neurochemical basis of depression, with selective serotonin reuptake inhibitors being a first-line treatment of depression. It has also been implicated in the modulation of decision processes. For instance, Doya (2002) proposed that low levels of serotonin may be associated with excessive discounting of future rewards, while others have argued that it is more specifically involved with inhibiting actions and thoughts associated with aversive outcomes (Daw et al., 2002; Dayan and Huys, 2008; Huys et al., 2012; Robinson et al., 2012). This view proposes that serotonin reductions enhance punishment predictions, but do not effect reward predictions. This raises another interesting line of research– whether individuals with depression are perhaps better at learning associations with negative rather than positive consequences (see Eshel and Roiser, 2010, for a review). Numerous studies have demonstrated that depressed individuals exhibit hypersensitivity to negative feedback (Elliott et al., 1997), and hyposensitivity to positive feedback (Pizzagalli et al., 2008), and highlight how aberrance in evaluation, and subsequent allocation of attention, has detrimental effects on contingency learning.

The emerging field of computational psychiatry has provided a promising new avenue for understanding psychiatric illnesses, through applying mathematical models to behavioral and biological problems. Within decision neuroscience, it aims to provide a systematic explanation of the core processes in decision-making in a manner consistent with neurobiologically relevant processes (Dayan and Huys, 2008). A series of studies have recently used this approach in discerning the specific decision-making deficits at play in depression. In this approach, reward sensitivity is related to valuation, while learning rate represents a dimension of contingency awareness. Chase et al. (2010) found a reduced learning rate in depression, however they did note that learning rate was more closely related to severity of anhedonia than diagnosis *per se*. A recent metaanalysis in un-medicated depression reported that reduced reward sensitivity (reduced prediction errors) had greater affect than learning rate on overall learning performance, and was correlated with anhedonia severity (Huys et al., 2013). This is supported by reduced striatal activation during reward receipt (Pizzagalli et al., 2009; Smoski et al., 2009). Using a medicated sample, however, we found that learning rate was reduced in depression, which may indicate that while overall choice behavior remains impaired, antidepressant medication may change the dynamics of the contributing processes (Griffiths et al., unpublished data).

#### **Structural and resting-state abnormalities in goal-directed circuitry**

The difficulties depressed individuals have with learning and performance of goal-directed action correspond with abnormalities in learning and choice related brain regions. Gray matter volumetric studies and postmortem examinations have show neuronal size reductions relative to controls in the OFC (Cotter et al., 2005; Drevets and Price, 2008), left ACC (Drevets et al., 1997; Coryell et al., 2005), dlPFC (Drevets, 2004), caudate and NAc (Baumann et al., 1999). Moreover, symptoms of anhedonia, depression severity and probability of suicide have all been associated with reduced caudate volume (Pizzagalli et al., 2008) and caudate activity (Forbes et al., 2009).

There is a complex relationship between depression severity and the OFC. Some studies report increased OFC activity in treatment responsive depressives, whereas more severely ill patients have relatively normal or decreased OFC metabolism (Drevets et al., 1997; Mayberg, 1997). Drevets et al. (1997) posit that increased OFC activity may reflect a cognitive compensatory effort to attenuate negative emotion, while reduced OFC activity may reflect a primary pathology related to monoamine dysfunction. This is supported by enhanced dextroamphetamineinduced rewarding effects compared to controls (Tremblay et al., 2002, 2005). Functional imaging during a range of tasks involving planning, reward, behavioral choice and feedback have reported abnormal recruitment of the mOFC (Elliott et al., 1998; Taylor Tavares et al., 2008), and lesions of the human OFC have been argued to increase the risk for developing depression (Drevets, 2007), although this is controversial (see e.g., Carson et al., 2000). Nevertheless, reports that this region plays a key role in valuation suggest that any compromised function will likely affect goaldirected action.

In addition to problems with the core circuitry associate with goal-directed action, imaging studies have shown abnormally low dlPFC activity during resting state (Galynker et al., 1998), yet overly activated activation during working memory and cognitive control tasks (Harvey et al., 2005; Wagner et al., 2006), potentially indicating inefficiency in this cognitive control region. This may contribute to the increased indecisiveness experienced in depression.

#### **Depression summary**

In summary, depression is characterized by impairments in reinforcement learning, and using affective information to guide behavior. Anhedonia, a common symptom in depression, maps closely onto deficits within outcome valuation circuitry, and is the clearest example of how problems with reward value lead to reductions in goal-directed action. Learned helplessness, or a lack of resistance to environmental stressors, may also occur when S-O associations outcompete A-O associations. This may cause depressed individuals to generalize action-outcome contingencies across different contexts, and become less adaptive to new environments.

#### **OTHER DISORDERS**

It is clear that an associative learning framework can provide testable hypotheses and explanations for a range of deficits in clinical disorders. Though we can only provide a brief discussion of three such disorders here, the potential exists for many others. For instance, Obsessive-Compulsive disorder, where behavior may exhibit an overreliance on habits due to dysfunctional goaldirected circuitry (Gillan et al., 2011), and anorexia nervosa, where there is a tendency to deprive oneself of food, despite, or likely because of, hyperactivity in evaluative neural circuitry during food presentation (Keating et al., 2012), provide interesting examples.

Importantly, assessment of decision-making deficits need not be constrained rigidly by diagnostic classifications. Most psychiatry research uses these classifications with the assumption that it will provide a homogenous subset of participants. However multiple systems may be differentially affected in these patients, and comorbidities and group averaging may contaminate both behavioral and neural results. Further, symptom commonalities also occur across diagnostic boundaries, for instance anhedonia, which can occur in a range of disorders, such as depression, posttraumatic stress disorder and schizophrenia. Thus, behavioral tests that probe specific processes and neural deficits could have great value in guiding research on biologically-based individualized classification.

It is worth mentioning that the wide-ranging use of medications and substance use in psychiatric groups makes testing these populations to clearly delineating the source of their illness very challenging. Most medications affect multiple, predominantly monoamine, neurotransmitter systems, and variance in functional effects occurs over different doses. These neurotransmitter systems are intricately involved in reward and decision processes, thus it can be difficult to distinguish disorder-related findings from those induced by medication, and to untangle the differential effects of medications across tasks. For instance, using SPECT, Paquet et al. (2004) found a correlation between procedural learning ability and D2 receptor occupancy. Patients on second generation antipsychotics (SGA) perform better at procedural learning tasks compared to those on first generation antipsychotics (FGA), which is thought to be due to the comparatively lower affinity for striatal D2 receptors in SGAs (Stevens et al., 2002; Scherer et al., 2004). Conversely, Beninger et al. (2003) found that SGAs adversely affected performance on the IGT, which they surmise may be due to the high affinity of SGAs for serotonin receptors in the PFC.

#### **CONCLUSIONS**

Though much progress has been made in elucidating the processes and neurobiology of decision-making, a great deal remains to be done. Contradictory findings and interpretations persist, and with contributions from diverse fields such as economics, computer science and psychology, a "common language" has not yet been achieved. Decision-making is an extremely complex process, and as such, the range of tasks used to assess this skill is broad. Great care must be taken when comparing results across tasks, as task-related variables may modulate the underlying circuitry involved.

A key strength of associative learning tasks is the strong theoretical basis, and the broad foundation of animal research that has helped develop our knowledge of the circuitry underlying specific learning processes. By establishing links between well-defined psychological processes (e.g., goal-directed action), neural circuits and even intracellular signaling, we can develop a biologically-based phenotype of psychopathology, grounded in translatable behavioral tests. Nevertheless, important questions remain regarding how we conceptualize the interaction between these learning systems. For instance, a flat architecture assumes that goal-directed and habitual processes exist in parallel, with an arbitrator determining which system is utilized for the following action. A hierarchical structure, however, proposes a global goal-directed system that incorporates habitual action sequences when they can achieve the desired goal. Although beyond the scope of this review, there are a number of neural and computational theories that debate how and where action values are compared and transformed into motor signals, and if in fact, cognitive action selection and motor planning occur as serial or simultaneous processes (Cisek and Kalaska, 2010; Hare et al., 2011; Cisek, 2012; Rushworth et al., 2012; Wunderlich et al., 2012; Dezfouli and Balleine, 2013). These theories are important considerations for determining precisely how fundamental processes such as outcome valuation and contingency learning are transformed into the motor choices producing goal-directed performance.

Decision neuroscience is an exciting field that incorporates translational research from a range of species and scientific techniques. Within this field, associative learning accounts have provided a theoretical basis for the development of a range of biologically relevant behavioral paradigms. This framework endeavors to draws together behavioral and neurological processes, creating impetus for a wide range of testable hypotheses. Through systematic application of biologically relevant paradigms, we could further identify specific problems contributing to maladaptive decision-making across psychiatric disorders. This review has attempted to highlight how a number of deficits across psychiatric disorders may be explained in terms of fundamental reward learning and performance impairments, which could shed some new light on the functional impairment and neurobiological underpinnings of these illnesses.

#### **ACKNOWLEDGMENTS**

The preparation of this manuscript was supported by a grant from the Australian Research Council (ARC FL0992409) to Bernard W. Balleine.

#### **REFERENCES**


and schizophrenia: diagnostic specificity and prognostic implications. *Am. J. Psychiatry* 162, 1706–1712. doi: 10.1176/appi.ajp.162.9.1706


second-order conditioning and reinforcer devaluation effects. *J. Neurosci.* 16, 5256–5265.


frontal gyrus and striatum in first-episode schizophrenia. *Schizophr. Res.* 145, 1–10. doi: 10.1016/j.schres.2012.11.028


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 18 February 2014; paper pending published: 01 April 2014; accepted: 09 May 2014; published online: 26 May 2014*.

*Citation: Griffiths KR, Morris RW and Balleine BW (2014) Translational studies of goal-directed action as a framework for classifying deficits across psychiatric disorders. Front. Syst. Neurosci. 8:101. doi: 10.3389/fnsys.2014.00101*

*This article was submitted to the journal Frontiers in Systems Neuroscience*.

*Copyright © 2014 Griffiths, Morris and Balleine. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.