Aversive Counterconditioning Attenuates Reward Signaling in the Ventral Striatum

Introduction

Appetitive conditioning is the process by which cues (CS) become associated with reward (US) and subsequently acquire incentive salience (reward motivation) themselves CR (Everitt and Robbins, 2005). Once these CRs are acquired they are difficult to extinguish, especially for highly salient reward such as food and drugs (Myers and Carlezon, 2010). For example, even after an extended period of abstinence, cue-induced reinstatement can occur after a single re-exposure to the drug itself, an associated stimulus or, environmental stress (Shaham et al., 2003). It is therefore important to gain insight into the mechanisms that underlie reward reinstatement and to find procedures to prevent reinstatement.

One of the possibilities to prevent reward reinstatement could be counterconditioning. This is the process of pairing of a CS+ with an US that has a valence opposite to the valence of the original US (e.g., unpleasant shock experience versus pleasant drug experience; Pearce and Dickinson, 1975). Several studies have shown that counterconditioning can reduce fear CRs (e.g., Paunović, 2003; Raes and De Raedt, 2012) and appetitive sexual responses (e.g., Tanner, 1974). However, studies that used counterconditioning to change appetitive CRs are scarce. An early, non-experimental study suggested that the administration of electrical shocks to addicted individuals while they relived past drug using experiences can result in a low relapse rate during the 2-years follow-up (Copeman, 1976). Despite this early positive finding of aversive counterconditioning on relapse in drug addiction, similar effects have only recently been reported in experimental studies in which it was demonstrated that aversive counterconditioning is more effective than extinction in reducing reward motivation in humans (Van Gucht et al., 2010) and rats (Tunstall et al., 2012). However, two important questions remain to be answered: (1) Does aversive counterconditioning also modulate reward reinstatement, and (2) what are the neurobiological mechanisms involved in aversive counterconditioning? This knowledge may help to develop aversive counterconditioning strategies to prevent relapse in addiction.

The mesocorticolimbic pathway, which connects the ventral tegmental area to the striatum, amygdala, hippocampus, ventral- and dorsal medial prefrontal cortex plays an essential role in both reward and fear learning (Everitt and Robbins, 2005; Abraham et al., 2014). A recent animal study demonstrated that the ability to learn the association between a cue and a negative outcome was retarded if that stimulus was previously paired with a rewarding outcome. This effect was associated with heightened neural activity within the thalamus, insula, amygdala, and ventral striatum (Nasser and McNally, 2012). To our knowledge, there is currently only one human study that reported on the neural correlates of counterconditioning, using a paradigm in which stimuli that previously predicted an aversive outcome, were paired with a rewarding outcome. Using this paradigm it was demonstrated that counterconditioning of conditioned fear responses was associated with reduced amygdala and hippocampus activation following reinstatement (Bulganin et al., 2014). Previous studies demonstrated that the prospect of pain reduces reward signaling within the ventral striatum (Talmi et al., 2009), and that the processing of positive and negative reward interact on a neural level (Park et al., 2011). However, there are no functional magnetic resonance imaging (fMRI) studies that assessed the neural correlates of aversive counterconditioning aiming to reduce (the reinstatement of) reward CRs. While based on these previous study it could be expected that aversive stimuli can be used to alter appetitive CRs, there are several indications that appetitive CRs are more difficult to alter than aversive CRs: for instance, while exposure treatments for anxiety disorders are highly successful in the extinction of fear CRs, the same exposure treatments are fairly unsuccessful in reducing drug-CRs and thus the treatment of substance use disorder (Kaplan et al., 2011). The current study therefore aims to investigate the neural mechanisms related to aversive counterconditioning of conditioned reward responses, specifically focussing on the effects of counterconditioning on reward reinstatement.

Given the increasing evidence of mesocorticolimbic interactions between aversive and appetitive learning, we hypothesized that aversive counterconditioning would reduce reward signaling following reinstatement within the mesocorticolimbic pathway. To test this hypothesis we developed an fMRI task that consisted of four phases: first, in the conditioning phase, two different stimuli were reinforced with a monetary reward if the participant hit the target on time. Second, in the counterconditioning phase, one of the stimuli was paired with an aversive electrical shock to the wrist. Third, in the extinction phase, none of the stimuli were reinforced neither by reward nor by shock. Fourth, in the reinstatement phase, reward was reinstated by informing the participants that the monetary gain could be doubled. It was expected that reward reinstatement in the ventral striatum would be prevented only for stimuli that were counterconditioned, but not for stimuli that were not counterconditioned. Because the ventral striatum is connected with several cortical and limbic regions, we also applied a functional connectivity analysis with the ventral striatum as seed region, as this may further clarify the underlying mechanisms involved in counterconditioning.

Materials and Methods

Participants

Twenty-five healthy volunteers participated in this study (mean age 22.0 ± 2.1 SD, 12 women). Participants were recruited at the University of Amsterdam. All participants had normal vision and reported taking no medication affecting the nervous system, including no illicit drugs. None of the participants used nicotine or more than 21 units of alcohol per week. Participants were reimbursed for their time (€15) and won an additional €10 in the counterconditioning task. The study was approved by the ethics committee of the University of Amsterdam and all participants gave written consent to participate in the study.

Experimental Paradigm

The counterconditioning task was based on the monetary incentive delay task (Knutson et al., 2000; Ossewaarde et al., 2011; Bulganin et al., 2014) and consisted of four phases: a conditioning phase, a counterconditioning phase, an extinction phase and a reinstatement phase (see Figure 1). During each phase the same stimuli (blue, yellow, and purple squares; colors were counterbalanced across subjects) were shown: the counterconditioned CS+_cc predicted a monetary reward during the conditioning phase but an aversive electrical shock to the wrist during the counterconditioning phase and was unreinforced during the other phases. The non-counterconditioned CS+_nc, predicted a monetary reward during the conditioning phase, but was unreinforced during the other phases. The CS- was never reinforced. The monetary reward consisted of an image of €0,50. Participants were instructed that they could win an amount up to €10. The intensity of the electrical shock was individually adjusted, to be unpleasant but not painful. In the conditioning, extinction, and reinstatement phase each stimulus was presented 20 times. In the counterconditioning phase, each stimulus was presented three times. Across all phases, each trial started with a cue (the CS+_cc, the CS+_nc, or the CS-) presented for 3000–6000 ms, followed by a target. If subjects responded to the target on time, the trial was reinforced with a monetary gain. The initial target duration was 270 ms. With every hit, the target duration was reduced by 18 ms, with every miss the target duration was increased by 36 ms. In this way the target duration was manipulated to ensure that approximately 50% of the trials were ‘hit’ trials and thus reinforced resulting in 10 reinforced CS+_cc and 10 reinforced CS+_nc trials. Thus, the CS was reinforced in 50% of the trials. As a result, subjects would always gain a total of €10. Although, the CS- trials are never reinforced, the target duration in these trials is manipulated in the same way as the target duration for the CS+ trials. Feedback followed target offset immediately, and was displayed for 1000 ms The inter-stimulus interval varied between 1000 and 3000 ms. Before task onset, subjects were instructed which stimuli predicted a reward but that they had to respond to the target in all trials (including the non-rewarding trials). Doing so ensures that differences in neural responsiveness cannot be due to differences in motor preparation. Also, subjects were informed that they could receive an electrical shock, but were not instructed about contingency. Normally, in fear conditioning studies, fear is reinstated by simply presenting the US (the electrical shock) without the CS (Bulganin et al., 2014). In the current study, however, the US was the visual presentation of €0,50, which they would later be received. We assumed that simply displaying the visual stimulus of €0,50 to reinstate reward conditioning, which would have been in line with previous fear conditioning studies, would not be sufficient to reinstate reward anticipation. Therefore, we aimed to reinstate the reward CR by presenting the following text to the participants: “You have already won €10. If you continue like this and keep responding on time, you can double your gain.” However, none of the post-reinstatement trials were actually reinforced. After the task, CS–US contingency awareness and CS valence were assessed using a visual analog scale.

FIGURE 1

FIGURE 1. (A) Trial sequence for conditioning phase. 50% of the CS+ trials were reinforced with a monetary gain. Each stimulus is presented 20 times. (B) Trial sequence for counterconditioning phase. The CS+_cc stimuli are reinforced with an electrical shock, the other stimuli are not reinforced. Each stimulus is presented three times. (C) Trial sequence for extinction phase. None of the stimuli is reinforced. Each stimulus is presented 20 times. (D) Reward is reinstated by showing this text. (E) The trial sequence for the reinstatement phase is similar to the trial sequence of the extinction phase. Each stimulus is presented 20 times.

This experimental paradigm was developed to study the effects of counterconditioning on reward reinstatement, but we wanted to prevent the relationship between the CS+cc and the shock to be too strong. Therefore the counterconditioning phase had a limited number of trials and as a consequence this phase could not be included in the fMRI and behavioral analysis.

Valence and Contingency Ratings

Contingency awareness and valence for each CS was assessed using a visual analog scale ranging from 0 to 10: “How pleasant/unpleasant did you find this square?” and “How likely was it that this square was followed by a shock.” Subjects were classified as being unaware of the stimulus-shock contingency if they indicated that the change of getting a shock after seeing the CS- was larger than zero and/or when they indicated that the change of getting a shock after seeing either CS+ stimulus was equal to zero.

Functional Magnetic Resonance Imaging Data Acquisition and Analysis

Images were acquired on a 3.0-T Achieva full-body scanner (Philips Medical Systems, Best, the Netherlands) using a 32 channel SENSE head coil. Echo planar images (EPIs) were taken covering the whole brain, with a total of 37 ascending axial slices (3 mm × 3 mm × 3 mm voxel size; slice gap 3 mm; TR/TE 2000 ms/28 ms; matrix 80 × 80). A T1-3D high resolution anatomical scan (TR/TE 8.2/3.7; matrix 240 × 187; 1 × 1 × 1 voxel; transverse slices) was acquired for spatial normalization purposes. Imaging data were analyzed using SPM8¹. Preprocessing included realignment, slice-time correction, co-registration of the structural and functional scans, normalization to MNI-space based on the segmented structural scan and smoothing with a kernel of 8 mm full-width at half maximum. First-level models included separate regressors for the CS+_cc, CS+_nc and CS-, which were modeled separately for the four different phases. These regressors were convolved with the canonical hemodynamic response function with a duration of 0 s. Six realignment parameters were included as regressors of no interest. A high pass filter (1/128 Hz) was included in the first level model to correct for low frequency signal drift. The contrast images were entered into a second-level full-factorial design.

To assess whether the paradigm resulted in a significant reward CR, we tested the main effect of stimulus (CS+_cc and CS+_nc versus CS-) during the conditioning phase. Subsequently we tested the effect of counterconditioning on extinction and reinstatement by testing the main effect of stimulus type (CS+_cc versus CS+_nc) during extinction and reinstatement. To assess whether counterconditioning resulted in a reduction of reward signaling, we tested a phase (conditioning versus reinstatement) by stimulus type (CS+_cc versus CS+_nc) interaction effect.

In order to assess whether differences in ventral striatal responsiveness were also associated with differences in ventral striatal functional connectivity, we used generalized psycho-physiological interaction analysis (gPPI; McLaren et al., 2012) with the ventral striatum as a seed region to compare functional connectivity between the counterconditioned and non-counterconditioned CS+ conditions after reinstatement. This type of analysis allows to investigate changes in functional connectivity with a seed region (in this case the ventral striatum) related to a certain psychological variable (in this case the presentation of either a counterconditioned or non-counterconditioned stimulus). The time series of the first eigenvariate of the BOLD signal were temporally filtered, mean corrected, and deconvolved to generate the time series of the neuronal signal for the ventral striatum (which was defined as the nucleus accumbens from the Harvard–Oxford subcortical structure probability atlas) for each individual subject. The interaction term – PPI – was computed by multiplying the time series from the psychological regressors with this physiological variable.

All voxel-wise statistical tests are family wise error rate corrected for multiple comparisons (p < 0.05). Whole brain analysis were corrected on the cluster level, using an initial height threshold on voxel level of p < 0.01. Because of the a priori hypothesis on the role of the ventral striatum in appetitive conditioning, small-volume corrections (SVCs) were applied using the same ventral striatal mask that was used in the PPI analyses (Worsley et al., 1996). The anatomical names of all significant clusters were identified using the Automatic Anatomated Labeling atlas toolbox in SPM8 (Tzourio-Mazoyer et al., 2002). To explore a possible correlation between brain activation and behavior, the individual change in valence ratings of the CS stimuli were entered as a covariate in the whole brain analyses.

Behavioral Data Analysis

The effects of counterconditioning on reaction time was assessed in a repeated measurements (rm) ANOVA with a test for a stimulus (CS+_cc, CS+_nc, CS-) by phase (conditioning, extinction, reinstatement) interaction effect using SPSSv20 (Statistical Package for the Social Sciences). A one way ANOVA was used to test for differences in valence ratings for each CS type.

Results

Contingency and Valence Ratings

Nine participants (34.6%) were unaware of the contingency between the stimulus and the electrical shock. As these participants did not learn the association between the cue that predicted the electrical shock and the electrical shock itself, they were excluded from further analysis. The final sample therefore consisted of 16 participants (mean age 22.1 ± 2.0 SD, nine women). The rmANOVA showed that there was a significant difference in CS-valence ratings (F_2,30 = 4.0, p = 0.03). Follow-up paired sampled t-test indicated that there was no significant difference in valence rating between the CS+_cc and CS-, whereas, the valence of CS+_nc was rated significantly higher than the valence of the CS+_cc (t₁₅ = 3.1, p = 0.007) and CS- (t₁₅ = 2.6, p = 0.02). In other words, these results indicate that subjects ‘liked’ the stimulus that predicted a monetary gain in the conditioning phase better than the stimulus that was additionally counterconditioned with aversive shocks.

Reaction Times

Repeated measurements ANOVA’s revealed no stimulus by task phase interaction effect (F_6,10 = 0.41, p = 0.86) and no main effect of task phase (F_3,13 = 1.34, p = 0.31). However, there was a main effect of stimulus type (F_2,14 = 19.67, p < 0.001) across all phases,. Follow-up tests revealed that there were no significant differences between the CS+_cc and CS+_nc reaction times (F_1,15 = 1.73, p = 0.29). However, the reaction times to the CS- were significantly longer than the reaction times the other two stimuli (F_1,15 = 40.72, p < 0.001): CS- = 280.51 (mean) ± 7.91 (standard error), CS+_cc = 262.22 ± 8.72 and CS+_nc = 267.22 ± 8.72. These results indicate that counterconditioning did not significantly affect the reaction time to the reward predicting cues.

Functional Magnetic Resonance Imaging

All results described below are family wise error rate corrected for multiple comparisons (p < 0.05). The exact statistics can be found in Table 1.

TABLE 1

TABLE 1. Main effects of appetitive conditioning and effect of aversive counterconditioning on reward-reinstatement.

During conditioning the CS+ (counterconditioned and non-counterconditioned) versus CS- contrast showed significant activation of the ventral striatum, caudate nucleus and bilateral motor cortex and a significant deactivation of the superior, middle, and inferior prefrontal cortex. These results indicate that the task indeed elicited significant activation of the reward network. In the conditioning phase there were no significant difference in neural activation between the counterconditioned and non-counterconditioned CS+.

During the extinction phase, the counterconditioned CS+ elicited more activation than the non-counterconditioned stimulus in the cerebellum and lingual gyrus, but not the ventral striatum. Importantly, during the reinstatement phase, however, the non-counterconditioned CS+ elicited more activation than the counterconditioned CS+ in the ventral striatum (small-volume corrected; Figure 2A) as well as in the bilateral cerebellum and regions within the parietal cortex. There was no significant interaction effect between stimulus type (CS+_cc and CS+_nc) and phase (conditioning versus extinction). However, there was a significant interaction effect between stimulus type (CS+_cc and CS+_nc) and phase (conditioning versus reinstatement) within the ventral striatum indicating that, compared to the conditioning phase, the ventral striatum response to the counterconditioned CS+ significantly reduced after reinstatement, whereas the ventral striatum response to the non-counterconditioned CS+ did not change (Figure 2B). These results suggest that aversive counterconditioning reduces reward signaling in ventral striatum following reinstatement. An exploratory regression analysis with the change in valence rating as covariate, did not reveal any significant brain-behavior correlations.

FIGURE 2

FIGURE 2. The neural response to counterconditioned versus non-counterconditioned CS+. The figures on the left (A) show the neural responses after reinstatement to the counterconditioned versus the non-counterconditioned CS+stimuli in the ventral striatum. The bar graphs on the right (B) are a visual presentation of the response to the counterconditioned CS+ (CS+_cc; black), non-counterconditioned CS+ (CS+_nc; gray) and CS- (white) at the peak voxel (MNI: 12 10 -6, z = 2.91). Compared to the conditioning phase, ventral striatal BOLD response during reinstatement is significantly decreased for the counterconditioned CS+, but not for the other stimuli. During reinstatement, the ventral striatal BOLD response for the non-counter CS is significantly higher compared to the BOLD response for the counterconditioned stimulus. The figures are displayed at p < 0.001 uncorrected for visualization purpose. ^∗ = whole brain significant stimulus type (CS+_cc versus CS+_nc) by phase (conditioning versus reinstatement) interaction effect.

Functional Connectivity

All results described below are were family wise error rate corrected for multiple comparisons (p < 0.05). The exact statistics can be found in Table 2.

TABLE 2

TABLE 2. Differences in ventral striatal connectivity following reward reinstatement.

To further explore the effects of aversive counterconditioning on functional connectivity of the ventral striatum after reinstatement, we conducted a psychophysiological interaction analysis with the right ventral striatum as seed region. This analysis compared functional connectivity of the ventral striatum following reinstatement during counterconditioned and non-counterconditioned CS+ conditions and showed that activity within the right ventral striatum following counterconditioned CS+ was more strongly coupled to activation in the left hippocampus, insula, and rolandic operculum, the right pallidum, thalamus and precuneus and the bilateral cerebellum than following non-counterconditioned CS+ (Figure 3; Table 2). Together with the fMRI analysis, these results indicate that counterconditioning increases functional connectivity to the ventral striatum, whereas it reduces neural responsiveness of the ventral striatum, following reward reinstatement.

FIGURE 3

FIGURE 3. Functional connectivity of the ventral striatum. The figure shows significant differences in functional connectivity for the counterconditioned versus non-counterconditioned CS+ following reinstatement. Counterconditioning strengthened the functional connectivity of the ventral striatum seed region (left) with the left insula (A) and left hippocampus (B). The figures are displayed at p < 0.001 uncorrected for visualization purpose. The graph (C) is a visual representation of the functional connectivity changes with the ventral striatum in response to counterconditioned and non-counterconditioned CS+, in the insula and hippocampus. ^∗ = whole brain significant main effect of stimulus type (CS+_cc versus CS+_nc) on ventral striatal connectivity following reward reinstatement.

Discussion

The tendency of conditioned reward reinstatement, even after an extended period of abstinence, is one of the hallmarks of addiction (Shaham et al., 2003). The aim of this study was to gain a better insight in the neural mechanisms involved in conditioned reward reinstatement and to test whether aversive counterconditioning could attenuate the reinstatement of reward. The results demonstrate that aversive counterconditioning indeed reduced reward signaling within the ventral striatum following reward reinstatement. Dopaminergic projections from the ventral tegmental area to the ventral striatum and prefrontal cortex are considered to play a key role in the processing of reward and it is well-established that dopaminergic neurons within the ventral striatum increase their firing in response to CS that predict a reward (Schultz, 2007; Volman et al., 2013; Abraham et al., 2014). Thus, reduced reinstatement related BOLD response within the ventral striatum following aversive counterconditioning may reflect reduced dopaminergic firing and thus reduced reward anticipation/expectation. This is confirmed by the finding that, on a behavioral level, the valence of the counterconditioned stimuli was scored significantly lower compared to the non-counterconditioned stimuli. Thus previously rewarding stimuli that were paired with an aversive outcome were perceived as significantly less pleasant compared to previously rewarding stimuli that were never paired with an aversive outcome.

In addition to reduced ventral striatal responsiveness after reward reinstatement, we found that aversive counter-conditioning also strengthened ventral striatal connectivity with the insula, hippocampus, thalamus, rolandic operculum and cerebellum. The insula is a key region in interoceptive processing (Craig, 2009) and may therefore provide body-relevant information about the coding for aversiveness during reward related processing (Paulus and Stewart, 2014). For example, cues that predict an aversive outcome have been shown to elicit strong activation of the insula as well as the hippocampus, thalamus, rolandic operculum, and cerebellum (Sehlmeyer et al., 2009). The hippocampus is involved in the retrieval of aversive memory (Milad et al., 2007) and reward reinstatement (Fuchs et al., 2005; Rogers and See, 2007). Both the insula (Haber and Knutson, 2010; Sesack and Grace, 2010; Cauda et al., 2011) and the hippocampus (Baudonnat et al., 2013) are anatomically and functionally connected to the ventral striatum. Given this connection it has been suggested that these regions interact dynamically during reward processing and decision making (Baudonnat et al., 2013; Paulus and Stewart, 2014). Similarly, increased insula-hippocampus functional connectivity has been demonstrated during the consolidation of fear (Feng et al., 2013). Therefore, the current finding of increased striatal connectivity with the insula and hippocampus after aversive counterconditioning is likely to reflect enhanced retrieval of the aversive (bodily) state associated with the electrical shock, thereby disrupting the retrieval of the appetitive reward memory previously associated with the CS+, resulting in reduced reward signaling within the ventral striatum following reward reinstatement.

Previous studies have demonstrated that aversive counterconditioning is more effective than extinction in reducing appetitive CRs (e.g., cue-induced chocolate craving; Van Gucht et al., 2010) and several other studies observed an interaction between aversive and appetitive processes in associative-learning and decision making in the mesocorticolimbic pathway (Hu et al., 2013; Bulganin et al., 2014; Engelmann et al., 2015). Our results contribute to these previous findings by demonstrating that aversive counterconditioning attenuates reward signaling within the ventral striatum following reward reinstatement, possibly by enhancing striatal connectivity with the hippocampus and insula.

The results of our study also support the hypothesis that aversive counterconditioning could be a successful method for changing appetitive CRs in addiction (Copeman, 1976; Rossi et al., 1988; Van Gucht et al., 2010, 2013). However, addiction is generally characterized by compulsive drug seeking, defined as persisting in the face of adverse consequences (Vanderschuren and Everitt, 2004; Robbins et al., 2008). In line with this, addicted individuals typically show diminished behavioral and neural sensitivity to monetary punishment (Bechara and Damasio, 2002; Wrase et al., 2007; Bjork et al., 2008; Beck et al., 2009). This suggests that aversive counterconditioning may be an effective method to modify appetitive CRs in healthy individuals. However, even in healthy individuals it has been demonstrated that addicted patients may actually be insensitive to aversive counterconditioning. However, to our knowledge, this is the first fMRI study to report on findings of an experimental paradigm that enables the investigation of the effect of a negative consequence other than a monetary loss on reward anticipation. Using this paradigm to investigate aversive-appetitive interactions not only in healthy but also addicted individuals may thus allow for the development of better pharmacological or behavioral treatments for substance dependence (Abraham et al., 2014).

It should be noted, however, that while we aimed to extinguish the reward CR, the ventral striatum did not show a differential effect already during the extinction phase, immediately after aversive counterconditioning. Visual inspection of Figure 2 may even suggest that ventral striatal activation during extinction was increased compared to the conditioning phase, although this effect was non-significant. A possible explanation for these initial increases in ventral striatum activation is that both anticipation of an aversive and anticipation of a rewarding outcome induces dopaminergic excitation within the ventral striatum (Volman et al., 2013; Abraham et al., 2014). Nonetheless, compared to the conditioning phase, ventral striatum activation after the entire extinction phase and following reinstatement was significantly reduced for the counterconditioned but not the non-counterconditioned stimulus. Because the current paradigm did not seem to induce extinction of the reward CR to the non-counterconditioned stimulus, the psychological mechanism underlying the attenuation of the ventral striatal response remains unclear. The reduction of its response after reinstatement due to competitive positive and negative associations is consistent with the ventral striatal role in value coding. However, the initial non-selective increase during extinction appears more consistent with salience signaling. Yet another interpretation is that counterconditioning attenuates the reinstatement of reward, though the ventral striatal response to the non-counterconditioned stimulus did not decrease over time. Future studies should aim to disentangle these interpretations, possibly by altering stimulus value by manipulating the rewarding and aversive reinforcers, as well as the number of extinction trials to obtain full extinction prior to reinstatement. Regardless of these possible interpretations, this study demonstrates that reward signaling in the ventral striatum can be attenuated using aversive counterconditioning.

An important limitation of this study is that a relatively large proportion of the recruited subjects (34.6%) remained unaware of the CS–US contingency and was therefore excluded from the analysis. This large percentage of subjects unaware of the CS–US contingency could be explained by the relatively short counterconditioning phase (each CS and the US was only presented three times) and future studies may consider using a longer counterconditioning phase. Nevertheless, even this short counterconditioning phase showed to be effective in two-thirds of the participants. Impaired CS–US learning might even form a risk for impaired decision making, risky behavior or even addiction, mediated by an abnormal interaction between aversive and appetitive processes, which could be explored in future studies. A second limitation is that our paradigm did not include a behavioral measure of reward reinstatement as all subjects were instructed to keep responding to the target, even though the trials were not rewarded. As such, the only behavioral measure of reward reinstatement is a difference in valence rating for the counterconditioned and non-counterconditioned stimulus. Future studies should therefore include a more detailed measure of reward reinstatement in order to investigate the behavioral effect of counterconditioning on reward reinstatement. Third, it could be argued that the changes in ventral striatal activation or connectivity are related more to motor preparation instead of reward motivation. Similar to other studies (Knutson et al., 2000), we have however, instructed the participants to respond to the target, irrespective of whether or not they thought they could gain a reward in the trial. Doing so we have aimed to minimalize the possible confounding effect of motor preparation on ventral striatal activation and connectivity, although a motor-preparation effect cannot be fully ruled out.

Conclusion

These results show that aversive counterconditioning prevents reward-signaling of the ventral striatum following reinstatement in healthy volunteers, thereby providing important evidence of how aversive counterconditioning could reduce appetitive CRs.

Author Contributions

Data were obtained by AK and RS. AK, RS, and GvW analyzed the data. The first draft was prepared by AK. RS, GvW, JH, LR, and WvdB actively participated in writing and revising the manuscript for publication.

Funding

This study was made possible by a grant provided by ZonMW (grant number 91211002).

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Abbreviations

CR, conditioned response; CS, conditioned stimulus; CS+_cc, counterconditioned stimulus; CS+_nc, non-counterconditioned stimulus; US, unconditioned stimulus.

Footnotes

1. ^http://www.fil.ion.ucl.ac.uk/spm

References

Abraham, A. D., Neve, K. A., and Lattal, K. M. (2014). Dopamine and extinction: a convergence of theory with fear and reward circuitry. Neurobiol. Learn. Mem. 108, 65–77. doi: 10.1016/j.nlm.2013.11.007

CrossRef Full Text | Google Scholar

Baudonnat, M., Huber, A., David, V., and Walton, M. E. (2013). Heads for learning, tails for memory: reward, reinforcement and a role of dopamine in determining behavioral relevance across multiple timescales. Front. Neurosci. 7:175. doi: 10.3389/fnins.2013.00175

CrossRef Full Text | Google Scholar

Bechara, A., and Damasio, H. (2002). Decision-making and addiction (part II): myopia for the future or hypersensitivity to reward? Neuropsychologia 40, 1690–1705. doi: 10.1016/S0028-3932(02)00016-7

CrossRef Full Text | Google Scholar

Beck, A., Schlagenhauf, F., Wüstenberg, T., Hein, J., Kienast, T., Kahnt, T., et al. (2009). Ventral striatal activation during reward anticipation correlates with impulsivity in alcoholics. Biol. Psychiatry 66, 734–742. doi: 10.1016/j.biopsych.2009.04.035

CrossRef Full Text | Google Scholar

Bjork, J. M., Momenan, R., Smith, A. R., and Hommer, D. W. (2008). Reduced posterior mesofrontal cortex activation by risky rewards in substance-dependent patients. Drug Alcohol Depend. 95, 115–128. doi: 10.1016/j.drugalcdep.2007.12.014

CrossRef Full Text | Google Scholar

Bulganin, L., Bach, D. R., and Wittmann, B. C. (2014). Prior fear conditioning and reward learning interact in fear and reward networks. Front. Behav. Neurosci. 8:67. doi: 10.3389/fnbeh.2014.00067

CrossRef Full Text | Google Scholar

Cauda, F., Cavanna, A. E., D’agata, F., Sacco, K., Duca, S., and Geminiani, G. C. (2011). Functional connectivity and coactivation of the nucleus accumbens: a combined functional connectivity and structure-based meta-analysis. J. Cogn. Neurosci. 23, 2864–2877. doi: 10.1162/jocn.2011.21624

CrossRef Full Text | Google Scholar

Copeman, C. D. (1976). Drug addiction: an aversive counterconditioning technique for treatment. Psychol. Rep. 38, 1271–1281. doi: 10.2466/pr0.1976.38.3c.1271

CrossRef Full Text | Google Scholar

Craig, A. D. B. (2009). How do you feel–now? The anterior insula and human awareness. Nat. Rev. Neurosci. 10, 59–70. doi: 10.1038/nrn2555

CrossRef Full Text | Google Scholar

Engelmann, J. B., Meyer, F., Fehr, E., and Ruff, C. C. (2015). Anticipatory anxiety disrupts neural valuation during risky choice. J. Neurosci. 35, 3085–3099. doi: 10.1523/JNEUROSCI.2880-14.2015

CrossRef Full Text | Google Scholar

Everitt, B. J., and Robbins, T. W. (2005). Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci. 8, 1481–1489. doi: 10.1038/nn1579

CrossRef Full Text | Google Scholar

Feng, P., Feng, T., Chen, Z., and Lei, X. (2013). Memory consolidation of fear conditioning: bi-stable amygdala connectivity with dorsal anterior cingulate and medial prefrontal cortex. Soc. Cogn. Affect. Neurosci. 9, 1730–1737. doi: 10.1093/scan/nst170

CrossRef Full Text | Google Scholar

Fuchs, R. A., Evans, K. A., Ledford, C. C., Parker, M. P., Case, J. M., Mehta, R. H., et al. (2005). The role of the dorsomedial prefrontal cortex, basolateral amygdala, and dorsal hippocampus in contextual reinstatement of cocaine seeking in rats. Neuropsychopharmacology 30, 296–309. doi: 10.1038/sj.npp.1300579

CrossRef Full Text | Google Scholar

Haber, S. N., and Knutson, B. (2010). The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35, 4–26. doi: 10.1038/npp.2009.129

CrossRef Full Text | Google Scholar

Hu, K., Padmala, S., and Pessoa, L. (2013). Interactions between reward and threat during visual processing. Neuropsychologia 51, 1763–1772. doi: 10.1016/j.neuropsychologia.2013.05.025

CrossRef Full Text | Google Scholar

Kaplan, G. B., Heinrichs, S. C., and Carey, R. J. (2011). Treatment of addiction and anxiety using extinction approaches: neural mechanisms and their treatment implications. Pharmacol. Biochem. Behav. 97, 619–625. doi: 10.1016/j.pbb.2010.08.004

CrossRef Full Text | Google Scholar

Knutson, B., Westdorp, A., Kaiser, E., and Hommer, D. (2000). FMRI visualization of brain activity during a monetary incentive delay task. Neuroimage 12, 20–27. doi: 10.1006/nimg.2000.0593

CrossRef Full Text | Google Scholar

McLaren, D. G., Ries, M. L., Xu, G., and Johnson, S. C. (2012). A generalized form of context-dependent psychophysiological interactions (gPPI): a comparison to standard approaches. Neuroimage 61, 1277–1286. doi: 10.1016/j.neuroimage.2012.03.068

CrossRef Full Text | Google Scholar

Milad, M. R., Wright, C. I., Orr, S. P., Pitman, R. K., Quirk, G. J., and Rauch, S. L. (2007). Recall of fear extinction in humans activates the ventromedial prefrontal cortex and hippocampus in concert. Biol. Psychiatry 62, 446–454. doi: 10.1016/j.biopsych.2006.10.011

CrossRef Full Text | Google Scholar

Myers, K. M., and Carlezon, W. A. (2010). Neuroscience and biobehavioral reviews extinction of drug- and withdrawal-paired cues in animal models?: relevance to the treatment of addiction. Neurosci. Biobehav. Rev. 35, 285–302. doi: 10.1016/j.neubiorev.2010.01.011

CrossRef Full Text | Google Scholar

Nasser, H. M., and McNally, G. P. (2012). Appetitive-aversive interactions in Pavlovian fear conditioning. Behav. Neurosci. 126, 404–422. doi: 10.1037/a0028341

CrossRef Full Text | Google Scholar

Ossewaarde, L., Qin, S., Van Marle, H. J. F., van Wingen, G. A., Fernández, G., and Hermans, E. J. (2011). Stress-induced reduction in reward-related prefrontal cortex function. Neuroimage 55, 345–352. doi: 10.1016/j.neuroimage.2010.11.068

CrossRef Full Text | Google Scholar

Park, S. Q., Kahnt, T., Rieskamp, J., and Heekeren, H. R. (2011). Neurobiology of value integration: when value impacts valuation. J. Neurosci. 31, 9307–9314. doi: 10.1523/JNEUROSCI.4973-10.2011

CrossRef Full Text | Google Scholar

Paulus, M. P., and Stewart, J. L. (2014). Interoception and drug addiction. Neuropharmacology 76, 342–350. doi: 10.1016/j.neuropharm.2013.07.002

CrossRef Full Text | Google Scholar

Paunović, N. (2003). Prolonged exposure counterconditioning as a treatment for chronic posttraumatic stress disorder. J. Anxiety Disord. 17, 479–499. doi: 10.1016/S0887-6185(02)00233-5

CrossRef Full Text | Google Scholar

Pearce, J. M., and Dickinson, A. (1975). Pavlovian counterconditioning: changing the suppressive properties of shock by association with food. J. Exp. Psychol. Anim. Behav. Process. 1, 170–177. doi: 10.1037/0097-7403.1.2.170

CrossRef Full Text | Google Scholar

Raes, A. K., and De Raedt, R. (2012). The effect of counterconditioning on evaluative responses and harm expectancy in a fear conditioning paradigm. Behav. Ther. 43, 757–767. doi: 10.1016/j.beth.2012.03.012

CrossRef Full Text | Google Scholar

Robbins, T. W., Ersche, K. D., and Everitt, B. J. (2008). Drug addiction and the memory systems of the brain. Ann. N. Y. Acad. Sci. 1141, 1–21. doi: 10.1196/annals.1441.020

CrossRef Full Text | Google Scholar

Rogers, J. L., and See, R. E. (2007). Selective inactivation of the ventral hippocampus attenuates cue-induced and cocaine-primed reinstatement of drug-seeking in rats. Neurobiol. Learn. Mem. 87, 688–692. doi: 10.1016/j.nlm.2007.01.003

CrossRef Full Text | Google Scholar

Rossi, J. S., Prochaska, J. O., and DiClemente, C. C. (1988). Processes of change in heavy and light smokers. J. Subst. Abuse 1, 1–9. doi: 10.1016/S0899-3289(88)80003-8

CrossRef Full Text | Google Scholar

Schultz, W. (2007). Behavioral dopamine signals. Trends Neurosci. 30, 203–210. doi: 10.1016/j.tins.2007.03.007

CrossRef Full Text | Google Scholar

Sehlmeyer, C., Schöning, S., Zwitserlood, P., Pfleiderer, B., Kircher, T., Arolt, V., et al. (2009). Human fear conditioning and extinction in neuroimaging: a systematic review. PLoS ONE 4:e5865. doi: 10.1371/journal.pone.0005865

CrossRef Full Text | Google Scholar

Sesack, S. R., and Grace, A. A. (2010). Cortico-basal ganglia reward network: microcircuitry. Neuropsychopharmacology 35, 27–47. doi: 10.1038/npp.2009.93

CrossRef Full Text | Google Scholar

Shaham, Y., Shalev, U., Lu, L., De Wit, H., and Stewart, J. (2003). The reinstatement model of drug relapse: history, methodology and major findings. Psychopharmacology (Berl) 168, 3–20. doi: 10.1007/s00213-002-1224-x

CrossRef Full Text | Google Scholar

Talmi, D., Dayan, P., Kiebel, S. J., Frith, C. D., and Dolan, R. J. (2009). How humans integrate the prospects of pain and reward during choice. J. Neurosci. 29, 14617–14626. doi: 10.1523/JNEUROSCI.2026-09.2009

CrossRef Full Text | Google Scholar

Tanner, B. A. (1974). A comparison of automated aversive conditioning and a waiting list control in the modification of homosexual behavior in males. Behav. Ther. 5, 29–32. doi: 10.1016/S0005-7894(74)80083-3

CrossRef Full Text | Google Scholar

Tunstall, B. J., Verendeev, A., and Kearns, D. N. (2012). A comparison of therapies for the treatment of drug cues: counterconditioning vs. extinction in male rats. Exp. Clin. Psychopharmacol. 20, 447–453. doi: 10.1037/a0030593

CrossRef Full Text | Google Scholar

Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N. et al. (2002). Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15, 273–289. doi: 10.1006/nimg.2001.0978

CrossRef Full Text | Google Scholar

Van Gucht, D., Baeyens, F., Hermans, D., and Beckers, T. (2013). The inertia of conditioned craving. Does context modulate the effect of counterconditioning? Appetite 65, 51–57. doi: 10.1016/j.appet.2013.01.019

CrossRef Full Text | Google Scholar

Van Gucht, D., Baeyens, F., Vansteenwegen, D., Hermans, D., and Beckers, T. (2010). Counterconditioning reduces cue-induced craving and actual cue-elicited consumption. Emotion 10, 688–695. doi: 10.1037/a0019463

CrossRef Full Text | Google Scholar

Vanderschuren, L. J. M. J., and Everitt, B. J. (2004). Drug seeking becomes compulsive after prolonged cocaine self-administration. Science 305, 1017–1019. doi: 10.1126/science.1098975

CrossRef Full Text | Google Scholar

Volman, S. F., Lammel, S., Margolis, E. B., Kim, Y., Richard, J. M., Roitman, M. F., et al. (2013). New insights into the specificity and plasticity of reward and aversion encoding in the mesolimbic system. J. Neurosci. 33, 17569–17576. doi: 10.1523/JNEUROSCI.3250-13.2013

CrossRef Full Text | Google Scholar

Worsley, K. J., Marrett, S., Neelin, P., Vandal, A. C., Friston, K. J., and Evans, A. C. (1996). A unified statistical approach for determining significant signals in images of cerebral activation. Hum. Brain Mapp. 4, 58–73. doi: 10.1002/(SICI)1097-0193(1996)4:1

CrossRef Full Text | Google Scholar

Wrase, J., Schlagenhauf, F., Kienast, T., Wüstenberg, T., Bermpohl, F., Kahnt, T., et al. (2007). Dysfunction of reward processing correlates with alcohol craving in detoxified alcoholics. Neuroimage 35, 787–794. doi: 10.1016/j.neuroimage.2006.11.043

CrossRef Full Text | Google Scholar