Conditioned reinforcement can be mediated by either outcome-specific or general affective representations

Introduction

Conditioned reinforcers are Pavlovian cues that support the acquisition and maintenance of new instrumental responses. Responding on the basis of conditioned rather than primary reinforcement is a pervasive part of modern life and even plays an important role in neuropsychiatric diseases such as drug addiction. For example, conditioned reinforcers include things such as money and corporate icons, which seem to have a value of their own, as well as items with more specific associations, such as the song that was playing when we met that special someone. Furthermore, relapse to drug-seeking after treatment often involves exposure to drug-associated cues. Yet we have a remarkably limited understanding of what underlying associative information is triggered by these cues to guide responding. Specifically, it is not certain whether conditioned reinforcers are effective because they evoke representations of specific outcomes or because they trigger general affective states that are independent of any specific outcome.

This question has important implications for how different brain regions or circuits might be involved in conditioned reinforcement. We know that conditioned reinforcement depends upon the amygdala, the orbitofrontal cortex, and the nucleus accumbens (Burns et al., 1993 ; Cador et al., 1989 ; Everitt and Robbins, 1992 ; Parkinson et al., 2001 ; Parkinson et al., 1999 ; Pears et al., 2003 ; Whitelaw et al., 1996 ). Although all of these areas signal information about past associations between cues and primary rewarding outcomes, the precise informational content of that signaling differs between regions.

The basolateral amygdala is perhaps most strongly implicated in signaling information about the outcome that is predicted by a cue. This role is most obvious in reinforcer devaluation tasks (Hatfield et al., 1996 ; Malkova et al., 1997 ). Rats and monkeys with amygdala lesions – especially basolateral amygdala – fail to modify conditioned responding as a result of reinforcer devaluation. These deficits demonstrate a critical role for basolateral amygdala in the process by which neutral cues are able to evoke representations of the outcomes they predict, particularly the value of those outcomes. Notably this function depends critically upon interactions with orbitofrontal cortex (Baxter et al., 2000 ; Gallagher et al., 1999 ; Gottfried et al., 2003 ; Izquierdo et al., 2004 ; Schoenbaum et al., 2003 ).

However, the role of the basolateral amygdala in Pavlovian learning is also evident in other settings, where behavior is not directly dependent on the value of the predicted outcome. For example, basolateral amygdala lesions abolish second-order conditioning in which a neutral cue is paired with a conditioned stimulus (Hatfield et al., 1996 ; Setlow et al., 2002a ). Similarly, basolateral amygdala has been implicated in Pavlovian-to-instrumental transfer (Corbit and Balleine, 2005 ). In this procedure, a cue that has been paired with an appetitive outcome, through Pavlovian conditioning, is able to increase a previously trained instrumental response. Normal performance in these tasks is not affected by devaluation of the primary reward (Holland, 2004 ; Holland and Rescorla, 1975 ). Instead, it has been suggested that performance depends on the ability of the cue to evoke general affective or motivational representations, which are independent of any specific outcome. The role of basolateral amygdala in mediating this function seems to involve projections to the central nucleus and to the nucleus accumbens (Balleine and Corbit, 2005 ; Corbit and Balleine, 2005 ; de Borchgrave et al., 2002 ; Hall et al., 2001 ; Holland and Gallagher, 2003 ; Setlow et al., 2002b ).

Basolateral amygdala and associated downstream areas in orbitofrontal cortex, central nucleus, and nucleus accumbens may support conditioned reinforcement because of their differential roles in Pavlovian learning, as outlined above. To demonstrate this, it is necessary to develop conditioned reinforcement procedures that can dissociate the influences of different types of associative information. Here, we have taken a first step in this direction by using specialized Pavlovian training procedures, transreinforcer blocking (Rescorla et al., 1999 ) and reinforcer devaluation (Holland and Rescorla, 1975 ), to test whether conditioned reinforcers guide responding either by directly evoking representations of outcome-specific, devaluation-sensitive information, or by triggering more general, devaluation-insensitive affect representations. Rats underwent Pavlovian conditioning using these procedures, thereby creating cues that were biased to evoke either outcome or affective information. Subsequently these cues, along with normally conditioned control cues, were presented contingent on instrumental responding. We found that intact rats responded for either the outcome or the affect cue to the same extent as for a normally conditioned cue. These results demonstrate that conditioned reinforcers can guide responding through either type of associative information. Interestingly, conditioned reinforcement was completely abolished in rats with basolateral amygdala lesions. Consistent with the extant literature, this result suggests a general role for basolateral amygdala in conditioned reinforcement. The implications of these data, combined with recent reports from our laboratory of a more specialized role of orbitofrontal cortex, will be discussed.

Materials and Methods

Subjects

Forty male Long-Evans rats (Charles River Laboratories) weighing between 275 and 300 g upon arrival were housed individually and placed on a 12 hour light/dark schedule. All rats were given ad libitum access to food except during testing periods. During behavioral testing, rats were food deprived to 85% of their baseline weight. All testing was conducted during the light period of their cycle.

Surgical procedures

Basolateral amygdala lesions were made in stereotaxic surgeries using intra-cerebral infusions of N-methyl-D-aspartic acid (NMDA, Sigma, St. Louis, MO) in saline vehicle (n = 12). Infusions were delivered at a rate of 0.1 µl/30 second of NMDA (12.5 µg/µl) and were made in two sites in each hemisphere: 0.1–0.2 µl volume at 2.8 mm anterior to bregma, at +/−5.0 mm lateral to the midline, and at a depth of 8.1 and 8.4 mm ventral from the skull surface. Controls received saline vehicle infusions with the same coordinates (n = 12). Following a 1 week recovery period, all rats were then placed on food restriction. Testing began 2 weeks after surgery.

Apparatus

All testing was conducted in eight standard sized behavioral boxes and other equipment modules purchased from Coulbourn Instruments (Allentown, PA). A recessed food cup was located in the right wall of the chamber approximately 2 cm above the floor, connected to a feeder mounted outside of the chamber to deliver differently flavored sucrose pellets, termed 01 or 02 (banana, chocolate, or grape, Research Diets, New Brunswick, NJ). The training boxes were also configured to allow delivery of four different cues, including a house light and cue light, which were located on the right wall of the chamber approximately 10 cm from the floor, and a white noise and tone (75 dB), from speakers mounted in the center of the left and right walls. Data were collected by computer using Graphic State behavioral software from Coulbourn Instruments.

Pavlovian conditioning

The order of training is illustrated in Figure 1 . All rats underwent Pavlovian conditioning for 2–3 weeks prior to conditioned reinforcement testing (15–22 days). Unless otherwise stated, the identities of the cues were counterbalanced, presented for 30 second and with average inter-trial intervals of 2.5 minute.

FIGURE 1

Figure 1. Outline of behavioral training. Rats underwent simple Pavlovian conditioning followed by either devaluation (top of figure-data in Sections Conditioned Reinforcement Mediated by Devaluation-Insensitive Representations of General Affect and Role of Amygdala in Conditioned Reinforcement; note five rats out of the group received blocking then devaluation, indicated by gray dotted line) or by transreinforcer blocking (bottom of figure data in Section Conditioned Reinforcement Mediated by Devaluation-Sensitive Representations of Specific Outcomes). Subsequently, Pavlovian cues from this training were delivered contingent on instrumental responding to test the associative basis of conditioned reinforcement. To test whether conditioned reinforcement could be mediated by devaluation-insensitive affective representations, we compared instrumental responding for B versus a control cue (CS− or X) in the devalued and non-devalued rats. To test whether conditioned reinforcement could be mediated by devaluation-sensitive outcome-specific information, we compared instrumental responding for A versus X and Y versus X. Unless noted in the methods, the outcomes, cues and responses were counterbalanced.

Devaluation of a primary reinforcer

For the experiments described in Section Conditioned Reinforcement Mediated by Devaluation-Insensitive Representations of General Affect (n = 11) and Role of Amygdala in Conditioned Reinforcement (n = 24), training consisted of simple Pavlovian conditioning followed by reinforcer devaluation. In the first week, sessions consisted of 16 presentations of B paired with delivery of flavored sucrose pellets (three pellets). Thereafter, a CS− was introduced, paired with no reward. Each cue was then presented 4–16 times per session as necessary to obtain differential responding. (Note for five rats initially trained in a standard blocking procedure, as illustrated by the gray arrow in Figure 1 , a blocked cue served as the CS−.) Subsequently, these rats were divided into two groups with similar conditioned responding, then rats in one group – the devalued group – were allowed to access to the flavored sucrose pellets that had been paired with B in their home cage for 10 minute. Immediately following this, the rats were injected with LiCl (0.3 M, 5 mg/kg, i.p.). Rats in the other group – the non-devalued group – were given LiCl injections on the same days as the devalued group but received access to the pellets on alternate days. Sessions continued until the devalued rats stopped eating the devalued food pellet (2–3 sessions).

Transreinforcer blocking

For the experiments described in Section Conditioned Reinforcement Mediated by Devaluation-Sensitive Representations of Specific Outcomes (n = 5), training consisted of Pavlovian conditioning followed by transreinforcer blocking. Initial conditioning consisted of sessions in which two visual cues, A or B, were presented 16 times each, paired with flavored sucrose pellets (three pellets), either banana and grape or banana and chocolate. Pellets were designated 01 and 02 and counterbalanced. Importantly these sucrose pellets were distinct yet equally preferred (see Figure 2 ). Following conditioning, all rats received one day of pre-exposure to two auditory cues, X and Y, followed by four sessions of compound conditioning in which each visual cue was presented simultaneously with one auditory cue eight times. Both compound cues, AX and BY, were paired with delivery of 01. 01 was the same flavored pellet already predicted by A, so learning for X was blocked. By contrast, 01 was a differently flavored but similarly preferred pellet to that predicted by B. As a result, Y was blocked from acquiring associations with the general affective properties of 01 while still becoming associated with its unique sensory-specific properties.

FIGURE 2

Figure 2. Taste preference testing for banana (B) versus grape (G) or chocolate (C)-flavored sucrose pellets. Food-deprived rats were tested in preference tests in which rats were given banana and chocolate or banana and grape flavored sucrose pellets in their home cage for 5 minute over 3 days. There was no difference in consumption across these preference tests (baseline, F(1,95) = 0.017, p = 0.89). On subsequent days, rats were given access to one of the flavors (banana, grape, or chocolate) for 20 minute in an unlimited quantity, then they were presented with the satiated and another non-satiated flavor together. Even though the different flavors were of equal value, the rats preferred the new flavor after satiation (selective satiation, F(1,47) = 41.1, p < 0.0001) (*, p < 0.05).

Conditioned reinforcement

Rats were placed into the same behavioral chambers used in conditioning. The food cup was removed, and two instrumental responses (levers or chains) were available. Responding on one lever or chain (termed R+) resulted in a 1 second presentation of a CS+. The CS+ was a fully conditioned cue, either A (data in Section Conditioned Reinforcement Mediated by Devaluation-Sensitive Representations of Specific Outcomes), or B (data in Sections Conditioned Reinforcement Mediated by Devaluation-Insensitive Representations of General Affect and Role of Amygdala in Conditioned Reinforcement) or the partially conditioned cue, Y (data in Section Conditioned Reinforcement Mediated by Devaluation-Sensitive Representations of Specific Outcomes). Responding on the other lever or chain (termed R−) resulted in a 1 second presentation of a control cue. The control cue was either a CS− that had never been paired with reward or X, the cue that had been presented in compound with A in the blocking procedure. Cues were presented on VR2 schedules. The instrumental responses were counterbalanced with respect to location and pairing with a CS+ or CS−. Sessions lasted 30 minute and continued for two or 3 days.

To test whether conditioned reinforcement could be mediated by devaluation-insensitive affective representations, we compared instrumental responding for B versus the control cue in devalued versus non-devalued rats. To test whether conditioned reinforcement could be mediated by devaluation-sensitive outcome-specific information, we compared instrumental responding for A versus X to instrumental responding for Y versus X in non-devalued rats.

Statistical analysis

Data on behavioral responses (food cup entries and exits, lever presses, chain pulls) were recorded by Coulbourn GS2 software and processed in Matlab. These data were analyzed by ANOVAs using Statistica software with post-hoc testing when appropriate (p < 0.05).

Results

Data outlined in the introduction suggests that different brain circuits might be involved in conditioned reinforcement due to their respective roles in reward learning. Thus, basolateral amygdala and orbitofrontal cortex may support conditioned reinforcement because they allow Pavlovian cues to evoke representations of the outcomes they predict. Similarly, projections from basolateral amygdala to central nucleus and nucleus accumbens may support conditioned reinforcement because they allow Pavlovian cues to evoke representations of general affect. However, such speculation is premature since we do not currently know whether conditioned reinforcers support behavior because of the outcomes they predict or due to some inherent value or “affect” that the cues have acquired. Intuitively one might expect it is both. However, to the best of our knowledge, with the exception of one particular study (Parkinson et al., 2005 ), this has not been empirically tested. To test this idea, it is necessary to utilize more specific Pavlovian training techniques to create cues that are biased to trigger or evoke either outcome or affect representations. These cues can then be used to assess conditioned reinforcement. Here, we will describe our initial attempt to use this approach.

Conditioned reinforcement mediated by devaluation-insensitive representations of general affect

To show that conditioned reinforcement can be mediated by devaluation-insensitive representations of general affect, we tested the ability of rats to acquire a novel instrumental response for a Pavlovian cue after devaluation of the outcome predicted by the cue. As illustrated in Figure 1 , these rats received presentations of a cue, B, paired with delivery of a flavored sucrose pellet, 02. Another cue, the control cue, was presented either without food (CS−) or for five rats (gray dotted line in Figure 1 ), simultaneously with A in a blocking procedure. After training, rats were assigned to one of two groups, such that their conditioned responding did not differ. ANOVA, analyzing the last day of conditioning, revealed no main effect of group (devalued versus non-devalued) (p < 0.05).

Rats in one group underwent reinforcer devaluation, in which the food was paired with illness induced by lithium chloride injections. Rats in the control group received similar exposure to the food and illness on alternate days. As in prior studies (Gallagher et al., 1999 ; Pic

Next, we tested the ability of these cues to support conditioned reinforcement. One response (lever or chain, counterbalanced) resulted in presentation of B, the cue that had been paired with the flavored pellet (02), which had been devalued for half the rats. Another response (chain or lever, counterbalanced) led to presentation of a control cue, the cue that had been presented without reward (CS−) or with A during blocking (cue X). As illustrated in Figure 3 , rats responded significantly more for B than for the control cue, with no significant effect of devaluation. A two-way ANOVA (devaluation × response) demonstrated a main effect of whether the response led to B or the control cue [F(1,26) = 4.515, p = 0.043), however there was no significant main effect nor any interaction with devaluation (F < 0.2501, p > 0.621). Consistent with prior results (Parkinson et al., 2005 ), these data show that devaluation-insensitive, general affective properties evoked by Pavlovian cues are sufficient to support conditioned reinforcement.

kens et al., 2003

; Schoenbaum and Setlow, 2005 ), rats in the devalued group significantly reduced their food consumption while rats in the non-devalued group showed no change. A two-way ANOVA (devaluation × trial) comparing pellet consumption revealed a main effect of devaluation as well as a devaluation × trial interaction (p's < 0.05).

FIGURE 3

Figure 3. Acquisition of a new response mediated by devaluation-insensitive representations of general affect. This graph shows the average total number of responses for cue B versus the control cue over three, 30-minute sessions on an VR2 schedule. The right side of the graph shows responding in rats for whom the 02 pellet was devalued (B-D). The left side of the graph shows responding in rats for whom the 02 pellet was not devalued (B-ND). Rats responded significantly more for B than for the control cue, and there was no effect of devaluation (*, p < 0.05; **, p < 0.10).

Conditioned reinforcement mediated by devaluation-sensitive representations of specific outcomes

To show that conditioned reinforcement can be mediated by outcome representations that are devaluation-sensitive, we used a Pavlovian training procedure termed transreinforcer blocking. This procedure minimizes the formation of an association between the cue and the general affective state normally evoked by the outcome, while allowing an association between the cue and devaluation-sensitive features of the specific outcome to form normally (Rescorla et al., 1999 )

Transreinforcer blocking requires two discriminable but equally preferred outcomes, 01 and 02. For this, we used differently flavored sucrose pellets that met these criteria (described in Figure 2 ). As illustrated in Figure 1 , two visual cues, A and B, were each paired with one of these flavored sucrose pellets (A-01 and B-02). After initial conditioning, the rats underwent transreinforcer blocking, in which these cues were presented in compound with two new cues, X and Y. AX was paired with 01, while BY was paired with 01. Because A predicted all features of 01, X was blocked from acquiring any associative representations. By contrast, B did not predict the specific sensory properties of 01 (properties that allowed the selective satiation of one outcome but not the other in Figure 2 . As a result, Y acquired an association with these unique devaluation-sensitive features of the 01 outcome, while it was blocked from acquiring associations with the general affect shared between the two outcomes (properties that led to a similar preference between the two outcomes in Figure 2 ). Accordingly when conditioned responding was assessed under extinction conditions in a probe test, rats responded most to the fully conditioned cues, A and B, somewhat for the partially conditioned cue, Y, and at levels comparable to the pre-CS period to the blocked cue, X. ANOVA (trial × cue period × cue) demonstrated a significant main effect of cue period (pre-CS vs. CS, F (1,50) = 36.262, p = 0.0001) and a significant main effect of cue (F (1,15) = 4.335, p = 0.2). Post-hoc testing revealed significant differences between responding for A versus X (F (1,15) = 8.924, p = 0.0092) and Y versus X (F (1,15) = 6.54, p = 0.02).

Next, we tested the ability of these cues to support conditioned reinforcement. In different sessions, one response (lever or chain, counterbalanced) resulted in presentation of either the fully conditioned cue, A, or the partially conditioned cue, Y. Another response (chain or lever, counterbalanced) always resulted in presentation of the blocked cue, X. As illustrated in Figure 4 , rats responded significantly more for either A or Y compared to X. A two-way ANOVA (group × response) revealed a main effect of whether the response led to the conditioned cues versus the blocked cue (F (1,14) = 24.099, p = 0.0002); however, there was no main effect nor any interaction with group (F < 0.558, p > 0.467). Thus, both A and Y supported conditioned reinforcement equally. Post-hoc tests showed significant difference between responding for A versus X (F (1,14) = 14.868, p = 0.002) and for Y versus X (F(1,14) = 10.385, p = 0.006). These data show that representations of outcome-specific information evoked by Pavlovian cues are sufficient to support conditioned reinforcement.

FIGURE 4

Figure 4. Acquisition of a new response mediated by devaluation-sensitive outcome-specific representations. This graph shows the average total number of responses over two, 30 minute sessions on a VR2 schedule. Rats, represented on the left side of the graph, had access to two instrumental responses: one leading to the fully conditioned cue, A, and the other leading to the fully blocked cue, X. Rats represented on the right-side of the graph had access to two responses: one leading to the partially conditioned outcome cue, Y, and the other leading to the fully blocked cue, X. Rats responded significantly more for the conditioned cues, A or Y, than for X. (*, p < 0.05).

Role of amygdala in conditioned reinforcement

Data described above suggest that Pavlovian cues function as conditioned reinforcers due to their ability to evoke representations of the outcomes they predict and also due to their ability to trigger the general affect normally evoked by those outcomes. Amygdala lesions – particularly selective lesions of the basolateral amygdala – have previously been shown to abolish the ability of Pavlovian cues to serve as conditioned reinforcers. To test the overall validity of our novel conditioned reinforcement procedures, rats with lesions targeted at basolateral amygdala were tested using the reinforcer devaluation and conditioned reinforcement task. Lesions encompassed the rostral and caudal basolateral amygdala complex. Note that for some rats the lesions also included rostral central nucleus (Figure 5 ).

FIGURE 5

Figure 5. The extent of amygdala lesions. Lesions encompassed >75% of the basolateral complex bilaterally. In addition, most lesions also affected rostral central nucleus of the amygdala.

The rats underwent simple conditioning followed by devaluation as illustrated in Figure 1 and described in Section Conditioned Reinforcement Mediated by Devaluation-Insensitive Representations of General Affect. Amygdala lesions had no effect on Pavlovian conditioning. ANOVA comparing conditioned responding on the final day showed neither a significant main effect nor any interaction with lesion (F < 0.537, p > 0.472). Furthermore, there were no differences in conditioned responding between rats to be placed in the devalued versus non-devalued groups (F < 0.537, p > 0.472). Devaluation resulted in a significant reduction in pellet consumption in the devalued but not the non-devalued group (p < 0.05).

Next, the rats underwent conditioned reinforcement testing, as described in Section Conditioned Reinforcement Mediated by Devaluation-Insensitive Representations of General Affect. Controls responded significantly more for B, the fully conditioned cue, than for the control cue. Differential responding increased across days of training and there was again no effect of devaluation. By contrast, lesioned rats showed no difference in responding for B and the control cue on any day of training. An analysis of the contrast in responding for B versus the control cue across three days of testing is shown in Figure 6 . ANOVA indicated a main effect of lesion (F (1,16) = 4.954, p = 0.0349) and no main effect nor any interaction with devaluation ( F < 0.661, p > 0.424). These results are consistent with previously published reports that amygdala damage causes general deficits in conditioned reinforcement (Parkinson et al., 2001 ).

FIGURE 6

Figure 6. Effects of amygdala lesions on conditioned reinforcement. Graphs show the contrast in responding for the conditioned cue B, which had been paired with 02, and the control cue. The contrast was calculated as ((R⁺ + R⁻)/(R⁺ − R⁻)).(Left) Responding in rats in which the 02 outcome was not devalued. (Right) Responding in rats in which the 02 outcome was devalued. Control rats (closed boxes) exhibited an increase in differential responding across days and there was no effect of devaluation. Lesioned rats (open boxes) exhibited no evidence of differntial responding, whether or not the outcome had been devalued.

Discussion

Despite their apparent differential involvement in different forms of associative learning, basolateral amygdala, and its various outflow pathways through orbitofrontal cortex, central nucleus, and nucleus accumbens are each critical for normal conditioned reinforcement. Thus, instrumental responding for cues previously paired with food reward is sensitive to damage to amygdala, particularly basolateral amygdala (Burns et al., 1993 ; Cador et al., 1989 ; Cousens and Otto, 2003 ; Hatfield et al., 1996 ; Parkinson et al., 2001 ; Setlow et al., 2002a ), and also to the outflow pathways described above, including orbitofrontal cortex (but not other prefrontal areas) (Cousens and Otto, 2003 ; Pears et al., 2003 ), central nucleus of the amygdala, and regions of nucleus accumbens (Parkinson et al., 1999 ; Robledo et al., 1996 ; Setlow et al., 2002b ; Taylor and Robbins, 1984 ). However, while damage to basolateral amygdala abolishes responding for these cues (Parkinson et al., 2001 ), manipulations elsewhere in these circuits have different effects. For example, in one report orbitofrontal-lesioned animals actually responded more for conditioned reinforcement compared to controls, as if their responding had become insensitive to some but not other aspects of the conditioned reinforcer (Pears et al., 2003 ). Similarly, central nucleus of the amygdala and nucleus accumbens seem to be important primarily for potentiating the control over behavior by conditioned reinforcers (Parkinson et al., 1999 ; Robledo et al., 1996 ; Taylor and Robbins, 1984 ). One interpretation of these data is that conditioned reinforcement is not a unitary process but in fact reflects parallel activation of different types of associative information, mediated by these different circuits. Here, we have presented evidence in support of this proposal by showing apparently normal conditioned reinforcement for cues that selectively evoked outcome-specific or general affective representations.

Interestingly, for orbitofrontal cortex and basolateral amygdala, there appears to be some correspondence between the role these areas play in processing associative information evoked by Pavlovian cues and their roles in conditioned reinforcement. Thus, basolateral amygdala is important in Pavlovian settings for allowing cues to evoke representations of outcomes and also the affective information with which those outcomes are associated. This global role of basolateral amygdala in Pavlovian learning potentially explains the general deficit in conditioned reinforcement observed after basolateral amygdala lesions in the current study. In other words, the basolateral amygdala may be primarily important for conditioned reinforcement because it is critical for allowing cues to become associated with these properties of the outcome, rather than because of any intrinsic role in guiding responses or actions. This proposal is consistent with observations that damage to basolateral amygdala is most effective at disrupting responding for secondary reinforcers when lesions are made before learning whereas damage to the outflow pathways – nucleus accumbens and orbitofrontal cortex – continues to be effective even when made after learning (Cousens and Otto, 2003 ; Pears et al., 2003 ; Setlow et al., 2002a ; Setlow et al., 2002b ).

One of the important outflow pathways from the amygdala, the orbitofrontal cortex, plays a more specific role in the process whereby cues evoke outcome representations in Pavlovian settings. Consistent with this, we have recently reported that orbitofrontal lesions cause a selective deficit in conditioned reinforcement mediated by these representations (Burke et al., 2007 ). In this study, rats with orbitofrontal lesions were tested for conditioned reinforcement after training in the transreinforcer task. We found that these rats showed normal responding on the lever that produced the normally conditioned cue but not on the lever that produced the partially conditioned, outcome cue. In controls, responding on this lever was completely abolished by devaluation of the primary outcome. It will be of interest in the future to determine whether conditioned reinforcement mediated by other areas linked to basolateral amygdala such as central nucleus or nucleus accumbens, involves affective information and also to identify whether different components of these circuits are differentially involved in learning versus using the cue-evoked information, as has been observed in Pavlovian settings (Cousens and Otto, 2003 ; Pickens et al., 2005 ; Pickens et al., 2003 ; Setlow et al., 2002a ; Setlow et al., 2002b ).

It is worth noting that our results parallel those from studies using a procedure termed Pavlovian-to-instrumental transfer. In this procedure, a cue that has been paired with an appetitive outcome, through Pavlovian conditioning, is able to increase performance of a previously trained instrumental response. This increased responding is termed “transfer”. Like conditioned reinforcement, transfer assesses the ability of Pavlovian cues to influence instrumental responding, and this effect has been divided into a general form, which is thought to be due to the general affective properties triggered by the cue, and a specific form, which is thought to reflect information about the specific outcome that the cue predicts.

There appears to be significant correspondence between the features and substrates of general and specific transfer and those of comparable forms of conditioned reinforcement demonstrated here and in our other report (Burke et al., 2007 ). This is particularly true for general transfer and general conditioned reinforcement, which both reflect devaluation-insensitive information and require processing in amygdala (Corbit and Balleine, 2005 ; Holland, 2004 ; Holland and Gallagher, 2003 ). Although general transfer is affected by central nucleus and not basolateral amygdala lesions, our lesions here did include parts of central nucleus in a significant number of rats. As a result, it is possible that the deficit in general conditioned reinforcement here reflects this lack of specificity.

The parallels between outcome-specific transfer and conditioned reinforcement however are less clear. Outcome-specific transfer is dependent on basolateral amygdala (Corbit and Balleine, 2005 ); presumably this is also true for outcome-specific conditioned reinforcement, since amygdala lesions completely abolished all responding for even a fully conditioned cue in the data presented here. However, outcome-specific transfer is insensitive to devaluation and is not affected by orbitofrontal lesions, at least when they are made before training (Holland, 2004 ; Ostlund and Balleine, 2007 ). This differs from conditioned reinforcement that is mediated by outcome information, which is both devaluation-sensitive and orbitofrontal-dependent. This suggests that there may be important differences between the circuits and neural substrates that mediate the ability of cues to directly control instrumental responding as conditioned reinforcers versus those that allow cues to modulate or influence established responding as assessed by transfer.

Conflict of Interest Statement

This research was conducted in the absence of any commercial or financial relationship that could be construed as a potential conflict of interest.

Acknowledgements

This work was supported by grants from the NIDA (DA015718, GS and DA021989, KB).

References

Balleine, B. W., and Corbit, L. H. (2005). Double dissociation of nucleus accumbens core and shell on the general and outcome-specific forms of pavlovian-instrumental transfer. Soc. Neurosci. Abstr. 71. 16.

Baxter, M. G., Parker, A., Lindner, C. C., Izquierdo, A. D., and Murray, E. A. (2000). Control of response selection by reinforcer value requires interaction of amygdala and orbitofrontal cortex. J. Neurosci. 20, 4311–4319.

Pubmed Abstract | Pubmed Full Text

Burke, K. A., Miller, D. N., Franz, T. M., and Schoebaum, G. (2007). Orbitofrontal cortex lesions abolish conditioned reinforcement mediated by a representation of the expected outcome. Paper presented at New York Academy of Sciences Conference, Linking Affect to Action: Critical Contributions of the Orbitofrontal Cortex.

Burns, L. H., Robbins, T. W., and Everitt, B. J. (1993). Differential effects of excitotoxic lesions of the basolateral amygdala, ventral subiculum, and medial perfrontal cortex on responding with conditioned reinforcement and locomotor activity potentiated by intra-accumbens infusions of d-amphetamine. Behav. Brain Res. 55, 167–183.

Pubmed Abstract | Pubmed Full Text

Cador, M., Robbins, T. W., and Everitt, B. J. (1989). Involvement of the amygdala in stimulus-reward associations: interactions with the ventral striatum. Neuroscience 30, 77–86.

Pubmed Abstract | Pubmed Full Text

Corbit, L. H., and Balleine, B. W. (2005). Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of pavlovian-instrumental transfer. J. Neurosci. 25, 962–970.

Pubmed Abstract | Pubmed Full Text

Cousens, G. A., and Otto, T. (2003). Neural substrates of olfactory discrimination learning with auditory secondary reinforcement. I. Contributions of the basolateral amygdaloid complex and orbitofrontal cortex. Integr. Physiol. Behav. Sci. 38, 272–294.

Pubmed Abstract | Pubmed Full Text

de Borchgrave, R., Rawlins, J. N., Dickinson, A., and Balleine, B. W. (2002). Effects of cytotoxic nucleus accumbens lesions on instrumental conditioning in rats. Exp. Brain Res. 144, 50–68.

Pubmed Abstract | Pubmed Full Text

Everitt, B. J., and Robbins, T. W. (1992). Amygdala-ventral striatal interactions and reward-related processes. In The Amygdala: Neurological Aspects of Emotion, Memory, and Mental Dysfunction, J. P. Aggleton, ed. (Oxford, John Wiley and Sons), pp. 401–429.

Gallagher, M., McMahan, R. W., and Schoenbaum, G. (1999). Orbitofrontal cortex and representation of incentive value in associative learning. J. Neurosci. 19, 6610–6614.

Pubmed Abstract | Pubmed Full Text

Gottfried, J. A., O'Doherty, J., and Dolan, R. J. (2003). Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science 301, 1104–1107.

Pubmed Abstract | Pubmed Full Text

Hall, J., Parkinson, J. A., Connor, T. M., Dickinson, A., and Everitt, B. J. (2001). Involvement of the central nucleus of the amygdala and nucleus accumbens core in mediating Pavlovian influences on instrumental behavior. Eur. J. Neurosci. 13, 1984–1992.

Pubmed Abstract | Pubmed Full Text

Hatfield, T., Han, J. S., Conley, M., Gallagher, M., and Holland, P. (1996). Neurotoxic lesions of basolateral, but not central, amygdala interfere with Pavlovian second-order conditioning and reinforcer devaluation effects. J. Neurosci. 16, 5256–5265.

Pubmed Abstract | Pubmed Full Text

Holland, P. C. (2004). Relations between Pavlovian-instrumental transfer and reinforcer devaluation. J. Exp. Psychol.: Anim. Behav. Process. 30, 104–117.

Pubmed Abstract | Pubmed Full Text

Holland, P. C., and Gallagher, M. (2003). Double dissociation of the effects of lesions of basolateral and central amygdala on conditioned stimulus-potentiated feeding and Pavlovian-instrumental transfer. Eur. J. Neurosci. 17, 1680–1694.

Pubmed Abstract | Pubmed Full Text

Holland, P. C., and Rescorla, R. A. (1975). The effects of two ways of devaluing the unconditioned stimulus after first and second-order appetitive conditioning. J. Exp. Psychol.: Anim. Behav. Process. 1, 355–363.

Pubmed Abstract | Pubmed Full Text

Izquierdo, A. D., Suda, R. K., and Murray, E. A. (2004). Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. J. Neurosci. 24, 7540–7548.

Pubmed Abstract | Pubmed Full Text

Malkova, L., Gaffan, D., and Murray, E. A. (1997). Excitotoxic lesions of the amygdala fail to produce impairment in visual learning for auditory secondary reinforcement but interfere with reinforcer devaluation effects in rhesus monkeys. J. Neurosci. 17, 6011–6020.

Pubmed Abstract | Pubmed Full Text

Ostlund, S. B., and Balleine, B. W. (2007). Orbitofrontal cortex mediates outcome encoding in Pavlovian but not instrumental learning. J. Neurosci. 27, 4819–4825.

Pubmed Abstract | Pubmed Full Text

Parkinson, J. A., Crofts, H. S., McGuigan, M., Tomic, D. L., Everitt, B. J., and Roberts, A. C. (2001). The role of the primate amygdala in conditioned reinforcement. J. Neurosci. 21, 7770–7780.

Pubmed Abstract | Pubmed Full Text

Parkinson, J. A., Olmstead, M. C., Burns, L. H., Robbins, T. W., and Everitt, B. J. (1999). Dissociation of effects of lesions of the nucleus accumbens core and shell on appetitive Pavlovian approach behavior and the potentiation of conditioned reinforcement and locomotor activity by d-amphetamine. J. Neurosci. 19, 2401–2411.

Pubmed Abstract | Pubmed Full Text

Parkinson, J. A., Roberts, A. C., Everitt, B. J., and Di Ciano, P. (2005). Acquisition of instrumental conditioned reinforcement is resistant to the devaluation of the unconditioned stimulus. Q. J. Exp. Psychol. 58, 19–30.

Pubmed Abstract | Pubmed Full Text

Pears, A., Parkinson, J. A., Hopewell, L., Everitt, B. J., and Roberts, A. C. (2003). Lesions of the orbitofrontal but not medial prefrontal cortex disrupt conditioned reinforcement in primates. J. Neurosci. 23, 11189–11201.

Pubmed Abstract | Pubmed Full Text

Pickens, C. L., Saddoris, M. P., Gallagher, M., and Holland, P. C. (2005). Orbitofrontal lesions impair use of cue-outcome associations in a devaluation task. Behav. Neurosci. 119, 317–322.

Pubmed Abstract | Pubmed Full Text

Pickens, C. L., Setlow, B., Saddoris, M. P., Gallagher, M., Holland, P. C., and Schoenbaum, G. (2003). Different roles for orbitofrontal cortex and basolateral amygdala in a reinforcer devaluation task. J. Neurosci. 23, 11078–11084.

Pubmed Abstract | Pubmed Full Text

Rescorla, R. A. (1999). Learning about qualitatively different outcomes during a blocking procedure. Anim. Learn. Behav. 27, 140–151.

Robledo, P., Robbins, T. W., and Everitt, B. J. (1996). Effects of excitotoxic lesions of hte central amygdaloid nucleus on the potentiation of reward-related stimuli by intra-accumbens amphetamine. Behav. Neurosci. 110, 981–990.

Pubmed Abstract | Pubmed Full Text

Schoenbaum, G., and Setlow, B. (2005). Cocaine makes actions insensitive to outcomes but not extinction: implications for altered orbitofrontal-amygdalar function. Cereb. Cortex 15, 1162–1169.

Pubmed Abstract | Pubmed Full Text

Schoenbaum, G., Setlow, B., Saddoris, M. P., and Gallagher, M. (2003). Encoding predicted outcome and acquired value in orbitofrontal cortex during cue sampling depends upon input from basolateral amygdala. Neuron 39, 855–867.

Pubmed Abstract | Pubmed Full Text

Setlow, B., Gallagher, M., and Holland, P. (2002a). The basolateral complex of the amygdala is necessary for acquisition but not expression of CS motivational value in appetitive Pavlovian second-order conditioning. Eur. J. Neurosci. 15, 1841–1853.

Setlow, B., Holland, P. C., and Gallagher, M. (2002b). Disconnection of the basolateral amygdala complex and nucleus accumbens impairs appetitive Pavlovian second-order conditioned responses. Behav. Neurosci. 116, 267–275.

Taylor, J. R., and Robbins, T. W. (1984). Enhanced behavioral control by conditioned reinforcers following microinjections of d-amphetamine into the nucleus accumbens. Psychopharmacology 84, 405–412.

Pubmed Abstract | Pubmed Full Text

Whitelaw, R. B., Markou, A., Robbins, T. W., and Everitt, B. J. (1996). Excitotoxic lesions of the basolateral amygdala impair the acquisition of cocaine-seeking behaviour under a second-order schedule of reinforcement. Psychopharmacology (Berl.) 127, 213–224.

Pubmed Abstract | Pubmed Full Text