Morphological elucidation of basal ganglia circuits contributing reward prediction

Electrophysiological studies in monkeys have shown that dopaminergic neurons respond to the reward prediction error. In addition, striatal neurons alter their responsiveness to cortical or thalamic inputs in response to the dopamine signal, via the mechanism of dopamine-regulated synaptic plasticity. These findings have led to the hypothesis that the striatum exhibits synaptic plasticity under the influence of the reward prediction error and conduct reinforcement learning throughout the basal ganglia circuits. The reinforcement learning model is useful; however, the mechanism by which such a process emerges in the basal ganglia needs to be anatomically explained. The actor–critic model has been previously proposed and extended by the existence of role sharing within the striatum, focusing on the striosome/matrix compartments. However, this hypothesis has been difficult to confirm morphologically, partly because of the complex structure of the striosome/matrix compartments. Here, we review recent morphological studies that elucidate the input/output organization of the striatal compartments.

Introduction

Reinforcement learning mechanisms have been recently proposed to be based in circuits of the basal ganglia, assuming that the dopamine nigrostriatal projection acts as a reinforcement signal pathway (Sutton, 1988; Schultz et al., 1997, 1998; Sutton and Barto, 1995; Bayer and Glimcher, 2005; Cohen et al., 2012; Hart et al., 2014). Further, Reynolds et al. (2001) have reported that synaptic potentiation in striatal neurons receiving dopamine projections depends on the input from the cerebral cortex and on the dopaminergic input from the substantia nigra. These reports have led to the hypothesis that synaptic plasticity in the striatum is under the influence of the reward prediction error, and that the striatum conducts reinforcement learning throughout the basal ganglia circuits (Barto, 1995; Montague et al., 1996; Doya, 1999, 2000a,b; Crittenden and Graybiel, 2011; Takahashi et al., 2011).

In the actor–critic models, the actor chooses actions according to some policy of behavior, and the critic offers immediate feedback that notifies the actor whether the selected action was good or bad for obtaining rewards in the long run (Barto et al., 1983; for review, see Takahashi et al., 2011). Houk et al. (1995) proposed the existence of role sharing within the striatum in reinforcement learning, focusing on the striosome/matrix compartments. According to the model, the matrix performs action selection through the basal ganglia output nuclei [the internal segment of globus pallidus (GPi)/substantia nigra pars reticulata (SNr)] (actor), whereas the striosomes perform reward prediction (critic). The projection that targets dopaminergic neurons calculates the reward prediction errors, and the actor–critic learning is processed by the dopaminergic projections to the striatum.

However, this hypothesis has been difficult to test, partly because of the complex structure of the striosome/matrix compartments in the striatum. In particular, because this structure is highly irregular and cannot be visualized without processing, such as immunostaining, identification of the exact input and output pathways is difficult. We recently elucidated the input/output organization of the striosome/matrix structure using single neuron tracing by a viral tracer with a membrane translocation signal and immunohistochemistry for vesicular glutamate transporters. In this manuscript, we review recent progress in understanding the anatomical basis of basal ganglia networks in terms of reinforcement learning models, particularly the actor-critic model.

Striatal Mosaic Organization and Actor–Critic Model

Striosome/Matrix Compartments

Neurons in the striatum, unlike those in the cerebral cortex and cerebellum, do not form a layered or columnar structure. Although they appear to be randomly distributed, they are actually scattered in two embryologically different compartments called striosomes (often referred to as “patches” in rodents) and the matrix. The striosome compartment is embryologically older and the “dopamine island,” observed only during development, corresponds to patch/striosome. The matrix develops later and eventually accounts for approximately 85% of the entire striatum (Johnston et al., 1990; Nakamura et al., 2009). The matrix compartment is densely stained with acetylcholinesterase, and calbindin and somatostatin are expressed at relatively high levels (Graybiel and Ragsdale, 1978; Gerfen and Young, 1988; Gerfen, 1992). The striosome compartment is rich in μ-opioid receptors (Delfs et al., 1994; Mansour et al., 1994, 1995; Minami et al., 1994; Arvidsson et al., 1995; Kaneko et al., 1995; Ding et al., 1996; Nakamura et al., 2009).

Cortical Input and Striosome/Matrix Structure

The striatum receives glutamatergic inputs from the cerebral cortex and thalamus (Smith and Bolam, 1990), dopaminergic inputs from the substantia nigra pars compacta (SNc), and serotonergic and noradrenergic inputs from the raphe and locus coeruleus of the brain stem. The cortical input arises from almost all areas of the cerebral cortex with a local responsiveness. For example, sensory and motor cortical areas project to the part of the putamen posterior to the anterior commissure, whereas inputs from the frontal–parietal–temporal association area project to the putamen anterior from the anterior commissure and the major part of the caudate nucleus. The limbic cortex projects to the caudate nucleus, to the anterior ventral part of the putamen, and to the nucleus accumbens (Albin et al., 1989; Alexander and Crutcher, 1990).

The cortical input to the striosomes primarily arises from the limbic cortex, specifically from the orbitofrontal cortex and insula. However, it is generally considered that the input to the matrix in the rat (Gerfen, 1984, 1989; Donoghue and Herkenham, 1986), cat (Malach and Graybiel, 1986; Ragsdale and Graybiel, 1991), and monkey (Flaherty and Graybiel, 1994) arises from a wide area of the neocortex, including the motor cortex, somatosensory area, and parietal lobe. A more distinct characteristic is the layer structure of the cerebral cortex. In rats, it has been reported that cortical layers III and Va project to the matrix and layers Vb and VI project to the striosomes and that subregions of the striatum, having both striosome and matrix compartments, are innervated by the related cortical regions (Kincaid and Wilson, 1996). Thus, the striosome and matrix compartments receive the “specific” but “related” information from the cortex to contribute to their putative roles as “actor” and “critic,” respectively. Striatal interneurons with dendrites that cross compartmental borders may be a key for sorting and integrating the corticostriatal projections.

Thalamic Input and Striosome/Matrix Structure

Projections from the thalamus primarily arise from the intralaminar nuclei (particularly the centromedian and parafascicular nucleus), from motor relay nuclei (the anterior ventral and ventral lateral nucleus), and from the posterior thalamus (the posterior lateral nucleus and pulvinar). The role of thalamostriatal projections in learning, particularly in the learning and memory of movement has been recently verified (Kimura et al., 2004; Kato et al., 2011).

With respect to the striatal mosaic organization, we have reported that there is approximately three times as much thalamic input to the matrix as to the striosomes (Figure 1; Fujiyama et al., 2006), using immunohistochemistry of vesicular glutamate transporters (Fujiyama et al., 2001, 2004; Kaneko and Fujiyama, 2002; Kaneko et al., 2002). The intralaminar thalamic nuclei are thought to be the major source of thalamostriatal terminals. Axons from the caudal part of the intralaminar nuclei chiefly project to the matrix compartment in monkeys (Sadikot et al., 1990, 1992), cats (Beckstead, 1984; Ragsdale and Graybiel, 1991), and rats (Herkenham and Pert, 1981; Gerfen, 1984, 1985, 1989, 2004; Deschênes et al., 1996). Conversely, the midline thalamic nuclei, including the paraventricular and rhomboid nuclei, mainly project to the neostriatal striosome compartment and ventral striatum in cat (Ragsdale and Graybiel, 1991). These results indicate that the striosome and matrix compartments not only receive different densities of thalamic inputs but also admit inputs from different thalamic nuclei. The midline nuclei receive inputs from limbic regions (Cornwall and Phillipson, 1988). Limbic afferents through the midline nuclei as well as those from limbic cortical areas are used in striosome compartment to calculate the “state value,” which is possibly a key variable in the process of reinforcement learning (for review, see Doya, 2000a,b; Doya et al., 2002). Thus, the distinct networks for the matrix and striosome compartments involve thalamic and cortical afferents.

FIGURE 1

Figure 1. Cortical–thalamic input and striosome–matrix structure (from Fujiyama et al., 2006 with modifications). (A) Double immunofluorescence images for MOR and either VGluT1 or VGluT2 in the neostriatum. Intense MOR-positive patch compartment corresponded to areas that were weak in VGluT2 immunoreactivity (D–F). In contrast, VGluT1 immunoreactivity distributed evenly in the neostriatum (A–C). (B) Immunoelectron micrographs for MOR and either VGluT1 or VGluT2 in the neostriatum. MOR, mu-opioid receptor; VGLUT, vesicular glutamate transporter.

Differences exist in the synaptic organization of thalamostriatal neurons between the striosome and matrix compartments (Fujiyama et al., 2006; Raju et al., 2006). Our quantitative analysis of ultrastructural images revealed that in striosomes, 84% of thalamostriatal synapses were made on dendritic spines, whereas in the matrix compartment, 70% were made on dendritic shafts. Contrastingly, corticostriatal terminals preferentially synapsed onto dendritic spines (~80%) in both compartments (Figure 1; Fujiyama et al., 2006). Furthermore, thalamostriatal axospinous synapses in striosomes were larger than corticostriatal axospinous synapses in either compartments (Fujiyama et al., 2006). Excitatory axospinous synapses, including corticostriatal axospinous synapses, often display a high degree of synaptic plasticity (Calabresi et al., 2000). Moreover, dendritic spines are known to rapidly and frequently change their form, presumably reflecting their plastic characteristics (Yuste and Bonhoeffer, 2001). These findings suggest that the thalamostriatal synapses on dendritic shafts in the matrix, contributing to the “Actor,” are less plastic than those on dendritic spines in the striosome compartment, contributing the “Critic” and also less plastic than corticostriatal axospinous synapses.

Nigrostriatal Dopaminergic Input

Beside the excitatory glutamatergic inputs from the cortex and thalamus to the striatum, there are other important striatopetal projections, such as dopaminergic inputs from the SNc. The dopaminergic neurons in the midbrain are known to respond to the reward prediction error (Schultz et al., 1993, 1997, 1998; Schultz, 2007a,b). As described above, dopamine likely modulates synaptic plasticity between the corticostriatal afferents and striatal projection neurons (Calabresi et al., 2000, 2007; Reynolds et al., 2001; Surmeier et al., 2007; Shen et al., 2008).

Motivational value and motivational salience signals of dopaminergic neurons are distributed in an anatomical gradient across the substantia nigra and ventral tegmental area (VTA) (Bromberg-Martin et al., 2010). Anatomically, dopaminergic neurons in SNc are divided into the calbindin-positive dorsal tier and calbindin-negative ventral tier. Conventionally, the nigrostriatal projection in the rat brain has been reported to have the organization such that dopaminergic neurons in dorsal SNc chiefly project to the matrix, whereas those in ventral SNc mainly project to the striosomes (Gerfen et al., 1987). A similar segregation of nigrostriatal projections to striosomes and matrix compartments has been reported in cats and primates (Jimenez-Castellanos and Graybiel, 1987; Langer and Graybiel, 1989). However, this segregated organization was only partly supported by the results from our single neuron tracing study. We found that all single dopaminergic neurons innervated both striosome and matrix compartments, although projections from dorsal SNc neurons favored the matrix compartment and those from ventral SNc neurons favored the striosome compartment (Matsuda et al., 2009). Single dopaminergic neurons in the dorsal and ventral SNc innervated both striosome and matrix compartments is important, suggesting that identical temporal difference (TD) signals are simultaneously sent to a large number of striosome and matrix neurons.

However, how dopaminergic nigrostriatal projection processes specific reward-related learning remains unknown. One possibility is that phasically released dopamine modifies excitatory synapses. Its principal action will thus be at those cortical and thalamic synapses that are “active,” aiding the “selection” of striatal neurons to be fired (see Bolam et al., 2006; Arbuthnott and Wickens, 2007).

How do Basal Ganglia Mediate Motor and Learning?

New Aspects of Direct/Indirect Pathways

Projection neurons in the striatum are classified into two groups, depending on their neurochemical properties and projection targets, which in turn transmit information via different routes to output nuclei, such as GPi and SNr. It is believed that the first projection group corresponds to a direct pathway, wherein the neurons containing both GABA and substance P directly project to the output nuclei, whereas the second one involves an indirect pathway, wherein the neurons containing GABA and enkephalin project to the output nuclei via the external segment of globus pallidus (GPe) and subthalamic nucleus (Albin et al., 1989; Alexander and Crutcher, 1990; Graybiel, 1990). Because these output nuclei contain GABAergic inhibitory neurons that discharge at a high rate, the projection targets in the thalamus and superior colliculus are usually in an inhibited state. Striatal projection neurons are GABAergic; therefore, excitation of these neurons by cortical inputs may lead to temporary inhibition of the output nuclei via the direct pathway and to disinhibition of the target regions (the thalamus and cerebral cortex), allowing selected movements to occur (Nambu et al., 2002). However, when the indirect pathway is activated, the target regions are further inhibited because projection neurons from GPe to the subthalamic nucleus are also GABAergic and those from the subthalamic nucleus to the output nuclei are glutamatergic. Therefore, while the direct pathway allows the expression of required movement via disinhibition during the necessary time period, the indirect pathway may be suppressing unnecessary movement and thus highlighting the outcome from the direct pathway. These findings suggest a “center–surround” model of basal ganglia function, comprising focused selection of an appropriate motor program and inhibition of competing motor programs (Mink and Thach, 1993; Mink, 1996; Hikosaka et al., 2000; Nambu et al., 2002).

Dopaminergic projections from SNc produce excitatory modulation of direct pathway neurons by an action at dopamine D1 receptors (D1Rs) and inhibitory modulation of indirect pathway neurons by an action at dopamine D2 receptors (D2Rs), effectively eliciting opposite effects in the direct and indirect pathways (Hong and Costa, 1978; Hong et al., 1978; Gerfen et al., 1990). This conceptualization has been widely accepted because it can explain clinical findings and therapeutic effects in disorders, such as Parkinson's disease.

However, it has been reported that a majority of striatal neurons are activated during movement (DeLong, 1990; Costa et al., 2004) and that both pathways are co-activated during movement initiation (Cui et al., 2013; Isomura et al., 2013). Further, recent optogenetic studies showed that both pathways were concomitantly active during sequence initiation but behaved differently during sequence performance (Jin et al., 2014). Single neuron tracing studies have revealed that almost all direct pathway neurons projected to GPe, a relay nucleus of the indirect pathway (Kawaguchi et al., 1990; Lévesque and Parent, 2005; Fujiyama et al., 2011), indicating that direct pathway neurons drive both direct and indirect pathways. Other projection systems have also been reported, such as the hyperdirect pathway (Nambu et al., 2002), cortico-dopaminergic projections (Watabe-Uchida et al., 2012), and differential cortical innervation of D1R- and D2R-positive striatal neurons (Wall et al., 2013). Furthermore, Kravitz et al. (2012) reported that positive reinforcement caused by direct pathway stimulation persists for long durations in mice, whereas punishment caused by indirect pathway stimulation was transient (Kravitz and Kreitzer, 2012; Kravitz et al., 2012). Jin et al. (2014) also showed that the basal ganglia contribute to behavior during learning rather than simple motor control (for review, see Friend and Kravitz, 2014). Thus, the original conceptualization of the direct and indirect pathways is likely to be modified through further studies, particularly those using behavioral experiments.

Direct/Indirect Pathways and Striosome/Matrix Structure

The output from striosome/matrix compartments has been difficult to examine with anterograde tracers because of the irregularities in the striatal structure. Using a single neuron tracing technique, we found that striosomes also include indirect pathway neurons projecting to GPe (Figure 2; Fujiyama et al., 2011). Further, unlike the matrix, direct pathway neurons in striosomes project not only to GPi/SNr but also directly to SNc, where dopaminergic neurons are present (Figure 2; Gerfen, 1984; Lévesque and Parent, 2005; Fujiyama et al., 2011; Watabe-Uchida et al., 2012).

FIGURE 2

Figure 2. A simplified diagram of output from the striosome–matrix structure (from Fujiyama et al., 2011 with modifications) GPe, external segment of globus pallidus; GPi, internal segment of globus pallidus; PPT, pedunculopontine nucleus; SNc, substantia nigra pars compacta; SNr, substantia nigra pars reticulata; STH, subthalamus.

The actor–critic model was supported by the finding that neurons responding to action and state values were distributed in the neostriatum (Samejima et al., 2005; Kawato and Samejima, 2007; Lau and Glimcher, 2008; Wang et al., 2013). Furthermore, dopamine signals contain information about reward and state value (Schultz et al., 1998; Schultz, 2007a,b); striosomal neurons, which directly control dopaminergic neurons in SNc, may provide the dopaminergic neurons with state-based signals. Dopaminergic neurons receive monosynaptic inhibitory input from SNr and polysynaptic disinihibitory input from the GPe (Grofova et al., 1982; Saitoh et al., 2004; Tepper and Lee, 2007; Brazhnik et al., 2008), and SNr and GPe are innervated by striosomal neurons. Hence, dopaminergic neurons may receive disinhibitory input through the SNr, and polysynaptic inhibitory input through the GPe, subthalamic nucleus, and SNr, from the striosomes. Dopaminergic neurons are also considered to receive a stochastic reward signal from the pedunculopontine tegmental nucleus and other structures (Figure 2; Okada et al., 2009). Our recent study revealed that the axonal arbors of most dopamine neurons covered a single large oval volume, occupying at most 5.7% of the total neostriatal volume. Furthermore, all single dopamine neurons we traced innervated striosomes and matrix compartments with or without volume transmission (Matsuda et al., 2009). Dopamine signals may thereby change the response properties of striatal neurons (Calabresi et al., 2000; Reynolds et al., 2001; Surmeier et al., 2007; Shen et al., 2008). The striosomal control of the dopaminergic neurons responsible for reward prediction errors is particularly relevant to its potential role as “critic;” it may serve this function by calculating the state value and sending it to dopaminergic neurons (Barto, 1995; Houk et al., 1995).

Ventral Striatum and Limbic Loop

In the basal ganglia, VTA–nucleus accumbens dopaminergic projection system has been considered along with the nigrostriatal dopaminergic projection system (see Groenewegen et al., 1999; Zahm, 1999). The ventral part of the striatum centered on the nucleus accumbens is called the limbic or ventral striatum; this structure is divided into a central portion (core) and periphery (shell) (Herkenham et al., 1984). GABAergic output from the shell projects either directly or through the ventromedial part of the ventral pallidum to the dorsomedial thalamic nucleus, the lateral hypothalamus, VTA, dopaminergic neurons in the substantia nigra, and is part of the mesolimbic dopaminergic projection system. The core involves a system that projects via the dorsolateral portion of the ventral pallidum to the subthalamic nucleus, SNr, and GPi, and subsequently enters the motor loop (see Groenewegen et al., 1999; Zahm, 1999). Therefore, it may help switch the input from the emotion system to the movement system. Dopamine projections from VTA to the nucleus accumbens are involved in selecting the environmental contexts leading to reward (O'Doherty et al., 2004; Canales, 2005; Goto and Grace, 2008; Humphries and Prescott, 2010; Glimcher, 2011; Morita et al., 2012, 2013; Hart et al., 2014). Takahashi et al. (2008); Takahashi et al. (2011) reported that information about task structure is represented in the orbitofrontal cortex and that it influences the computation of reward prediction error in VTA dopaminergic neurons. This presumably occurs via the ventral striatum, where the state value would be computed. However, a recent optogenetic study reported that the striatal medium spiny neurons, including striosomal neurons, synapse onto dopamine neurons only very weakly and instead strongly synapse onto GABAergic neurons in VTA (Chuhma et al., 2011), which in turn project to cholinergic neurons in the accumbens (Brown et al., 2012). Furthermore, up to 17% of accumbens shell neurons co-express D1R and D2R (Rashid et al., 2006; Bertran-Gonzalez et al., 2008; Ng et al., 2010); therefore, ventral striatal pathways are not well adapted to the direct/indirect schema of the dorsal striatum. Hence, further anatomical and physiological studies are necessary to test the synaptic connections of VTA–nucleus accumbens dopaminergic projection system.

Conclusion

Although both involve the striatum, the direct/indirect pathways and striosome/matrix compartments have been investigated separately. Two functions of the basal ganglia— motor control and reinforcement learning—have also been traditionally discussed and understood separately. However, recent studies have shown that direct pathway neurons mediate movement, reinforcement, and reward, whereas indirect pathway neurons inhibit movement and mediate punishment and aversion (see Kravitz and Kreitzer, 2012). Recent morphological studies have shown how the striatal mosaic organization and direct/indirect pathway neurons coexist in the striatum. This suggests that striosomes are involved in controlling dopaminergic neurons responsible for reward prediction errors and for direct and indirect pathways and is of a particular relevance to both motor control and reinforcement learning. The dual anatomical and physiological pathways in the striatum may hold the key to why the basal ganglia have two functions. Further studies should examine pathway- and compartment-specific activity simultaneously in different contexts to clarify how the structure of the basal ganglia contributes to behavioral learning.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This study was supported by Grants-in-Aid from The Ministry of Education, Culture, Sports, Science, and Technology (MEXT) for Scientific Research (25282247); and for Scientific Researches on Innovative Areas “Adaptive Circuit Shift” (26112001) and “Prediction and Decision Making” (26120725).

References

Albin, R. L., Young, A. B., and Penney, J. B. (1989). The functional anatomy of basal ganglia disorders. Trends Neurosci. 12, 366–375. doi: 10.1016/0166-2236(89)90074-X

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Alexander, G. E., and Crutcher, M. D. (1990). Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci. 13, 266–271. doi: 10.1016/0166-2236(90)90107-L

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Arbuthnott, G. W., and Wickens, J. (2007). Space, time and dopamine. Trends Neurosci. 30, 62–69. doi: 10.1016/j.tins.2006.12.003

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Arvidsson, U., Riedl, M., Chakrabarti, S., Lee, J. H., Nakano, A. H., Dado, R. J., et al. (1995). Distribution and targeting of a mu-opioid receptor (MOR1) in brain and spinal cord. J. Neurosci. 15, 3328–3341.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Barto, A. G. (1995). “Adaptive critics and the basal ganglia,” in Models of Information Processing in the Basal Ganglia, eds J. C. Houk, J. Davis, and D. Beiser (Cambridge, MA: MIT Press), 215–232.

Google Scholar

Barto, A. G., Sutton, R. S., and Anderson, C. W. (1983). Neuron-like adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 13, 834–846. doi: 10.1109/TSMC.1983.6313077

CrossRef Full Text | Google Scholar

Bayer, H. M., and Glimcher, P. W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141. doi: 10.1016/j.neuron.2005.05.020

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Beckstead, R. M. (1984). A projection to the striatum from the medial subdivision of the posterior group of the thalamus in the cat. Brain Res. 300, 351–356. doi: 10.1016/0006-8993(84)90845-X

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Bertran-Gonzalez, J. B., Bosch, C., Maroteaux, M., Herve, D., Valjent, E., and Girault, J. A. (2008). Opposing patterns of signaling activation in dopamine D1 and D2 receptor-expressing striatal neurons in response to cocaine and haloperidol. J. Neurosci. 28, 5671–5685. doi: 10.1523/JNEUROSCI.1039-08.2008

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Bolam, J. P., Bergman, H., Graybiel, A., Kimura, M., Plenz, D., Seung, H. S., et al. (2006). “Molecules, microcircuits and motivated behaviour: microcircuits in the striatum,” in Microcircuits: The Interface between Neurons and Global Brain Function. Dahlem Workshop Report 93, eds S. Grillner and A. Graybiel (Cambridge, MA: The MIT Press) 165–190.

Brazhnik, E., Shah, F., and Tepper, J. M. (2008). GABAergic afferents activate both GABAA and GABAB receptors in mouse substantia nigra dopaminergic neurons in vivo. J. Neurosci. 28, 10386–10398. doi: 10.1523/JNEUROSCI.2387-08.2008

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Bromberg-Martin, E. S., Matsumoto, M., and Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68, 815–834. doi: 10.1016/j.neuron.2010.11.022

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Brown, M. T. G., Tan, K. R., O'Connor, E. C., Nikonenko, I., Muller, D., and Lüscher, C. (2012). Ventral tegmental area GABA projections pause accumbal cholinergic interneurons to enhance associative learning. Nature 492, 452–456. doi: 10.1038/nature11657

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Calabresi, P., Centonze, D., and Bernardi, G. (2000). Electrophysiology of dopamine in normal and denervated striatal neurons. Trends Neurosci. 23, 57–63. doi: 10.1016/S1471-1931(00)00017-3

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Calabresi, P., Picconi, B., Tozzi, A., and Di Filippo, M. (2007). Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends Neurosci. 30, 211–219. doi: 10.1016/j.tins.2007.03.001

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Canales, J. J. (2005). Stimulant-induced adaptations in neostriatal matrix and striosome systems: transiting from instrumental responding to habitual behavior in drug addiction. Neurobiol. Learn. Mem. 83, 93–103. doi: 10.1016/j.nlm.2004.10.006

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Chuhma, N., Tanaka, K. F., Hen, R., and Rayport, S. (2011). Functional connectome of the striatal medium spiny neuron. J. Neurosci. 31, 1183–1192. doi: 10.1523/JNEUROSCI.3833-10.2011

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., and Uchida, N. (2012). Neuron-type specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88. doi: 10.1038/nature10754

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Cornwall, J., and Phillipson, O. T. (1988). Afferent projections to the dorsal thalamus of the rat as shown by retrograde lectin transport. II. The midline nuclei. Brain Res. Bull. 21, 147–161. doi: 10.1016/0361-9230(88)90227-4

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Costa, R. M., Cohen, D., and Nicolelis, M. A. L. (2004). Differential corticostriatal plasticity during fast and slow motor skill learnig in mice. Curr. Biol. 14, 1124–1134. doi: 10.1016/j.cub.2004.06.053

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Crittenden, J. R., and Graybiel, A. M. (2011). Basal Ganglia disorders associated with imbalances in the striatal striosome and matrix compartments. Front. Neuroanat. 5:59. doi: 10.3389/fnana.2011.00059

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Cui, G., Jun, S. B., Jin, X., Pham, M. D., Vogel, S. S., Lovinger, D. M., et al. (2013). Concurrent activation of striatal direct and indirect pathways during action initiation. Nature 494, 238–242. doi: 10.1038/nature11846

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Delfs, J. M., Kong, H., Mestek, A., Chen, Y., Yu, L., Reisine, T., et al. (1994). Expression of mu opioid receptor mRNA in rat brain: an in situ hybridization study at the single cell level. J. Comp. Neurol. 345, 46–68. doi: 10.1002/cne.903450104

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

DeLong, M. R. (1990). Primate models of movement disorders of basal ganglia origin. Trends Neurosci. 13, 281–285. doi: 10.1016/0166-2236(90)90110-V

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Deschênes, M., Bourassa, J., Doan, V. D., and Parent, A. (1996). A single-cell study of the axonal projections arising from the posterior intralaminar thalamic nuclei in the rat. Eur. J. Neurosci. 8, 329–343. doi: 10.1111/j.1460-9568.1996.tb01217.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Ding, Y. Q., Kaneko, T., Nomura, S., and Mizuno, N. (1996). Immunohistochemical localization of mu-opioid receptors in the central nervous system of the rat. J. Comp. Neurol. 367, 375–402.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Donoghue, J. P., and Herkenham, M. (1986). Neostriatal projections from individual cortical fields conform to histochemically distinct striatal compartments in the rat. Brain Res. 365, 397–403. doi: 10.1016/0006-8993(86)91658-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw. 12, 961–974. doi: 10.1016/S0893-6080(99)00046-5

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Doya, K. (2000a). Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol. 10, 732–739. doi: 10.1016/S0959-4388(00)00153-7

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Doya, K. (2000b). Reinforcement learning in continuous time and space. Neural Comput. 12, 219–245. doi: 10.1162/089976600300015961

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Doya, K., Samejima, K., Katagiri, K., and Kawato, M. (2002). Multiple model-based reinforcement learning. Neural Comput. 14, 1347–1369. doi: 10.1162/089976602753712972

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Flaherty, A. W., and Graybiel, A. M. (1994). Input-output organization of the sensorimotor striatum in the squirrel monkey. J. Neurosci. 14, 599–610.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Friend, D. M., and Kravitz, A. V. (2014). Working together: basal ganglia pathways in action selection. Trends Neurosci. 37, 301–303. doi: 10.1016/j.tins.2014.04.004

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Fujiyama, F., Furuta, T., and Kaneko, T. (2001). Immunocytochemical localization of candidates for vesicular glutamate transporters in the rat cerebral cortex. J. Comp. Neurol. 435, 379–387. doi: 10.1002/cne.1037

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Fujiyama, F., Kuramoto, E., Okamoto, K., Hioki, H., Furuta, T., Zhou, L., et al. (2004). Presynaptic localization of an AMPA-type glutamate receptor in corticostriatal and thalamostriatal axon terminals. Eur. J. Neurosci. 20, 3322–3330. doi: 10.1111/j.1460-9568.2004.03807.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Fujiyama, F., Sohn, J., Nakano, T., Furuta, T., Nakamura, K. C., Matsuda, W., et al. (2011). Exclusive and common targets of neostriatofugal projections of rat striosome neurons: a single neuron-tracing study using a viral vector. Eur. J. Neurosci. 33, 668–677. doi: 10.1111/j.1460-9568.2010.07564.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Fujiyama, F., Unzai, T., Nakamura, K., Nomura, S., and Kaneko, T. (2006). Difference in organization of corticostriatal and thalamostriatal synapses between patch and matrix compartments of rat neostriatum. Eur. J. Neurosci. 24, 2813–2824. doi: 10.1111/j.1460-9568.2006.05177.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Gerfen, C. R. (1984). The neostriatal mosaic: compartmentalization of corticostriatal input and striatonigral output systems. Nature 311, 461–464. doi: 10.1038/311461a0

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Gerfen, C. R. (1985). The neostriatal mosaic. I. Compartmental organization of projections from the striatum to the substantia nigra in the rat. J. Comp. Neurol. 236, 454–476. doi: 10.1002/cne.902360404

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Gerfen, C. R. (1989). The neostriatal mosaic: striatal patch-matrix organization is related to cortical lamination. Science 246, 385–388. doi: 10.1126/science.2799392

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Gerfen, C. R. (1992). The neostriatal mosaic: multiple levels of compartmental organization. Trends Neurosci. 15, 133–139. doi: 10.1016/0166-2236(92)90355-C

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Gerfen, C. R. (2004). “The basal ganglia,” in The Rat Nervous System, ed G. Paxinos (San Diego, CA: Elsevier Academic Press), 455–508.

Google Scholar

Gerfen, C. R., Engber, T. M., Mahan, L. C., Susel, Z., Chase, T. N., Monsma, F. J., et al. (1990). D1 and D2 dopamine receptor regulated gene expression of striatonigral and striatopallidal neurons. Science 250, 1429–1432. doi: 10.1126/science.2147780

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Gerfen, C. R., Herkenham, M., and Thibault, J. (1987). The neostriatal mosaic. II. Patch- and matrix- directe mesostriatal dopaminergic and non-dopaminergic systems. J. Neurosci. 7, 3915–3934.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Gerfen, C. R., and Young, W. S. (1988). Distribution of striatonigral and striatopallidal peptidergic neurons in both patch and matrix compartments: an in situ hybridization histochemistry and fluorescent retrograde study. Brain Res. 460, 161–167. doi: 10.1016/0006-8993(88)91217-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Glimcher, P. W. (2011). Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. U.S.A. 108, 15647–15654. doi: 10.1073/pnas.1014269108

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Goto, Y., and Grace, A. A. (2008). Limbic and cortical information processing in the nucleus accumbens. Trends Neurosci. 31, 552–558. doi: 10.1016/j.tins.2008.08.002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Graybiel, A. M. (1990). Neurotransmitters and neuromodulators in the basal ganglia. Trends Neurosci. 13, 244–254. doi: 10.1016/0166-2236(90)90104-I

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Graybiel, A. M., and Ragsdale, C. W. Jr. (1978). Histochemically distinct compartments in the striatum of human, monkeys, and cat demonstrated by acetylthiocholinesterase staining. Proc. Natl. Acad. Sci. U.S.A. 75, 5723–5726. doi: 10.1073/pnas.75.11.5723

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Groenewegen, H. J., Wright, C. I., Beijer, A. V. J., and Voorn, P. (1999). Convergence and segregation of ventral striatal inputs and outputs. Ann. N.Y. Acad. Sci. 877, 49–63. doi: 10.1111/j.1749-6632.1999.tb09260.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Grofova, I., Deniau, J. M., and Kitai, S. T. (1982). Morphology of the substantia nigra pars reticulata projection neurons intracellularly labeled with HRP. J. Comp. Neurol. 208, 352–368. doi: 10.1002/cne.902080406

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Hart, A. S., Rutledge, R. B., Glimcher, P. W., and Phillips, P. E. M. (2014). Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. J. Neurosci. 34, 698–704. doi: 10.1523/JNEUROSCI.2489-13.2014

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Herkenham, M., Edley, S. M., and Stuart, J. (1984). Cell clusters in the nucleus accumbens of the rat, and the mosaic relationship of opiate receptors, acetylcolinesterase and subcortical afferent terminations. Neuroscience 11, 561–593. doi: 10.1016/0306-4522(84)90045-9

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Herkenham, M., and Pert, C. B. (1981). Mosaic distribution of opiate receptors, parafascicular projections and acetylcholinesterase in rat striatum. Nature 291, 415–418. doi: 10.1038/291415a0

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Hikosaka, O., Takikawa, Y., and Kawagoe, R. (2000). Role of the basal ganglia in the control of purposive saccadic eye movements. Physiol. Rev. 80, 953–978. Available online at: http://physrev.physiology.org/content/physrev/80/3/953.full.pdf#pdfjs.action=download

Pubmed Abstract | Pubmed Full Text

Hong, J. S., and Costa, E. (1978). Participation of substance P in the action of haloperidol. Psychopharmacol. Bull. 14, 23–24.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Hong, J. S., Yang, H. Y., Fratta, W., and Costa, E. (1978). Rat striatal methionine-enkephalin content after chronic treatment with cataleptogenic and noncataleptogenic antischizophrenic drugs. J. Pharmacol. Exp. Ther. 205, 141–147.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Houk, J. C., Adams, J. L., and Barto, A. G. (1995). “A model of how the basal ganglia generate and use neural signals that predict reinforcement,” in Models of Information Processing in the Basal Ganglia, eds J. C. Houk, J. Davis, and D. Beiser (Cambridge, MA: The MIT Press), 215–232.

Google Scholar

Humphries, M. D., and Prescott, T. J. (2010). The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Prog. Neurobiol. 90, 385–417. doi: 10.1016/j.pneurobio.2009.11.003

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Isomura, Y., Takekawa, T., Harukuni, R., Handa, T., Aizawa, H., Takada, M., et al. (2013). Reward-modulated motor information in identified striatum neurons. J. Neurosci. 33, 10209–10220. doi: 10.1523/JNEUROSCI.0381-13.2013

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Jimenez-Castellanos, J., and Graybiel, A. M. (1987). Subdivisions of the dopamine-containing A8-A9-A10 complex identified by their differential mesostriatal innervation of striosomes and extrastriosomal matrix. Neuroscience 23, 223–242. doi: 10.1016/0306-4522(87)90285-5

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Jin, X., Tecuapetla, F., and Costa, R. M. (2014). Basal ganglia subcircuits distinctively encode the parsing and concatenation of action sequences. Nat. Neurosci. 17, 423–430. doi: 10.1038/nn.3632

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Johnston, J. G., Gerfen, C. R., Haber, S. N., and van der Kooy, D. (1990). Mechanisms of striatal pattern formation: conservation of mammalian compartmentalization. Brain Res. Dev. Brain Res. 57, 93–102. doi: 10.1016/0165-3806(90)90189-6

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kaneko, T., and Fujiyama, F. (2002). Complementary distribution of vesicular glutamate transporters in the central nervous system. Neurosci. Res. 42, 243–250. doi: 10.1016/S0168-0102(02)00009-3

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kaneko, T., Fujiyama, F., and Hioki, H. (2002). Immunohistochemical localization of candidates for vesicular glutamate transporters in the rat brain. J. Comp. Neurol. 444, 39–62. doi: 10.1002/cne.10129

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kaneko, T., Minami, M., Satoh, M., and Mizuno, N. (1995). Immunocytochemical localization of mu-opioid receptor in the rat caudate-putamen. Neurosci. Lett. 184, 149–152. doi: 10.1016/0304-3940(94)11192-L

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kato, S., Kuramochi, M., Kobayashi, K., Fukabori, R., Okada, K., Uchigashima, M., et al. (2011). Selective neural pathway targeting reveals key roles of thalamostriatal projection in the control of visual discrimination. J. Neurosci. 31, 17169–17179. doi: 10.1523/JNEUROSCI.4005-11.2011

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kawaguchi, Y., Wilson, C. J., and Emson, P. C. (1990). Projection subtypes of rat neostriatal matrix cells revealed by intracellular injection of biocytin. J. Neurosci. 10, 3421–3438.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Kawato, M., and Samejima, K. (2007). Efficient reinforcement learning: computational theories, neuroscience and robotics. Curr. Opin. Neurobiol. 17, 205–212. doi: 10.1016/j.conb.2007.03.004

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kimura, M., Minamimoto, T., Matsumoto, N., and Hori, Y. (2004). Monitoring and switching of cortico-basal ganglia loop functions by the thalamo-striatal system. Neurosci. Res. 48, 355–360. doi: 10.1016/j.neures.2003.12.002

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kincaid, A. E., and Wilson, C. J. (1996). Corticostriatal innervation of the patch and matrix in the rat neostriatum. J. Comp. Neurol. 374, 578–592.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Kravitz, A. V., and Kreitzer, A. C. (2012). Striatal mechanisms underlying movement, reinforcement, and punishment. Physiology 27, 167–177. doi: 10.1152/physiol.00004.2012

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Kravitz, A. V., Tye, L. D., and Kreitzer, A. C. (2012). Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat. Neurosci. 15, 816–818. doi: 10.1038/nn.3100

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Langer, L. F., and Graybiel, A. M. (1989). Distinct nigrostriatal projection systems innervate striosomes and matrix in the primate striatum. Brain Res. 498, 344–350. doi: 10.1016/0006-8993(89)91114-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Lau, B., and Glimcher, P. W. (2008). Value representations in the primate striatum during matching behaviour. Neuron 58, 451–463. doi: 10.1016/j.neuron.2008.02.021

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Lévesque, M., and Parent, A. (2005). The striatofugal fiber system in primates: a reevaluation of its organization based on single-axon tracing studies. Proc. Natl. Acad. Sci. U.S.A. 102, 11888–11893. doi: 10.1073/pnas.0502710102

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Malach, R., and Graybiel, A. M. (1986). Mosaic architecture of the somatic sensory-recipient sector of the cat's striatum. J. Neurosci. 6, 3436–3458.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Mansour, A., Fox, C. A., Burke, S., Akil, H., and Watson, S. J. (1995). Immunohistochemical localization of the cloned mu opioid receptor in the rat CNS. J. Chem. Neuroanat. 8, 283–305. doi: 10.1016/0891-0618(95)00055-C

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Mansour, A., Fox, C. A., Burke, S., Meng, F., Thompson, R. C., Akil, H., et al. (1994). Mu, delta, and kappa opioid receptor mRNA expression in the rat CNS: an in situ hybridization study. J. Comp. Neurol. 350, 412–438. doi: 10.1002/cne.903500307

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Matsuda, W., Furuta, T., Nakamura, K. C., Hioki, H., Fujiyama, F., Arai, R., et al. (2009). Single nigrostriatal dopaminergic neurons form widely spread and highly dense axonal arborizations in the neostriatum. J. Neurosci. 29, 444–453. doi: 10.1523/JNEUROSCI.4029-08.2009

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Minami, M., Onogi, T., Toya, T., Katao, Y., Hosoi, Y., Maekawa, K., et al. (1994). Molecular cloning and in situ hybridization histochemistry for rat mu-opioid receptor. Neurosci. Res. 18, 315–322. doi: 10.1016/0168-0102(94)90167-8

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Mink, J. W. (1996). The basal ganglia: focused selection and inhibition of competing motor programs. Prog. Neurobiol. 50, 381–425. doi: 10.1016/S0301-0082(96)00042-1

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Mink, J. W., and Thach, W. T. (1993). Basal ganglia intrinsic circuits and their role in behaviour. Curr. Opin. Neurobiol. 3, 950–957.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Montague, P. R., Doyan, P., and Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Morita, K., Morishima, M., Sakai, K., and Kawaguchi, Y. (2012). Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways. Trends Neurosci. 35, 457–467. doi: 10.1016/j.tins.2012.04.009

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Morita, K., Morishima, M., Sakai, K., and Kawaguchi, Y. (2013). Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior. J. Neurosci. 33, 8866–8890. doi: 10.1523/JNEUROSCI.4614-12.2013

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Nakamura, K. C., Fujiyama, F., Furuta, T., Hioki, H., and Kaneko, T. (2009). Afferent islands are larger than μ-opioid receptor patch in striatum of rat pups. Neuroreport 20, 584–588. doi: 10.1097/WNR.0b013e328329cbf9

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Nambu, A., Tokuno, H., and Takada, M. (2002). Functional significance of the cortico–subthalamo–pallidal ‘hyperdirect’ pathway. Neurosci. Res. 43, 111–117. doi: 10.1016/S0168-0102(02)00027-5

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Ng, J., Rashid, A. J., So, C. H., O'Dowd, B. F., and George, S. R. (2010). Activation of calcium/calmodulin-dependent protein kinase llalpha in the striatum by the heteromeric D1-D2 dopamine receptor complex. Neuroscience 165, 535–541. doi: 10.1016/j.neuroscience.2009.10.017

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., and Dolan, R. (2004). Dissociable roles of dorsal and ventral striatum in instrumental conditioning. Science 304, 452–454. doi: 10.1126/science.1094285

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Okada, K., Toyama, K., Inoue, Y., Isa, T., and Kobayashi, Y. (2009). Different pedunculopontine tegmental neurons signal predicted and actual task rewards. J. Neurosci. 29, 4858–4870. doi: 10.1523/JNEUROSCI.4415-08.2009

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Ragsdale, C. W. Jr., and Graybiel, A. M. (1991). Compartmental organization of the thalamostriatal connection in the cat. J. Comp. Neurol. 311, 134–167. doi: 10.1002/cne.903110110

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Raju, D. V., Shah, D. J., Wright, T. M., Hall, R. A., and Smith, Y. (2006). Differential synaptology of vGluT2-containing thalamostriatal afferents between the patch and matrix compartments in rats. J. Comp. Neurol. 499, 231–243. doi: 10.1002/cne.21099

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Rashid, A. J., So, C. H., Kong, M. M., Furtak, T., El-Ghundil, M., Cheng, R., et al. (2006). D1-D2 dopaminereceptor heterooligomers with unique pharmacology are coupled to rapid activation of Gq/11 in the striatum. Proc. Natl. Acad. Sci. U.S.A. 104, 654–659. doi: 10.1073/pnas.0604049104

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Reynolds, J. N., Hyland, B. I., and Wickens, J. R. (2001). A cellular mechanism of reward-related learning. Nature 413, 67–70. doi: 10.1038/35092560

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Sadikot, A. F., Parent, A., and Francois, C. (1990). The centre median and parafascicular thalamic nuclei project respectively to the sensorimotor and associative-limbic striatal territories in the squirrel monkey. Brain Res. 510, 161–165. doi: 10.1016/0006-8993(90)90746-X

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Sadikot, A. F., Parent, A., Smith, Y., and Bolam, J. P. (1992). Efferent connections of the centromedian and parafascicular thalamic nuclei in the squirrel monkey: a light and electron microscopic study of the thalamostriatal projection in relation to striatal heterogeneity. J. Comp. Neurol. 320, 228–242. doi: 10.1002/cne.903200207

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Saitoh, K., Isa, T., and Takakusaki, K. (2004). Nigral GABAergic inhibition upon mesencephalic dopaminergic cell groups in rats. Eur. J. Neurosci. 19, 2399–2409. doi: 10.1111/j.0953-816X.2004.03337.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Samejima, K., Ueda, Y., Doya, K., and Kimura, M. (2005). Representation of action-specific reward values in the striatum. Science 310, 1337–1340. doi: 10.1126/science.1115270

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Schultz, W. (2007a). Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288. doi: 10.1146/annurev.neuro.28.061604.135722

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Schultz, W. (2007b). Behavioral dopamine signals. Trends Neurosci. 30, 203–210. doi: 10.1016/j.tins.2007.03.007

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Schultz, W., Apicella, P., and Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913.

Pubmed Abstract | Pubmed Full Text | Google Scholar

Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593–1599. doi: 10.1126/science.275.5306.1593

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Schultz, W., Tremblay, L., and Hollerman, J. R. (1998). Reward prediction in primate basal ganglia and frontal cortex. Neuropharmacology 37, 421–429. doi: 10.1016/S0028-3908(98)00071-9

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Shen, W., Flajolet, M., Greengard, P., and Surmeier, D. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321, 848–851. doi: 10.1126/science.1160575

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Smith, A. D., and Bolam, J. P. (1990). The neural network of the basal ganglia as revealed by the study of synaptic connections of identified neurones. Trends Neurosci. 13, 259–265. doi: 10.1016/0166-2236(90)90106-K

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Surmeier, D. J., Ding, J., Day, M., Wang, Z., and Shen, W. (2007). D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci. 30, 228–235. doi: 10.1016/j.tins.2007.03.008

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Sutton, R. S. (1988). Learning to Predict by the Methods of Temporal Differences. Mach. Learn. 3, 9–44. doi: 10.1007/BF00115009

CrossRef Full Text | Google Scholar

Sutton, R. S., and Barto, A. G. (1995). Reinforcement Learning: An Introduction. Cambridge, MA: The MIT Press.

Google Scholar

Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., et al. (2011). Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat. Neurosci. 14, 1590–1597. doi: 10.1038/nn.2957

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Takahashi, Y., Schoenbaum, G., and Niv, Y. (2008). Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an Actor/Critic model. Front. Neurosci. 2:86–99. doi: 10.3389/neuro.01.014.2008

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Tepper, J. M., and Lee, C. R. (2007). GABAergic control of substantia nigra dopaminergic neurons. Prog. Brain Res. 160, 189–208. doi: 10.1016/S0079-6123(06)60011-3

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Wall, N. R., Parra, M. D. L., Callaway, E. M., and Kreitzer, A. C. (2013). Differential innervation of direct- and indirect-pathway striatal projection neurons. Neuron 79, 1–14. doi: 10.1016/j.neuron.2013.05.014

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Wang, A. Y., Miura, K., and Uchida, N. (2013). The dorsomedial striatum encodes net expected return, critical for energizing performance vigor. Nat. Neurosci. 16, 639–647. doi: 10.1038/nn.3377

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Watabe-Uchida, M., Zhu, L., Ogawa, S. K., Vamanrao, A., and Uchida, N. (2012). Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron 74, 858–873. doi: 10.1016/j.neuron.2012.03.017

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Yuste, R., and Bonhoeffer, T. (2001). Morphological changes in dendritic spines associated with long-term synaptic plasticity. Annu. Rev. Neurosci. 24, 1071–1089. doi: 10.1146/annurev.neuro.24.1.1071

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar

Zahm, D. S. (1999). Functional-anatomical implications of the nucleus accumbens core and shell subterritories. Ann. N.Y. Acad. Sci. 877, 113–128. doi: 10.1111/j.1749-6632.1999.tb09264.x

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar