# SYNAESTHESIA

EDITED BY: Michael Banissy, Roi Cohen Kadosh and Clare Jonas PUBLISHED IN: Frontiers in Psychology

### *Frontiers Copyright Statement*

*© Copyright 2007-2015 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

> *All copyright, and all rights therein, are protected by national and international copyright laws.*

*The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88919-559-6 DOI 10.3389/978-2-88919-559-6

# About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

# Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

# Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

# What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# **SYNAESTHESIA**

Topic Editors: **Michael Banissy,** Goldsmiths University of London, UK **Roi Cohen Kadosh,** University of Oxford, UK **Clare Jonas,** University of East London, UK

Synaesthesia is a rare experience in which one property of a stimulus evokes a secondary experience that is not typically associated with the first (e.g. hearing words can evoke tastes). In recent years a number of studies have highlighted the authenticity of synaesthesia and attempted to use the experience to inform us about typical processes in perception and cognition.

This Research Topic brings together research on synaesthesia and typical cross modal interactions to discuss the mechanisms of synaesthesia and what it can tell us about typical perceptual processes. Topics include, but are not limited to, the neurocognitive mechanisms that give rise to synaesthesia; the extent to which synaesthesia does / does not share commonalities with typical cross-modal correspondences; broader cognitive and perceptual consequences that are linked to synaesthesia; and perspectives on the origins / defining characteristics of synaesthesia.

**Citation:** Banissy, M., Cohen Kadosh, R., Jonas, C., eds. (2015). Synaesthesia. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-559-6

# Table of Contents



Alastair Haigh, David J. Brown, Peter Meijer and Michael J. Proulx

*153 Combined structural and functional imaging reveals cortical deactivations in grapheme-color synaesthesia*

Erik O'Hanlon, Fiona N. Newell and Kevin J. Mitchell

*166 A taste for words and sounds: a case of lexical-gustatory and sound-gustatory synesthesia*

Olympia Colizoli, Jaap M. J. Murre and Romke Rouw


# Synesthesia: an introduction

### *Michael J. Banissy1 \*, Clare Jonas <sup>2</sup> and Roi Cohen Kadosh3*

*<sup>1</sup> Department of Psychology, Goldsmiths, University of London, London, UK*

*<sup>2</sup> School of Psychology, University of East London, London, UK*

*<sup>3</sup> Department of Experimental Psychology, University of Oxford, Oxford, UK*

*\*Correspondence: m.banissy@gold.ac.uk*

### *Edited and reviewed by:*

*Eddy J. Davelaar, Birkbeck College, UK*

**Keywords: synaesthesia/synesthesia, multisensory, sensory substitution, perception, mirror-touch, grapheme-color synaesthesia, lexical-gustatory synaesthesia, cross-modal correspondence**

Synesthesia is a rare experience where one property of a stimulus evokes a second experience not associated with the first. For example, in lexical-gustatory synesthesia words evoke the experience of tastes (Ward and Simner, 2003). There are at least 60 known variants of synesthesia (Day, 2013), including reports of synesthetic experiences of color (Baron-Cohen et al., 1987), taste (Ward and Simner, 2003), touch (Ward et al., 2008), and sound (Saenz and Koch, 2008). The lower bound prevalence of the condition is considered to be approximately 4% (Simner et al., 2006). While synesthetic experiences have been documented since the 1800s (Jewanski et al., 2009), it is only in the last few decades that the authenticity of synesthetic experiences and mechanisms that contribute to them has been explored in depth (Ward, 2013). This resurgence in research has led to developments in our understanding of mechanisms that contribute to the synesthetic experience and the use of synesthesia as a unique experimental preparation to inform us about typical models of cognition and perception (e.g., Cohen Kadosh and Henik, 2007; Simner, 2007; Bargary and Mitchell, 2008; Rouw et al., 2011). This has also resulted in many open questions and debates, several of which are touched upon in this research topic. Specifically, this research topic is focused around the following themes: What constitutes synesthesia and how does it relate to typical cross-modal interactions? What mechanisms contribute to synesthetic experiences? Are there broader cognitive and perceptual traits associated with synesthesia, and what mechanisms mediate their relationship? In total, there are 20 articles, each addressing at least one of these themes.

# **WHAT CONSTITUTES SYNESTHESIA AND HOW DOES IT RELATE TO TYPICAL CROSS-MODAL INTERACTIONS?**

Several authors focus on discussing what synesthesia is and how it relates to typical cross-modal interactions. Mylopoulos and Ro (2013) provide a critical review of methods used for understanding and classifying synesthesia and provide a set of markers to aid in distinguishing synesthesia from other psychological phenomena. Marks and Mulvenna (2013) provide an interesting opinion article on cases that border on traditional forms of synesthesia and discuss whether these forms do or do not constitute forms of synesthesia. Similarly, Deroy and Spence (2013) discuss the notion of "induced synesthesia," arguing that current attempts to induce synesthesia may not be evidentially linked to developmental synesthesia.

Moos et al. (2013) touch on the theme of how synesthesia relates to typical cross-modal interactions, reporting findings on color and texture associations in voice induced synesthesia, suggesting common underlying mechanisms in cross-modal associations between synesthetes and non-synesthetes. Additionally, Rouw et al. (2014) examine the relationship between synesthetic and non-synesthetic cross-modal representations by assessing color associations for days and letters across different languages.

Haigh et al. (2013) examine the acuity of visual-to-auditory sensory substation and discuss whether visual imagery evoked by the device is a form of or synthetic synesthesia. Similarly, Simner (2013) contributes to themes of what mechanisms contribute to synesthetic experiences and what constitutes synesthesia by discussing the role of visual mental imagery in different types of synesthetes. In her thoughtful discussion, she suggests that differences between projector (for whom synesthetic experiences are projected onto external objects) and associator (for whom synesthetic experiences appear in the "minds eye") synesthetes may emerge from individual differences in visual mental imagery.

# **WHAT MECHANISMS CONTRIBUTE TO SYNESTHETIC EXPERIENCES?**

In addition to Simner's (2013) proposal that synesthetic experience is associated with individual differences in visual mental imagery, several other articles also address the mechanisms underlying synesthesia. Gertner et al. (2013) propose that numerical synesthesia is more than a symbol-induced phenomenon, and may also be induced by non-symbolic magnitudes. Perry and Henik (2013) report an experiment addressing emotional conflict sensations evoked in synesthesia when synesthetic photisms and veridical experiences conflict (e.g., when a numeral is presented in the wrong color). They discuss their findings in relation to emotional experience in synesthesia and the extent to which synesthesia may be used as a vehicle to inform us about emotional processing in the wider population. Dael et al. (2013) also address affect in synesthesia by providing a thoughtful review article on affect-related synesthesias and underlying mechanisms. Additionally, Jarick et al. (2013) examine vantage point preference and visual dominance in a time-space synesthesia, reporting that their synesthete is able to reverse her perspective on "time."

Moving to neural mechanisms, O'Hanlon et al. (2013) provide a research article examining structural and functional brain correlates of grapheme-color synesthesia. Colizoli et al. (2013) report a single case brain imaging study of lexical-gustatory and sound-gustatory synesthesia. Additionally, Luke and Terhune (2013) provide an important review of the induction of synesthesia with chemical agents, highlighting the potential role of the serotonergic system in synesthesia.

Finally Chiou and Rich (2014) discuss the role of conceptual knowledge in understanding synesthesia. Using a "hub and spokes" approach they present a model of synesthesia in which the inducer and concurrent are linked within a conceptual-level representation.

# **ARE THERE BROADER COGNITIVE AND PERCEPTUAL TRAITS ASSOCIATED WITH SYNESTHESIA, AND WHAT MECHANISMS MEDIATE THEIR RELATIONSHIP?**

Several authors highlight that synesthetes show different performance on tasks that are not directly related to their synesthetic experience. For example, Chun and Hupé (2013) address the relationship between synesthesia and other perceptual experiences by examining the prevalence of mirror-touch (Banissy et al., 2009) and ticker tape (Day, 2005) experiences in synesthesia. Bouvet et al. (2014) report an interesting case study involving an autistic individual who possesses savant abilities in addition to absolute pitch and synesthetic-like associations. They discuss the case in relation to the role of veridical mapping in autism, absolute pitch and synesthesia. Nielsen et al. (2013) examine and discuss the influence of synesthetic perceptions on sexual experience.

In relation to cognitive differences between synesthetes and non-synesthetes, Ward et al. (2013) address the relationship between grapheme-color synesthesia and enhanced recognition memory by comparing visual recognition memory in graphemecolor and lexical gustatory synesthesia. They show that graphemecolor synesthetes show enhanced visual recognition memory, but this is not found for lexical-gustatory synesthesia. Meier and Rothen (2013) investigate whether synesthesia is associated with a particular cognitive style, and provide evidence to suggest that grapheme-color synesthetes show a preference for a verbal and a specific visual cognitive style.

### **CONCLUDING REMARKS**

In summary, this Research Topic provides a novel collection of articles on synesthesia addressing a range of issues. We envisage that many of these will provide productive new research areas, and conceptual frameworks for the future study of synesthesia and related processes.

### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 28 October 2014; accepted: 19 November 2014; published online: 15 December 2014.*

*Citation: Banissy MJ, Jonas C and Cohen Kadosh R (2014) Synesthesia: an introduction. Front. Psychol. 5:1414. doi: 10.3389/fpsyg.2014.01414*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Banissy, Jonas and Cohen Kadosh. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Why are there different types of synesthete?

# *Julia Simner\**

*Department of Psychology, School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, UK \*Correspondence: jsimner@ed.ac.uk*

### *Edited by:*

*Clare Jonas, University of East London, UK*

### *Reviewed by:*

*Nicolas Rothen, University of Sussex, UK Michelle Jarick, MacEwan University, Canada*

**Keywords: individual differences, visual imagery, synaesthesia, synesthesia, projector, associator**

For people with synesthesia, sensations in two modalities are experienced when only one is stimulated (e.g., auditory stimuli might trigger colors and sounds). Synesthetes are manifestly different to the general population, but can also be different to each other. First, the condition is widely heterogeneous in that 60–150 different manifestations of synesthesia have been identified (e.g., auditory stimuli might trigger colors, shapes, flavors and so on; Cytowic and Eagleman, 2009). Second, synesthetes can differ on the *quality* of their synesthetic perceptions even within a given variant. Some experience their synesthetic percepts as being similar in quality to a real-world perceptions (e.g., synesthetic colors might be projected onto external objects and be difficult to dissociate from real-world colors) while other synesthetes experience less "veridical" percepts (see below). In this opinion piece I ask whether this particular difference known as the "projector" vs. "associator" distinction—might fall out naturally from another, independent psychological quality.

The projector/associator distinction was first phrased (by Dixon et al., 2004) in terms of *grapheme-color synesthesia*, in which colors are triggered by letters and/or digits. Some grapheme-color synesthetes report that their colors are experienced outside their own body space, projected into the world, and these are termed "projector" synesthetes. Ward and colleagues further divide this category into two: "surface-projectors" experience color projected onto the written type-face (or more generally, onto the inducing stimulus whatever that might be), and "spaceprojectors" project color onto some other externalized near-space. In contrast to this external projection, other synesthetes (termed "associators") can be thought of as "non-projectors" in that their colors exist within their internal mental space. Those associators who claim to see colors in their mind's eye have been termed "seeassociators," while those who simply claim to *know* the colors of graphemes without any associated impression of "seeing" at all have been termed "know-associators" (Ward et al., 2007). In other words there is a four-way divide between synesthetes experiencing colors in a way resembling real-world experiences (projected onto the stimulus or into near-space) and those experiencing them "internally" (in the mind's eye or as a type of propositional association). The claim put forward in this piece is a parsimonious one: that these differences might emerge from an otherwise unrelated individual difference, in the ability to form a visual mental image.

Visual imagery—the mental construction of a scene-like object—is known to vary across individuals (e.g., Marks, 1973). At the upper extreme end there are individuals with "eidetic" imagery who report strongly evoked mental images with an almost veridical quality. At the opposite end of the healthy spectrum are those who report being unable to form mental images at all. Taking these latter into consideration, and assuming a prevalence of synesthesia of at least 4% (Simner et al., 2006), we might assume—all other things being equal—that 4% of people with poor or no imagery abilities will have synesthesia. The proposal here is that these individuals are precisely those termed "know associators." Equally, those with extreme imagery abilities might be those we recognise as projectors, because their high imagery allows their synesthetic associations to become "scene-like" to an extreme extent. Note that this view (if it were correct) would rely on the assumption that imagery and synesthesia are independent qualities and I now compare this view with current thinking in the literature.

Some researchers have proposed that synesthetes are in fact characterized, as a group, by superior mental imagery (e.g., Barnett and Newell, 2008; Price, 2009a,b), and some have gone so far as to ask whether some visual synesthesias may be *nothing more than* extreme visual imagery (Galton, 1880; Phillips, 1897; Price, 2009a). One well-phrased expression of a possible link between synesthesia and imagery is that normal cross-modal sensations (say, between numbers and space) may enter into consciousness only if the individual has heightened imagery (Eagleman, 2009; Price, 2009a), and this is precisely when he/she would become considered a "synesthete" (Simner, 2012). (The resultant synesthesia in this example would be *sequence-space synesthesia* in which sequences such as numbers are consciously experienced in spatial arrays). In contrast with this view, others have suggested that synesthesia and imagery might be quasi-independent in that synesthetes might vary in their imagery ability, and hence some could have relatively weak imagery (see Grossenbacher and Lovelace, 2001, who put this view in passing).

Questions about synesthesia and imagery have also been tested empirically. Barnett and Newell (2008) showed that (*n* = 38) synesthetes with colored language (e.g., grapheme-color synesthesia) report significantly stronger vivid everyday mental images than controls, in a self-report questionnaire (*Vividness of Mental Imagery Questionnaire*, *VVIQ*, Marks, 1973). Meier and Rothen (under review) show similarly for (*n* = 24) grapheme-color synesthetes using the *Verbalizer-Visualizer Questionnaire* (VVQ; Kirby et al., 1988). And the same has been shown by Price (2009a) testing 12 sequence-space synesthetes, who outperformed controls in two questionnaires of visual imagery (*Subjective Use of Imagery Scale*, *SUIS*; Reisberg et al., 2003; and the visual [but not spatial] component of the *Object–Spatial Imagery Questionnaire*, *OSIQ*; Blajenkova et al., 2006). However, there has been some variation in the consistency of this self-reported superiority in imagery. Seron et al. (1992) failed to find higher than average visual imagery in 26 sequence-space synesthetes, using Paivio's *Individual Difference Questionnaire* (IDQ; Paivio, 1971). Furthermore, Spiller and Jansari (2008) failed to find a group difference in the VVIQ between six graphemecolor synesthetes and matched controls, although this may be due to low power.

In addition to poor consistency across synesthesia studies showing self-reported superiority in imagery, there has also been some variation in the extent to which this has translated into behavioral support. Despite no *self-reported* differences, grapheme-color synesthetes in Spiller and Jansari (2008) out-performed controls on a task that relied on mental imagery (imaging a grapheme within a divided circle and assessing which segment contained the majority of that image). Another study also found behavior suggesting synesthetes have superior imagery (e.g., sequencespace in 3D mental rotation; Simner et al., 2009). However, their group size was small (*n* = 5) and this effect was not replicated in a larger sample of nine synesthetes with a similar (sequence-space) variant (Rizza and Price, 2012). Nonetheless, Simner et al. tested somewhat unusual synesthetes with an unusually large array of synesthetic forms (e.g., for minutes, hours, days, months, numerals, letters, temperature etc.; mean = 7.0 forms; *SD* = 2*.*3). In contrast, Rizza and Price's subjects required only two forms to be selected for study. Superior performance in behavioral imagery tasks might therefore be tied to "superior" (more extreme) synesthesia. Nonetheless, a superiority in mental rotation for (*n* = 15) sequence-space synesthetes has now also been replicated by Brang et al. (2013; in 2D rotation), although across all three studies, we must conclude that the effect does not appear to be robust enough to survive repeated replication, especially where synesthetes might vary in their "strength."

Finally, Spiller and Jansari found no correlation between their self-reported imagery scores and a behavioral visual imagery task, suggesting that self-report may not always reflect performance in labbased imagery tests. Furthermore, Rizza and Price (2012) point out that mental rotation has not previously correlated with self-report in visual imagery, but rather *spatial* imagery, even though synesthetes do not report higher than average spatial skills (Rizza and Price, 2012; Meier and Rothen, under review). However, as Logie et al. (2011; p. 3072) "subjective reports [of mental imagery] do not always correlate with performance on mental imagery tasks *...* [perhaps because] people have poor insight into their mental operations when rating them."

From these somewhat conflicting findings we can draw the following conclusions: although grapheme-color and sequence-space synesthetes have reported higher imagery in self-report visual imagery questionnaires, this self-report advantage has not been found consistently across studies of grapheme-color synesthetes (e.g., Barnett and Newell, 2008 vs. Spiller and Jansari, 2008 and Seron et al., 1992) and nor has it consistently translated into behavioral superiority for sequencespace synesthetes in mental rotation (e.g., Simner et al., 2009; Brang et al., 2013 vs. Rizza and Price, 2012). Furthermore, there is conflict both across and within studies between the predictions of self-report imagery questionnaires and behavioral tests such as imaging graphemes (for grapheme-color synesthetes; e.g., Spiller and Jansari, 2008) and mental rotation (for sequence-space synesthetes; e.g., Rizza and Price, 2012 vs. Simner et al., 2009 and Brang et al., 2013). Where conflicts arise across different cohorts of synesthetes with the same variant, we might conclude either that low power in some studies may be to blame, or subtle differences in methodologies, or indeed that meaningful individual differences in imagery might exist. In other words, if conflicts across studies represent genuine differences in imagery across synesthetes, then imagery and synesthesia may be either partially or fully independent.

If synesthesia and imagery are only partially independent, we might conclude that whatever causes synesthesia could perhaps sometimes have repercussions for high imagery, or vice versa, but that high imagery is not a necessary component of synesthesia. This would certainly be compatible with findings in functional magnetic resonance imaging (fMRI) that synesthetic colors are experienced with at least some differences in regions compared to the colors within the mental imagery of non-synesthetes (Rich et al., 2006). Partial independence could also explain why some synesthetes show no significant advantage in tests/questionnaires of imagery: if synesthesia and high imagery only *tend* to be linked, some cases would exist where the synesthete's imagery was low. However the tendency itself would also explain why there is an overall trend toward finding higher imagery across the body of literature as a whole (e.g., Barnett and Newell, 2008; Spiller and Jansari, 2008; Meier and Rothen, under review).

But let us now consider the possibility that these skills may be *fully* independent. If correct, this theory would smartly explain why some studies find no imagery advantages in their synesthetes. However, it would also need to explain why some synesthetes do score highly in imagery, and perhaps more often than we might expect by chance alone. Barnett and Newell (2008) and others have suggested that high imagery is a general characteristic of synesthesia but I would suggest it may simply be a characteristic of those synesthetes that self-report for testing. What then is different about these individuals? One consistent quality of all participants in the imagery studies reviewed here is that all were fully aware of their synesthesia (hence their self-referral and/or affirmations of synesthesia), but importantly, this need not be true of all synesthetes. It is reasonable to propose that an individual's awareness of synesthesia might be heightened if individual differences in imagery make synesthetic colors highly image-able for some synesthetes (but not others—who instead might simply "know" their associations at some propositional level). In other words, I suggest that individual and randomly dispersed variations in mental imagery within the synesthesia population would lead to some synesthetes having high imagery and therefore highly imaged synesthetic colors. This in turn could lead to enhanced conscious awareness, and consequently a greater likelihood of selfreferral for scientific studies. This proposal is in opposition to the view that synesthetes overall are necessarily high imagers as a *sin qua non* and/or that synesthesia is nothing more than high mental imagery. (Indeed, this same selfreferral argument might hold equally over other hypotheses, such as whether synesthetes have enhanced memory; e.g., Rothen et al., 2012).

In summary, I have proposed that empirical evidence is currently equivocal for synesthetes as high imagers. I have also suggested that trends toward high imagery in published studies might arise from a recruitment bias in which high-imaging synesthetes are more likely to self-refer. If high imagery is neither sufficient nor necessary for synesthesia, this would explain why some studies fail to show superior imagery, and would also nicely capture why some synesthetes describe their colors (or other sensations) as simply "known." Put differently, for those without the ability to form a mental image, there would be no visual-like quality to their synesthetic sensations. This theory would also explain why other synesthetes do, in contrast, describe their colors as highly imaged, even eidetically so. My proposal sits alongside another theory, by Ward et al. (2007), that other differences within this same spectrum (e.g., surface vs. near-space projectors) could also fall out naturally from other *a priori* individual differences (this time from differences in spatial reference frames). In future studies it will be important to establish whether trends found in smaller populations are true even when larger numbers are assessed. It is also important to look at recruitment methods since self-referred samples might be expected to show stronger imagery than cohorts more randomly sampled. Finally, future studies might seek a standardized assessment of projector/associators, given that previous efforts to classify individuals have shown differences across labs

(c.f., Dixon et al., 2004; Ward et al., 2007) and even over time (Edquist et al., 2006). A useful classification would also extend to all forms of synesthesia, including variants (e.g., sequence-space) in which the projector/associator distinction has been recognized thus far only in self-report.

# **REFERENCES**


*Received: 04 June 2013; accepted: 06 August 2013; published online: 02 September 2013.*

*Citation: Simner J (2013) Why are there different types of synesthete? Front. Psychol. 4:558. doi: 10.3389/fpsyg. 2013.00558*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Simner. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

**REVIEW ARTICLE** published: 22 October 2013 doi: 10.3389/fpsyg.2013.00763

### *Myrto I. Mylopoulos <sup>1</sup> \* and Tony Ro2 \**

*<sup>1</sup> Philosophy Program, The Graduate Center, City University of New York, New York, NY, USA*

*<sup>2</sup> Department of Psychology, The City College and Graduate Center, Program in Cognitive Neuroscience, City University of New York, New York, NY, USA*

### *Edited by:*

*Roi C. Kadosh, University of Oxford, UK*

### *Reviewed by:*

*Dan Smilek, University of Waterloo, Canada Devin Terhune, University of Oxford, UK*

### *\*Correspondence:*

*Myrto I. Mylopoulos, Philosophy Program, The Graduate Center, City University of New York, 160 Convent Ave., New York, NY 10031, USA e-mail: myrto.mylopoulos@ gmail.com; Tony Ro, Department of Psychology, NAC 7/120, The City College and Graduate Center, Program in Cognitive Neuroscience, City University of New York, 160 Convent Ave., New York, NY 10031, USA e-mail: tro@ccny.cuny.edu*

Synesthesia is a fairly common condition in which individuals experience atypical responses (such as color experiences) in association with certain types of stimuli (such as non-colored letters). Although synesthesia has been described for centuries, only very recently has there been an explosive growth of systematic scientific examinations of this condition. In this article, we review and critically evaluate current methods for both assessing synesthesia and examining its psychological basis, including the "test-retest" procedure, online battery assessments, and behavioral experiments. We highlight the limitations of these methods for understanding the nature of this complex condition and propose potential solutions to address some of these limitations. We also provide a set of markers that aid in distinguishing synesthesia from other closely related psychological phenomena.

**Keywords: synesthesia, sensation, perception, imagery, memory**

# **INTRODUCTION**

Synesthesia is a condition in which individuals experience atypical responses to certain types of stimuli, in addition to the typical responses elicited by those stimuli. For example, a synesthete may perceive tastes when seeing certain shapes or might perceive colors when seeing achromatic letters. Synesthesia comes in many forms, covering a wide range of sensory interactions both cross-modally and within a single modality1 .

Over the years, a variety of research programs have emerged to better understand this condition. Some research on synesthesia is focused on determining how and why synesthetic associations are developed or acquired, and the patterns, if any, that govern them (Watson et al., 2010, 2012; Witthoft and Winawer, 2013). Other research has been devoted to determining the nature and characteristic features of synesthetic associations, raising questions, for example, regarding the types of stimuli that can trigger synesthetic responses (Ramachandran et al., 2002). But perhaps the most pressing questions pertaining to synesthetic associations concern their psychological *kind*. Are they genuinely perceptual, as many claim? In other words, are synesthetic responses the outputs of sensory modalities, exhibiting features that correspond to the sensory qualities of stimuli such as color, shape, and sound? Or are they purely cognitive responses? Are they mnemonic associations? Or are they some combination thereof?

While significant steps forward have been taken in synesthesia research in the past couple of decades, there is still further to go when it comes to establishing conclusive answers to these questions using objective measures. Some of the biggest strides are being made using neuroimaging techniques that are helping to reveal the neural basis of synesthesia. While we think these techniques are particularly promising for understanding synesthesia, in this short review, we focus instead on surveying and critically evaluating popular behavioral strategies for assessing and understanding synesthesia. We highlight the limitations of these strategies when it comes to both accurately assessing cases of synesthesia and examining the nature of the responses involved, and we propose potential solutions to some of these limitations along the way.

# **ASSESSING SYNESTHESIA**

In order to assess or diagnose a psychological condition, one must, of course, know what to look for. Although much about synesthesia is still unknown, and novel forms and varieties of the condition may manifest themselves, it will nonetheless be useful to highlight some characteristics of synesthesia that serve to distinguish it from other perceptual phenomena, such as visual imagery and certain forms of imagistic memory. There are three such characteristic features: (1) automaticity, (2) reliability, and (3) consistency <sup>2</sup> . First, there is ample evidence that synesthetic associations are *automatic* in nature (Lupiáñez and Callejas, 2006; Jarick et al., 2011). They are typically produced outside the intentional control of the individual and cannot be directly inhibited. The automaticity of synesthesia helps to distinguish it from

<sup>1</sup>For a comprehensive list, as well as a discussion of the different forms that synesthesia can take, see Novich et al. (2011).

<sup>2</sup>In addition to automaticity, reliability, and consistency most definitions of synesthesia either state or imply that the synesthetic response occurs consciously. However, as of yet there has been no empirical work done to determine whether these responses may sometimes occur unconsciously, similar to how subliminal perception occurs (though see Mattingley et al., 2001). As such, we do not include consciousness as a feature of synesthetic associations.

paradigm cases of mental imagery. While hearing a certain sound may lead one to imagine certain scenes or colors, for example, such visual imagery is typically under a significant degree of intentional control. One can usually start or stop imagining something at will. This is not to deny that synesthetic responses have qualities—for example, shapes and colors—that are similar to or even the same as those exhibited by mental imagery. However, synesthetic responses are, at the very least, distinguishable from mental imagery in virtue of their automaticity.

Second, it is typically the case that synesthetes *reliably* experience synesthetic responses when presented with triggering stimuli. When synesthetes come into perceptual contact with the triggering stimulus, their responses will be induced. These responses are not transient or inconsistent, though they may sometimes be when induced neuropharmacologically, for example, by psychoactive, hallucinogenic substances <sup>3</sup> . Indeed, synesthesia is often present from early childhood onwards (Cytowic, 2002). This helps to distinguish the condition from ordinary associations grounded in memory. One sometimes has vivid mnemonic imagery associated with specific smells or sounds, for example, and these memories may sometimes even arise automatically. But it is uncommon for these to be reliably generated and persistent throughout an individual's life in the way that synesthetic responses are reported to be.

Finally, although there is variability across synesthetes, synesthetic associations within an individual appear to remain relatively *consistent* over time in that the same types of stimuli (e.g., specific auditory tones) tend to elicit the same types of synesthetic responses (e.g., specific colors)4,5 (Dixon et al., 2000). This feature, too, helps to distinguish synesthetic associations from ordinary mental imagery, which displays more flexible associations. But as Simner (2012) argues, consistency may not be central to synesthesia in the way that many have supposed. The problem is that many of the tests used to assess synesthesia, as we will see, treat consistency as the main measure of synesthetic association. As a result, the synesthetes who are examined in the psychological literature are all those that exhibit consistency in their associations. This bias may have created an inflated sense of how common this characteristic really is among synesthetic associations, to the point where it has become a defining feature of the condition. We therefore restrict ourselves to the following claim: when consistency is present, this provides evidence in favor of the relevant association being a synesthetic one. However, when consistency is absent, this is *not* evidence *against* the relevant association being a synesthetic one.

Given that assessment strategies are used as diagnostic tools for establishing whether a given individual is a synesthete, it is important that they are able to establish that certain reported associations exhibit each of these characteristics and perhaps others that remain to be identified. One of the more widely accepted measures for determining whether one has synesthesia is the test-retest procedure, known as the "test of genuineness" (TOG) among synesthesia researchers. The TOG is considered by some to be the "gold standard" of synesthesia assessment. In this procedure, synesthetes are asked to indicate, either through verbal report or color swatch matching, the character of their synesthetic responses to certain stimuli, and then they are retested—often without warning—as much as a year or more later (Baron-Cohen et al., 1987; Cytowic, 2002; Asher et al., 2006).

The rationale is that if a person has synesthesia, their consistency of responses at the retest phase will be significantly higher than those without synesthesia, who are told to simply assign associations to the same set of stimuli. For example, Baron-Cohen et al. (1993) found that 92.3% of the reported synesthetes they tested gave consistent responses when they were retested one year later without warning, whereas this was true of only 37.6% of control subjects who were tested one week later with warning. This finding, and others like it, are made all the more impressive by the fact that synesthetic responses are often very precise, for example, sometimes with highly specific hues and shades in graphemecolor synesthesia (Eagleman et al., 2007). In cases where there is a high consistency of responses, therefore, it is likely the result of a stable association. The TOG can therefore provide strong evidence that reported associations possess one of the common characteristics of synesthesia. If one fails the TOG, however, in line with our earlier caveats, this alone is not evidence against a reported association being a synesthetic one, since it may simply be that the association is not consistent.

The TOG faces a number of shortcomings. Response consistencies over time do not by themselves indicate whether the associations in question are automatic, or whether they are instead, in some cases, being conjured up at will. Moreover, they do not by themselves establish the reliability of the associations, as typically only two instances of each association are observed—once in the testing phase and then again in the retest phase. In addition, this procedure provides no evidence as to whether the associations are cognitive, perceptual, mnemonic, or some combination of these. It might be that some individuals who are classified as having grapheme-color synesthesia may simply have better or stronger memories overall, as it has been reported in at least one case study (Smilek et al., 2002; see also Tammet, 2009, p. 73) and in a larger group study (Radvansky et al., 2011). They may thus have robust mnemonic associations of certain colors with certain letters or numbers, though these enhanced memory effects may be modest in size (see Yaro and Ward, 2007; Rothen et al., 2012) <sup>6</sup> . When taking the TOG, some individuals may simply be

<sup>3</sup>And there is a question, furthermore, about whether the experiences induced by drugs are appropriately classified as synesthesia. Hubbard (2007), for example, notes that congenital synesthetes have synesthetic experiences that typically involve simple colors and movements, while drug-induced hallucinations are often complex scenes or images.

<sup>4</sup>We use the term "association" loosely, to characterize whatever relationship holds between the stimulus trigger and the synesthetic percept. For some synesthetes, this relationship is much tighter than the term suggests, as illustrated by the following description of one synesthete: "The shapes are not distinct from hearing them—they are part of what hearing is. The vibraphone, the musical instrument, makes a round shape. Each is like a little gold ball falling. That's what the sound is; it couldn't possibly be anything else" (Cytowic, 2002, p. 69).

<sup>5</sup>This is not to rule out cases in which the concurrent is nonspecific or changes slightly or gradually over time (e.g., Eagleman et al., 2007).

<sup>6</sup>Smilek et al. (2002) and Tammet (2009) suggest that synesthetes have stronger memories due to their synesthesia, but the causal relationship may run in the opposite direction, which would be problematic for the TOG.

voluntarily associating in memory rather than automatically perceiving the color red with the letter "A," just as one may recall his or her grandmother's face when smelling freshly baked chocolate chip cookies. These associations, which may in some other cases be automatic, could have been learned early on in childhood, such as through television, books, toys, or refrigerator magnets (see Witthoft and Winawer, 2006, 2013) (**Figure 1**). Though voluntarily associating with superior memory is unlikely to account for all cases in which consistency is displayed, our point here is that it may be operative in enough cases that the reliability of the TOG is diminished.

We propose that this issue may possibly be addressed by adding to the TOG a further test and retest using stimuli that are not claimed by self-reported synesthetes to yield synesthetic associations, such as stimuli from a sensory modality that does not produce synesthetic responses. For example, a grapheme-color synesthete could be tested with auditory stimuli. The results of this test-retest for non-synesthetic associations could then be compared with those involving the reportedly synesthetic associations. This would control for the role that superior memory may play in generating the consistent results *within* each individual.

We also note some complications regarding the role of memory in synesthetic associations. We may distinguish among three types of sensory associations: (1) associations between a stimulus and a response that are generated by willful perceptual imagery (whether mnemonically based or not), (2) associations between a stimulus and a response that are generated by automatic perceptual imagery that are also mnemonically based, and (3) associations between a stimulus and a response that are generated by automatic perceptual imagery, but that are not mnemonically based. We are inclined not to view the first kind of association

as synesthetic, since it is not automatic in character. We take the third kind of association to be very similar to, if not the same, as non-imagery based synesthetic associations. There is a question, however, about whether the second type of association is properly understood as synesthetic association, involving as it does an indirect pathway between the stimulus and response—going through mnemonic systems and then back to perceptual systems, rather than directly through perceptual systems. We leave this question open for future theorizing.

Given the subjective nature of the responses generated by synesthesia, especially problematic are online assessments or batteries for this condition (e.g., Eagleman et al., 2007). In these assessments, individuals are given a battery of tests aiming to capture any synesthetic responses the online test takers may possess. These assessments suffer from two main flaws. The first is a problem for online assessments or batteries more generally, which is that the way in which one responds on such batteries cannot be directly monitored. Thus, subjects may use inappropriate strategies, may take the battery on several occasions (see Birnbaum, 2004), may not respond consistently just to get through the assessment faster, or may use notes or visual aids to produce their consistent responses. They may also use other cues, such as spatial ones to indicate a "perceived" color on a color bar or wheel that might increase accuracy and consistency in their responses. For example, if a color bar always has the same colors in the same order from the top to the bottom of a monitor, as in some synesthesia batteries, subjects may use position on the bar relative to the monitor, as well as other potential screen landmarks, to increase their precision<sup>7</sup> . The inability to verify whether subjects are responding appropriately to the assessment, while a general problem with online tests, is especially problematic in the case of assessing synesthesia, where there are already difficulties in classifying the condition, and in determining how to carry these assessments out so as to capture its main subjective features. In addition, there is a bias present in such assessments since many of those who take the test believe themselves to already have synesthesia<sup>8</sup> .

Another problem with the online battery is that, like the TOG, it cannot establish the automaticity or reliability of synesthetic associations. It is therefore not comprehensive in the way that a serious assessment of a psychological condition must be if it is to achieve reliable results. In addition, though frequency estimates of synesthesia are not commonly based solely on such measures, any that are (e.g., Novich et al., 2011 regarding the frequency of types of synesthesia) should be interpreted cautiously.

We acknowledge that there is a benefit in using online batteries in that they allow researchers to cast a much wider net than laboratory studies and to thereby collect data on a larger pool of individuals. But we propose that, in order for the larger set of data to guide research appropriately, these tests be used in

**colors, as illustrated in these toys.**

<sup>7</sup>We note that this may also be problematic in laboratory studies.

<sup>8</sup>Another potential issue is that web-based samples may not be representative in the way required (Skitka and Sargis, 2006). But this is a general issue in recruiting subjects for laboratory studies as well.

tandem with laboratory studies to validate the results collected online.

# **UNDERSTANDING SYNESTHESIA: PSYCHOLOGICAL PARADIGMS**

In this section, we explore three commonly used paradigms for examining the character of synesthetic responses. We stress some concerns with the current state of the literature, and offer some potential remedies.

# **STROOP TASK**

Early attempts to explore the nature of synesthetic associations made use of a variation of the Stroop task (Stroop, 1935; MacLeod, 1991), in which color naming responses are substantially delayed for words that spell out incompatible colors (e.g., naming the color red when the word "GREEN" is printed or displayed in red). In this variation, synesthetes are presented with graphemes that are written in a color that is either congruent or incongruent with the synesthetic colors that they report associating with these graphemes. So, for example, if a synesthete reports associating the letter "A" with the color red, she would be presented with a red "A" in the congruent condition, and an "A" in some other color in the incongruent condition. The subjects are then asked to name aloud the color of the grapheme. Synesthetes tend to respond more slowly on incongruent trials than on congruent trials compared with controls, for whom no effect on reaction times is found (Dixon et al., 2000; Mattingley et al., 2001; Lupiáñez and Callejas, 2006).

The Stroop task is a valuable tool for establishing the automaticity of synesthetic associations. The slower reaction times on the incongruent trials are interpreted as an inability to inhibit the interfering synesthetic response—and this is a key characteristic of automatic association. As this is not something that the TOG or online battery tests probe, we propose that a more comprehensive assessment of synesthesia may combine these tests with a Stroop-like test, so that the automatic dimension of the condition may be examined. In general, a successful assessment strategy for synesthesia may need to combine multiple tests in order to get a clearer profile of the condition within any given individual.

But, as others have noted (e.g., Hubbard et al., 2005; Gheri et al., 2008), these results do not help to establish that synesthetic responses are perceptual in nature since the interference could be due to purely cognitive rather than perceptual processes. Indeed, Elias et al. (2003) used this variant of the Stroop paradigm to compare the performance of a graphemecolor synesthete and a control who was trained on semantic color-number associations using a set of specific cross-stitch knitting patterns over a period of 8 years. For example, the number 5 might indicate that one should use red thread, and would thereby be strongly associated with the color red. They found that *both* the synesthete and the semantically trained subject were slower to name colors of numbers that were incongruent with their specific associations, suggesting that interference in this Stroop paradigm need not be perceptual in nature. Colizoli et al. (2012) similarly found Stroop interference effects in a training study using non-synesthetes. In light of these results, the Stroop paradigm, like the test-retest procedure, may be inappropriate for objectively establishing the perceptual character of synesthesia, at least when used in isolation.

There are two obstacles to this conclusion, however. The first is that other training studies using non-synesthetes have not found interference effects post-training (Rothen et al., 2011; Kusnir and Thut, 2012), so further research is needed in order to determine whether interference in the Stroop paradigm can be the result of learned, non-perceptual associations. Second, the magnitude of the interference found in studies with trained non-synesthetes may be smaller than that found for synesthetes in the Stroop task, though direct comparisons between studies is difficult because of the different stimuli and scales that were used (see also Elias et al., 2003, in which the trained non-synesthete displayed more interference on one of the tasks). This suggests a more subtle difference between synesthetic automaticity and that of non-synesthetic associations. Using neuroimaging along with the Stroop task will help to establish both the perceptual nature of synesthesia, if perceptual regions of the brain such as color areas are activated, as well as the automaticity of these effects, given that the Stroop task is a reliable measure of this feature.

# **VISUAL SEARCH TASKS**

Another popular paradigm used to understand synesthetic associations explores the relationship between synesthesia and attention. In a visual search task, one is presented with an array of stimuli and asked to respond to the presence or absence of a "target" stimulus that differs from the other "distractor" stimuli on the basis of some visual feature, such as color, orientation, or shape. When the target stimulus differs from the distractor stimuli with respect to just one of these features, it tends to grab attention, regardless of the number of distractors. One commonly offered explanation for this is that individual features of stimuli are processed automatically and in parallel by different parts of the visual system before they are bound together by attention, and thus seem to "pop-out" from the feature search visual array (Treisman and Gelade, 1980). Applying this to synesthesia, the rationale is that if a synesthetic color exhibits this pop-out effect during a visual search task on which the target differs from distractors on the basis of a unique feature, then it must be processed preattentively in the same way that veridical colors are in such tasks.

To test this hypothesis, Ramachandran and Hubbard (2001) conducted a pioneering study, where they briefly presented two projector synesthetes and forty controls with arrays of achromatic graphemes, where one group of graphemes formed an embedded shape (square, rectangle, triangle, or diamond). For example, the array might consist of a triangle of 2 s amongst background filled with 5s. The particular graphemes used to form the embedded shape were predicted to trigger synesthetic colors for the two synesthetes. Participants in the experiment were asked to identify the embedded shape. The two synesthetes were significantly better than controls at successfully identifying the embedded shape from a set of four options, presumably because the graphemes induced colors that popped out from the background. However, other studies seeking to replicate this result have not all been



successful, and some with even larger samples of synesthetes (see **Table 1**).

Studies using a more traditional visual search paradigm, in which participants are not asked to identify an embedded shape, but rather to locate a single target stimulus among distractors, have come up with mixed results. For example, Palmeri et al. (2002) found that the synesthete WO, who associates a specific color with the digit "2," was both significantly faster than the controls at spotting a target "2" among a set of "5" s and showed a significantly smaller effect of set size on search time. Laeng et al. (2004) also found superior performance in one synesthete using the same task. However, others have not arrived at the same findings (see **Table 1**). Until these contradictory results across studies can be sorted out, the general conclusion that synesthetes outperform controls on visual search tasks, either the embedded figure variant or the single target version, is not warranted.

A further concern is that it is not clear that the superior performance of synesthetes, in cases where it is indeed present, is due to synesthetic responses preattentively generating the pop-out effect. Ward et al. (2010) examined this very assumption. They conducted a study with 36 synesthetes, using Ramachandran and Hubbard's embedded-figure task, but this time including an assessment of the synesthetes' self-reports during the experiment. They found that, although synesthetes did tend to outperform the controls, most synesthetes reported that they did not experience synesthetic responses across the entire array during the task. Perhaps more importantly, the synesthetes that did experience synesthetic responses reported them as appearing piecemeal, rather than all at once, suggesting that these experiences depend on attention and perhaps other higher-order processes, and do not therefore pop-out like veridical colors in such tasks. As an example of a typical participant's report, Ward et al. offer this revealing quotation: "I definitely do NOT see all the colors in one go. I have to attend to the symbols/shapes or process them in some way, and then it has a color attributed to it. It's not like I could be looking somewhere else, and in the corner I see a shape made out of shapes of one color."

This verbal report is corroborated by a recent study by Nijboer et al. (2011) using a visual search task on which synesthetes and controls had to spot a single target digit among a set of distractor digits, for example, a "2" among a set of "5" s. An interesting feature of their experiment is that participants were required to make just a single, direct eye movement to the target, rather than being allowed to wander their eyes around the array. The target and distractors were either all gray (achromatic condition) or all colored. The target stimulus was always a different color from the distractors. Nijboer et al. found that synesthetes performed comparably to controls in both the chromatic and achromatic conditions. Importantly, accuracy decreased with increases in set size in the achromatic condition for both the synesthetes and the controls, indicating that no pop-out effect occurred for either group. Furthermore, there was no effect of set size on accuracy in the chromatic condition, indicating that pop-out *did* occur in this condition for both groups. This evidence seems to cast into doubt the claim that synesthetic responses are generated preattentively.

Perhaps a better way to assess whether attention is necessary for synesthetic responses is to determine whether perceptual load (e.g., Lavie and Tsal, 1994; Lavie, 1995) affects the occurrence of these responses. Perceptual load refers to the amount of perceptual information in a stimulus or a set of stimuli, with more attentional resources being required for processing when perceptual load is high. In particular, if attention is required for a synesthetic response to occur, then these responses should be measured more frequently under conditions of low perceptual load vs. high perceptual load. For example, synesthetes should report experiencing synesthetic responses more often when the triggering stimuli are presented under low load conditions.

# **PERCEPTUAL CROWDING EXPERIMENTS**

An additional paradigm used to examine synesthesia takes advantage of another well-known perceptual effect. A grapheme presented alone in the periphery is relatively easy to visually identify, whereas it is much more difficult to identify when it is flanked by distractor graphemes—an effect known as "crowding" (Flom et al., 1963; Bouma, 1970; Chung et al., 2001; Levi, 2008). However, identification in the flanking condition is made easier if the target grapheme is a different color than the distractors (Gheri et al., 2007).

Grapheme-color synesthetes have been tested using this paradigm in order to determine if synesthetic "colors" facilitate identification of a flanked target in the same way that regular colors do. Here, too, results have not been consistent. Ramachandran and Hubbard (2001)tested a synesthete on this task, who reported that his synesthetic "color" response was triggered, *but only on this basis* was he able to identify the flanked grapheme. In this case, the synesthetic "color" merely helped him to infer what the grapheme must have been, rather than helping him to consciously see the grapheme in the way that regular colors allow under the same conditions. Hubbard et al. (2005) also performed a crowding experiment using six synesthetes and found evidence for superior performance on the task over the control subjects in only three of the six synesthetes. Thus, the synesthetes as a whole did not exhibit significantly superior performance over controls for the task, as one might expect if synesthetic colors behaved like regular colors.

# **INDIVIDUAL DIFFERENCES: IS SYNESTHESIA "OUT THERE" OR "ALL IN THE HEAD"?**

What are we to make of the inconsistent results that plague the literature on synesthesia? A tempting solution is to appeal to individual differences among synesthetes to explain them. One distinction that is sometimes drawn in the literature is that between "projector" and "associator" synesthetes (Smilek et al., 2001; Dixon et al., 2004; Dixon and Smilek, 2005; Ward and Mattingley, 2006; Ward et al., 2007, 2010; Jarick et al., 2011). Projector synesthetes report experiencing their synesthetic responses (e.g., color in grapheme-color synesthesia) as being located "out there in space," whereas associator synesthetes report experiencing them as being present instead in their "mind's eye."

There are two problems with appealing to the associator/projector distinction to explain the varying performance of synesthetes in the tasks we have just discussed: (1) Most of the motivation for positing this distinction stems from subjective reports that are difficult to interpret, and (2) the objective methods used for evaluating the associator/projector distinction fall short of establishing it.

Although there is divergence among the subjective reports of synesthetes, which is in large part the basis for the associator/projector distinction, it is difficult to determine whether this is due to differing experiences or simply varying idiolects. Indeed, some of these reports have been found to be inconsistent with one another. Edquist et al. (2006) carried out a study in which one part required that fourteen grapheme-color synesthetes respond to a questionnaire that asked them to indicate their agreement with the following: "the color is out there in space," "the color is in my mind's eye," or "neither." Strikingly, the results of the questionnaire revealed seemingly conflicting reports within individual subjects. For example, two synesthetes agreed with the sentence "the color is out there in space," but on a separate questionnaire administered subsequently, these same two synesthetes also strongly agreed with the sentence "the color is in my mind's eye." And another three synesthetes who indicated their agreement with the sentence "the color is in my mind's eye" also agreed with the sentence "the color looks like it is on the page." These competing responses not only highlight the need for caution in using and interpreting subjective reports for the purposes of theorizing about synesthesia and the varieties thereof, but also raise general concerns regarding the classification of individuals as synesthetes based solely on such reports.

One way to mitigate these concerns might be to more carefully choose the wording involved in questionnaires intended to probe the experiences of synesthetes, such that ambiguities are avoided. For instance, the reason some synesthetes may have agreed that their synesthetic responses are in their "mind's eye" *as well as* being "out there on the page," is that despite experiencing their synesthetic responses as being out there on the page, they still retain the *belief* that they are not actually properties of external objects. And insofar as they hold this belief, they might be inclined to respond that the synesthetic response is in their "mind's eye." Their seemingly inconsistent responses to the questionnaire might have been an attempt to reflect this specific stance.

Given the difficulties involved in interpreting subjective reports, the most reliable source of evidence will likely come from objective measures that corroborate these reports—at least when they are sufficiently well-understood. The main behavioral task used for these purposes is the variation of the Stroop paradigm described earlier (Stroop, 1935; MacLeod, 1991). Those who identify as projector synesthetes tend to be faster at naming synesthetic colors of letters over their veridical colors, whereas those who identify as associator synesthetes tend to be faster at naming veridical colors over synesthetic colors (Dixon et al., 2004; Ward et al., 2007). Projectors also display larger Stroop interference effects than associators when it comes to their performance on the color-naming task.

From these findings, Dixon et al. (2004) suggest that synesthetic responses in the Stroop task are more automatic for projectors than for associators because external projections of color are more difficult to ignore than internal ones. Ward et al. (2007), having replicated the results of Dixon et al., elaborate on this by offering an explanation in terms of shifting spatial frames of reference. They suggest that in order to successfully complete the task, associators must attend to the grapheme located on the computer screen and then retrieve the corresponding color from a different spatial location (i.e., their "mind's eye"). This slows them down relative to projectors, who need only attend at or near the location of the grapheme to report their synesthetic colors. As for projectors being slower at naming veridical colors than synesthetic colors, Ward et al. suggest that the real and synesthetic color in the same location leads to competition between the two.

These interpretations, while perhaps promising, require much further support before they can be used to validate the projector/associator distinction. The results might be explained equally well by those reporting to be projectors simply having stronger grapheme-color associations than those reporting to be associators, rather than a different quality in their perceptual experience, or any perceptual experience whatsoever for that matter. A more solid result comes from van Leeuwen et al. (2011), who found in an fMRI study using dynamic causal modeling that the synesthetic responses of projector grapheme-color synesthetes were driven primarily by bottom–up processes via the fusiform gyrus, while those of the associator synesthetes were generated mainly by top–down processes via the superior parietal lobe (see also Rouw and Scholte, 2007). We propose that until more reliable evidence along these lines is gathered regarding the purported projector/associator distinction, this distinction cannot help to explain away the inconsistencies in the literature on synesthesia.

Another distinction that is sometimes appealed to in characterizing individual differences in synesthesia is that between "higher" and "lower" synesthesia (Ramachandran et al., 2002)<sup>9</sup> . Higher synesthetes are characterized as those individuals whose synesthetic responses may be triggered in the absence of an inducing physical stimulus, just by thinking about or imagining the relevant stimulus. Lower synesthetes are those for whom the presence of the inducing physical stimulus is required in order to experience a synesthetic response. Higher synesthetes are also characterized in a second way, as those individuals for whom the conceptual properties, and not merely the sensory properties, of a physical stimulus trigger their synesthetic responses. For example, a higher grapheme-color synesthete that associates the number five with the color red might experience a red sensation in response to the written word "FIVE," the roman numeral "V," a cluster of five dots as on a rolling die, and the symbol "5," all of which differ in their sensory properties but share in common the conceptual property of representing the number five (see, e.g., Ward and Sagiv, 2007). Lower synesthetes are thought not to have synesthetic experiences that are sensitive to the conceptual properties of stimuli to this degree10.

Some evidence for this distinction derives once again from subjective reports (e.g., Ramachandran and Hubbard, 2001). However, here too the distinction is not well-established by objective measures. Dixon et al. (2000) attempted to provide some experimental evidence for this higher category of synesthesia using a variant of the Stroop paradigm. They presented the grapheme-color synesthete, C, with arithmetic problems and asked her to calculate the solution. The solution was always a number that would typically elicit in C a report of a highly specific color experience. After calculating each sum, C then had to name the color of a patch that was either congruent or incongruent with that of the response triggered by the sum that she had just calculated. They found that C performed faster on congruent vs. incongruent trials, suggesting that, for her, having a thought about a specific number is enough to trigger the corresponding association.

Even if one accepts this study as conclusive evidence of the higher vs. lower synesthete distinction, it would not account for the inconsistent results discussed in the previous section. Higher synesthetes should consistently perform better than controls in these tasks. In addition, as with other Stroop-like tasks, C's performance could be explained without positing a perceptual synesthetic response. For example, it might be that C calculated the sum of two numbers and then rapidly or automatically recalled the color that went along with the sum rather than undergoing a perceptual experience.

Again, more evidence is required before such purported individual differences between synesthetes can be used to explain inconsistent results on perceptual tasks and classify different types of synesthetes. Thus, these various distinctions between different synesthesia subtypes may not help at this stage of inquiry.

We propose, however, that a promising way forward is to ensure that there is consistency across the various studies that examine the performance of synesthetes on different types of paradigms. For one, many of the studies do not use the same types of visual stimuli or displays. And for those that do, consistency and other control measures to rule out alternatives to synesthesia, such as superior memory, are often lacking. A standardized battery to assess synesthesia might therefore include a set of tests that are each precisely calibrated and validated to assess different defining characteristics of synesthesia. Once a standard set of criteria are used to correctly indentify and classify synesthesia, future studies will be in a better position to examine the nature of the condition.

# **CONCLUSIONS**

Based on this selective review that highlights some of the many challenges in synesthesia research, it is clear that the field needs more convincing evidence and better tools to assess and more fully understand the nature of this condition. As it stands, current methods leave many questions unanswered concerning the psychological kind or kinds under which synesthetic responses fall. And current assessment strategies leave too much room for error. Some of the more promising tools for better understanding and assessing synesthesia are ones that may measure it more objectively, such as those that measure brain responses during synesthetic responses using neuroimaging techniques. These tools could be used in tandem with some of the behavioral assessments and experimental paradigms we have surveyed here. Based on the

<sup>9</sup>Some have suggested that the higher vs. lower synesthete distinction maps onto the associator vs. projector synesthete distinction (Dixon et al., 2004; Dixon and Smilek, 2005) Indeed, it may be that there is only one distinction between higher/associator synesthesia and between lower/projector synesthesia. This way of distinguishing different types of synesthesias may be reasonable, but higher synesthetes sometimes have no inducing stimulus onto which to project. In these cases, higher synesthetes behave like associator synesthetes and would be classified differently. In general, it may be important to keep these pairs of synesthesia types conceptually distinct, since one is based on the nature of the inducing stimulus, and the other is based on the nature of the synesthetic experience, and higher synesthetes do, at least sometimes, respond to physical stimuli, as well as imagined or thought about stimuli. Moreover, Ward et al. (2007) administered a probing questionnaire to fourteen synesthetes that had been previously classified as either projectors or associators and found no evidence to suggest that the projectors were more likely to be lower synesthetes or that the associators were more likely to be higher synesthetes.

<sup>10</sup>It is important to note, however, that being a lower synesthete, in the sense just described, does not mean that the way a physical stimulus is conceptualized makes no difference to the induced synesthetic response. Ramachandran and Hubbard (2001) used ambiguous stimuli on two synesthetes that were classified as lower grapheme-color synesthetes because only Arabic numerals evoke colors for them, whereas other stimuli that represent numerosity do not. But they found that when presented with the words "THE CAT," in which they used identical, ambiguous symbols for the "H" and the "A" (Selfridge, 1955), both synesthetes reported seeing their usual colors for "H" and "A." The quality of their synesthetic experiences seemed to depend on whether they saw the identical symbol as an "H" or as an "A." This finding related to ambiguous stimuli has been reported by others as well (Myles et al., 2003). Thus, there is evidence to suggest that the way that a stimulus is conceptualized makes a difference for the induced photism, even in lower synesthetes. This somewhat blurs the boundary between higher and lower synesthesia as characterized by sensitivity to conceptual properties rather than just perceptual properties, since both seem to involve such sensitivity to some degree. Consequently, it may be more reasonable to primarily characterize higher synesthesia as that which may be triggered by thought or imagination alone and lower synesthesia as that which requires the presence of physical inducing stimuli.

current state of the neuropsychological literature on synesthesia, much further research is required before we have a clearer grasp on the underlying mechanisms involved with synesthesia.

# **REFERENCES**


standardized test battery for the study of synesthesia. *J. Neurosci. Methods* 159, 139–145. doi: 10.1016/j.jneumeth.2006.07.012


# **ACKNOWLEDGMENTS**

This work was supported in part by NIH grant MH64606 and NSF grants 0642801, 0847607, and 0843148 to Tony Ro.

manipulation of letter-colour pairings. *Neuropsychologia* 50, 3641–3652. doi: 10.1016/j. neuropsychologia.2012.09.032


analysis reveals subgroups. *J. Neuropsychol.* 5, 353–371. doi: 10.1111/j.1748-6653.2011.02015.x


the nature of subjective experience in grapheme-color synesthesia. *J. Neurosci.* 31, 9879–9884. doi: 10.1523/jneurosci.0569-11.2011


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 May 2013; accepted: 29 September 2013; published online: 22 October 2013.*

*Citation: Mylopoulos MI and Ro T (2013) Synesthesia: a colorful word with a touching sound? Front. Psychol. 4:763. doi: 10.3389/fpsyg.2013.00763*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Mylopoulos and Ro. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

**OPINION ARTICLE** published: 26 September 2013 doi: 10.3389/fpsyg.2013.00651

# Synesthesia, at and near its borders

# *Lawrence E. Marks 1,2\* and Catherine M. Mulvenna3*

*<sup>1</sup> John B. Pierce Laboratory, Department of Environmental Health Sciences, Yale School of Public Health, New Haven, CT, USA*

*<sup>2</sup> Department of Psychology, Yale University, New Haven, CT, USA*

*<sup>3</sup> Child Study Center, Yale University School of Medicine, New Haven, CT, USA*

### *\*Correspondence: marks@jbpierce.org*

### *Edited by:*

*Michael Banissy, Goldsmiths University of London, UK*

### *Reviewed by:*

*O. Deroy, University of London, UK*

**Keywords: synesthesia, cross-modal correspondence, cross-modal imagery, cross-modal memory, empathic perception, hallucination, Doppler illusion**

# **INTRODUCTION**

In synesthesia, experiences in one domain evoke additional experiences in another, as when musical notes or letters of the alphabet evoke colors. Both the domains and their pairings are diverse. Indeed, Day's (2013) recently tabulated 60 types of synesthesia, each referring to a different combination of inducing and induced domains. The domains conjoined through synesthesia may belong to different sense modalities, as in music-color synesthesia, but may also belong to the same modality: In grapheme-color synesthesia, seeing printed letters or numbers evokes color experiences.

In music-color and grapheme-color synesthesia, the inducing stimuli are *perceptual*, reflecting culture-specific categories (notes of the Western musical scale, letters of the alphabet) learned by synesthetes and non-synesthetes alike. Synesthesia may be triggered not only by sounds, tastes, smells, and pains, but also by more complex signals: words(e.g., Simner, 2007), emotional states (e.g., Ward, 2004), and even personalities (e.g., Novich et al., 2011). Analogously, the domains of synesthetic responses too can range widely. The composer Rimsky-Korsakoff "saw" the key of D-major as golden (Myers, 1914), while a graphemepersonification synesthete reported, "Ts are generally crabbed, ungenerous creatures" (Calkins, 1893; p. 454). Other phenomena, however, such as crossmodally evoked images or memories, are not typically considered examples of synesthesia.

In this article, we briefly describe half a dozen illustrative cases that border on traditional forms of synesthesia: cross-modal correspondence, cross-modal imagery, sensory (cross-modal) autobiographical memory, empathic perception, hallucination, and the Doppler illusion. Do any or all of the six constitute forms of synesthesia? The answer depends, we suggest, on the framework for characterizing synesthesia. Consequently, after describing the six phenomena, we sketch three frameworks that differ in how they characterize these phenomena relative to prototypical forms of synesthesia. Several investigators, taking different perspectives and coming to different conclusions, have already considered possible relations to synesthesia in three of the six: cross-modal correspondence (Martino and Marks, 2001; Deroy and Spence, 2013); cross-modal imagery (Craver-Lemley and Reeves, 2013; Spence and Deroy, 2013); and empathic perception (Fitzgibbon et al., 2010; Rothen and Meier, 2013).

# **SIX AT THE BORDERS: SYNESTHESIA'S FAR AND NEAR KIN?**

### **CROSS-MODAL CORRESPONDENCE**

Cross-modal correspondences pervade not only several forms of traditional synesthesia but also, importantly, the experiences of individuals typically deemed non-synesthetic (Marks, 1975, 1978; Spence, 2011). Even non-synesthetes perceive high-pitched vs. low-pitched sounds to resemble bright vs. dark colors—the resemblances evident in various tasks of cross-modal comparison (Marks, 1975; Ward et al., 2006). Where music-color synesthetes *see* brighter colors in highpitched notes (e.g., "gold, yellow and white moving . . . like a rippling stream": Mulvenna and Walsh, 2005; p. 399), people lacking the induced *qualia* of synesthesia nevertheless *recognize* cross-modal similarities.

Cross-modal correspondences often reflect alignments between bipolar dimensions, such as higher pitch being associated with greater lightness, greater brightness, higher vertical location, and smaller size (e.g., Karwoski et al., 1942; Wicker, 1968; Marks, 1974, 1989; Ward et al., 2006). Several auditory-visual correspondences reveal themselves in young children (Marks et al., 1987; Mondloch and Maurer, 2004) and infants (Lewkowicz and Turkewitz, 1980; Walker et al., 2010; Haryu and Kajikawa, 2012), as well as in denizens of disparate cultures: Members of a remote, semi-nomadic, preliterate desert-tribe in southern Africa, having virtually no contact with Western culture, nevertheless overwhelmingly matched lighter gray colors to higher-pitched tones—thereby revealing pitch-lightness correspondence (Mulvenna, 2012).

The tendency for non-synesthetes to perceive similarities between experiences in different domains, despite the absence of secondary *qualia*, has been called "synesthetic thinking" (Karwoski et al., 1942) and "weak synesthesia" (Martino and Marks, 2001), consistent with the notion that cross-modal correspondence reflects general perceptual and cognitive processes. Further, by capitalizing on cross-modal correspondences, synesthesia too presumably capitalizes on these general processes of perception and cognition (e.g., Karwoski et al., 1942; Marks, 1978; Ward et al., 2006).

## **CROSS-MODAL IMAGERY**

Compared to cross-modal correspondence, which can lack induced *qualia*, cross-modal imagery is nearer, phenomenologicaly, to prototypical synesthesia. In cross-modal imagery, as in synesthesia, stimulation in one modality may arouse mental images in another, although cross-modal imagery exhibits greater voluntary control (e.g., Karwoski and Odbert, 1938). Synesthetic responses commonly arise *automatically*, without requiring effort and being under relatively little control (e.g., Mattingley et al., 2001). By comparison, some non-synesthetes can voluntarily conjure up images, for example, imagining colors while listening to music (Karwoski et al., 1942). In some instances, there may be an especially intimate connection between visual imagery and prototypical sound-color synesthesia. Karwoski and Odbert (1938) inferred that a small subset of their subjects experienced visual imagery that could be modulated by music—a phenomenon that seems more automatic (less voluntary) than typical visual imagery, albeit less automatic (more voluntary) than synesthesia. Perhaps music-modulated imagery bears an especially close connection to traditional music-color synesthesia.

# **SENSORY AUTOBIOGRAPHICAL MEMORY (PROUST PHENOMENON)**

Marcel Proust (1922) famously described the floods of detailed, sensory memories from childhood evoked by tasting a teasoaked *madeleine*. Sensory, autobiographical memory of this sort has been dubbed the "Proust phenomenon" in honor of the eponymous author. In the Proust phenomenon, odors or flavors in particular evoke strong sensory-based memories of associated events experienced in childhood (Chu and Downes, 2000). Proustian memory resembles traditional synesthesia, but also differs from it—resembling synesthesia in the automatic manner in which sensory experiences evoke memory images, but differing in the episodic character of the memories. In this regard, the sensory *qualia* of traditional synesthesia seem more "semantic" than "episodic."

# **EMPATHIC PERCEPTION: PAIN, TOUCH, COUVADE SYNDROME**

In several respects, empathetic perception strongly resembles synesthesia. In empathic pain, seeing or hearing evidence of another person's pain or discomfort produces analogous pain or discomfort (e.g., Jackson et al., 2005). Similarly, seeing another person being touched may produce an analogous tactile sensation—often called "mirror touch" (e.g., Banissy and Ward, 2007). Although the inducing stimuli come from another modality—typically, vision or hearing the mechanisms underlying empathic perception presumably rely on an underlying within-domain equivalence: where implicitly-recognized sensations evoke sensory experiences of the same or similar kind, perhaps through merging constructs of "self" and "other."

Possibly related to empathic pain is the *couvade syndrome*, which refers to a set of empathic symptoms (such as nausea, vomiting, and abdominal pain) reported by the partners of pregnant women. The couvade syndrome appears to be fairly common, having a reported prevalence of about 22% (Lipkin and Lamb, 1982), roughly five times that of traditional synesthesia (Simner et al., 2006).

A feature that distinguishes empathic perception from traditional forms of synesthesia is the very characteristic that makes empathy empathic—the intrinsic equivalence between the emotional qualities of the inducing and induced experiences. In traditional forms of synesthesia, however, as when sound evokes visual color or shape, the inducing and induced sensations not only reflect different domains but are also usually related more abstractly, even "metaphorically" (Rothen and Meier, 2013).

# **HALLUCINATION**

A hallucination is a "percept-like experience which (a) occurs in the absence of appropriate stimulus, (b) has the full force or impact of the corresponding (real) perception, and (c) is not amenable to direct and voluntary control by the experiencer" (Slade and Bentall, 1988; p. 23). Several of these attributes also characterize synesthesia. Hallucinations may involve any of the senses (Ohayon, 2000) and are easily distinguished as "perceptions not confirmed by others" (Ohayon, 2000; p. 154). Some types of hallucinations, though not all, may fall near the borders of synesthesia. Thus, synesthetic experiences commonly include colors and shapes (Day, 2013). Analogously, "simple hallucinations," often triggered by migraines or hallucinogens (Ermentrout and Cowan, 1979), commonly include simple recurring shapes and patterns, or 'form constants (Klüver, 1966), which apparently reflect patterns of neural activation in visual cortex (Ermentrout and Cowan, 1979).

No longer considered explicitly pathological, hallucinations are now generally treated as independent perceptual phenomena (Romme and Escher, 1989; see Strauss, 1969). Auditory verbal hallucinations (hearing voices), for instance, occur in about 13% of adults in the general population (Beavan et al., 2011), and their presence does not correlate significantly with psychopathology (Johns et al., 2002; Sommer et al., 2010). Although the strongest predictor of psychopathology in hallucinations is distress over their content or possible basis (Romme and Escher, 1989; Chadwick and Birchwood, 1994; Beavan et al., 2011), distress is rarely associated with synesthetic experience.

# **THE DOPPLER ILLUSION**

Day's (2013) table listing 60 types of synesthesia includes only one type that is explicitly intra-modal, namely, graphemecolor synesthesia. We note here another phenomenon in which sensory experiences in one domain induce experiences in the same modality, the "Doppler illusion," reported by Neuhoff and McBeath (1996): When a tone increases continuously in intensity over time (as though a sound-emitting source were approaching at constant velocity) but maintains a constant sound frequency, observers nevertheless report hearing the tone's pitch to increase as loudness increases. Neuhoff and McBeath dubbed the illusory increase in pitch the "Doppler illusion" because a sound source approaching at constant velocity will produce, at the observer's location, an elevated, albeit constant, sound frequency (the physical Doppler effect). But might we not also call the Doppler illusion a case of intra-modal (loudness-pitch) synesthesia?

# **SYNESTHESIA: CONTINUOUS, DISCRETE, PLURALISTIC?**

If, as the term implies, synesthesia is first and foremost a "conjoining of experiences," then one might construe several or perhaps all six of our cases as examples of synesthesia. If, on the other hand, synesthesia is defined more narrowly, for example, by requiring it to include *qualia* and to arise automatically and consistently, then fewer cases would make the cut. Recently, the first author outlined three theoretical frameworks—monism, dualism, and pluralism—that, in different ways, characterize how synesthesia could relate to borderline perceptual and conceptual phenomena like the six just described (Marks, 2011, 2012).

Synesthetic *monism* refers to the notion that synesthesia may appropriately be considered a spectrum or continuum. Using this framework, traditional forms of synesthesia, such as music-color and graphemecolor, serve as prototypes, residing at the high end of the continuum, with weaker forms, such as cross-modal correspondence, residing toward the low end. In the present examples, music-modulated imagery, hallucinations, empathic perceptions, Proustian evoked memories, and the Doppler illusion might lie at various loci between the ends of the synesthetic spectrum—although the differences amongst them suggest that the hypothesized spectrum is multidimensional.

The other two frameworks both distinguish sharply between synesthesia and non-synesthetic forms of perception and conception. *Dualism* posits a simple dichotomy between the two categories, with synesthesia incorporating both traditional forms (e.g., soundcolor, grapheme-color, word-flavor, taste-shape) and others (e.g., graphemepersonification, ordinal sequence-spatial sequence), and non-synesthesia incorporating cross-modal correspondence, cross-modal imagery, and hallucination. How to categorize other borderline examples, such as empathic perception, Proustian memory, and the Doppler illusion, however, is less clear.

Like dualism, the third framework, *pluralism*, explicitly distinguishes synesthetic from non-synesthetic experiences, but rests on the additional assumption that synesthesia is well characterized as (appropriating James's, 1890; p. 224, expression) "a teeming multiplicity." Like monism, synesthetic pluralism recognizes that some forms of synesthesia, such as music-color, are better exemplars than are others, such as empathic perception and the Doppler illusion. In the pluralistic view, however, the broad category of synesthesia itself contains a cornucopia of distinct subcategories, lacking common denominators but perhaps linked one to another along the lines suggested by Wittgenstein's (1953) notion of family resemblance.

Critical, in our view, to choosing amongst frameworks is characterizing the role of phenomenal experience in defining synesthesia; many investigators judge this role to be significant (see, e.g., the exchange among Cohen Kadosh and Terhune, 2012; Eagleman, 2012; Simner, 2012a,b). Monism in particular relies substantially on the notion that phenomenal experience plays a central, and ineluctable, role in characterizing synesthesia. Alternatively, jettisoning phenomenology may be conducive to pluralistic frameworks that rely on mechanism-based distinctions amongst multiple forms of synesthesia. And jettisoning phenomenology may be especially conducive to dualistic frameworks that rely on a mechanismbased distinction between synesthesia and borderline phenomena—perhaps akin to distinguishing mechanistically between rhinovirus-induced sniffles and pollen-induced seasonal nasal allergies.

But we ask, Are phenomenal experiences (*qualia)* analogous to sneezes?

# **REFERENCES**


daysyn*.*com/types-of-syn*.*html (Accesed May 17, 2013).


*Applications*, eds S. Lacey and R. Lawson (New York, NY: Springer), 157–183.


*Received: 13 July 2013; accepted: 02 September 2013; published online: 26 September 2013.*

*Citation: Marks LE and Mulvenna CM (2013) Synesthesia, at and near its borders. Front. Psychol. 4:651. doi: 10.3389/fpsyg.2013.00651*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Marks and Mulvenna. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Training, hypnosis, and drugs: artificial synaesthesia, or artificial paradises?

### *Ophelia Deroy1 \* and Charles Spence2*

*<sup>1</sup> Centre for the Study of the Senses, School of Advanced Study, University of London, London, UK*

*<sup>2</sup> Department of Experimental Psychology, Crossmodal Research Laboratory, University of Oxford, Oxford, UK*

### *Edited by:*

*Roi C. Kadosh, University of Oxford, UK*

### *Reviewed by:*

*Beat Meier, University of Bern, Switzerland David Luke, University of Greenwich, UK*

### *\*Correspondence:*

*Ophelia Deroy, Centre for the Study of the Senses, School of Advanced Study, University of London, Senate House, R 276, Malet Street, WC1E 7HU London, UK e-mail: ophelia.deroy@sas.ac.uk*

The last few years have seen the publication of a number of studies by researchers claiming to have induced "synaesthesia," "pseudo-synaesthesia," or "synaesthesia-like" phenomena in non-synaesthetic participants. Although the intention of these studies has been to try and shed light on the way in which synaesthesia might have been acquired in developmental synaesthestes, we argue that they may only have documented a phenomenon that has elsewhere been accounted for in terms of the acquisition of sensory associations and is not evidently linked to synaesthesia. As synaesthesia remains largely defined in terms of the involuntary elicitation of conscious concurrents, we suggest that the theoretical rapprochement with synaesthesia (in any of its guises) is unnecessary, and potentially distracting. It might therefore, be less confusing if researchers were to avoid referring to synaesthesia when characterizing cases that lack robust evidence of a conscious manifestation. Even in the case of those other conditions for which conscious experiences are better evidenced, when training has been occurred during hypnotic suggestion, or when it has been combined with drugs, we argue that not every conscious manifestation should necessarily be counted as synaesthetic. Finally, we stress that cases of associative learning are unlikely to shed light on two highly specific characteristic of the majority of cases of developmental synaesthesia in terms of learning patterns: First, their resistance to change through exposure once the synaesthetic repertoire has been fixed; Second, the transfer of conditioned responses between concurrents and inducers after training. We conclude by questioning whether, in adulthood, it is ever possible to acquire the kind of synaesthesia that is typically documented in the developmental form of the condition. The available evidence instead seems to point to there being a critical period for the development of synaesthesia, probably only in those with a genetic predisposition to develop the condition.

**Keywords: induced synaesthesia, synaesthesia, crossmodal correspondences, pseudo-synaesthesia, learning, genetic basis, drugs, hypnosis**

# **INTRODUCTION**

In *Prometheus, Poem of Fire* (1910), Scriabin attempted to communicate to the audience the synaesthetic experience of colored musical keys that he claimed to have enjoyed for most of his life. A color organ was used to project colored lights during the concert (Peacock, 1985; see also Gawboy and Townsend, 2012, for a recent performance given at Yale). Critics left the first performance of Scriabin's work stressing that the lights had "no possible connection to the music," and merely "served to divert the senses of the audience from a too concentrated attention on the music" (Clarence Lucas quoted in Hull, 1927, p. 227). Several other artists have used their synaesthetic experiences as material for their work, but always leaving the public short of having a first-person experience of the kind of involuntary "sensory blending" that only they seem to have enjoyed (Harrison, 2001). But could training succeed where art has so far failed?

Synaesthesia is mostly known as a relatively rare developmental condition, present in approximately 5% of the adult population. The condition appears to have a genetic basis and runs in families (e.g., Asher et al., 2009; Brang and Ramachandran, 2011). For synaesthetes, the presentation of a stimulus in a certain modality, or rather its recognition, elicits an additional atypical sensory concurrent in another unstimulated sensory channel (Cytowic, 1989/2002; Baron-Cohen and Harrison, 1997; Day, 2005; Cytowic and Eagleman, 2009). Synaesthetes have been shown to exhibit significantly different performance, as a group, in a series of behavioral tests, such as the speeded congruency test presented by Eagleman et al. (2007), designed to capture the involuntary and robust elicitation of conscious concurrents by specific inducers. The repertoire of inducer-concurrent pairings—for instance, the precise shade of color elicited by different musical notes—is usually not determined before an individual reaches the age of 9 years of age, after which time it appears to remain surprisingly consistent over the rest of the individual's lifetime (e.g., Simner et al., 2009; Niccolai et al., 2012a). Some individuals claim to have lost their synaesthesia after their teenage years, but, of all the adult synaesthetes who have been tested in contemporary research, most remember having it since childhood. Recently though, researchers have started to talk more frequently about the possibility of "acquiring" synaesthesia later in life (Ward, 2007), following short-term training, but also as a result of hypnosis, drug-use, or the extended use of certain sensory-substitution devices that systematically convert visual images into patterns of sound designed for the blind (see also Auvray and Farina, in press for discussion). As the innate character of synaesthesia is itself controversial (see Deroy and Spence, 2013a), the label "acquired synaesthesia" is potentially misleading: After all, all synaesthesia might be acquired. This is why here we consider that the real question is that of *artificially* induced synaesthesia—a label which maintains the contrast with the spontaneous forms of synaesthesia that have been documented to develop in children, and encompass cases of training, including following the use of certain conversion devices, as well as drugs or hypnosis. Possible artificially induced synaesthesia, in turn, can be distinguished from what might be better called late emerging synaesthesia that is supposed to occur spontaneously after certain kinds of brain damage or sensory deprivation (e.g., Lessel and Cohen, 1979; Jacobs et al., 1981; Bender et al., 1982; Vike et al., 1984; Rizzo and Eslinger, 1989; Harrison and Baron-Cohen, 1996; Armel and Ramachandran, 1999; Ro et al., 2007; see also Ward, 2007; Afra et al., 2009). The current article will focus on the possibility of an artificial induction of synaesthesia through associative learning. As associative learning has sometimes been complemented by drugs, or substituted for sessions of hypnotic suggestion, we will also consider the role of hypnosis and drugs, but only as far as they relate to training (see Auvray and Farina, in press; Terhune et al., in press, for broader considerations on these points). We are faced here with an alternative: have training, hypnosis and drugs been capable of artificially inducing synaesthesia or is it a mere ungrounded hope—an artificial paradise, as mentioned in the title, in reference to the famous essay where Baudelaire describes the extraordinary blending of sensory experiences he and his friends had when under the influence of drugs? (see also Gautier, 1843/1962).

Testing for the possibility of inducing synaesthesia in nonsynaesthetes through training is not an end in-and-of-itself. The majority of the studies that have reached for the "acquired synaesthesia" label, either as an artificially induced form of the condition, or as a result of its emergence later in life have seemingly intended to shed some light on the processes by which a few adults end-up as synaesthetes. Given the questions that still surround the neurological (e.g., whether it comes from "*extra wires or altered function*," Bargary and Mitchell, 2008, p. 335) and developmental origins of synaesthesia (whether it is innate or due to over-learned associations, see Baron-Cohen, 1996; Maurer and Mondloch, 2005; Cohen Kadosh et al., 2009; Holcombe et al., 2009; Deroy and Spence, 2013a), and given the higher and higher prevalence granted to the phenomenon (up to 1 on 10 once spatial and non-sensory cases are included, see Sagiv and Ward, 2006; see also Simner et al., 2006), this is a legitimate and important scientific query. However, we want to question whether the evidence collected in any of the artificially induced cases actually sheds light on the fundamental issues raised by the occurrence of synaesthesia in adults.

In the next section, we critically evaluate the problem with the series of recent studies (see **Table 1** for an overview) which, despite the negative results obtained, remain ambivalent regarding the relation between these results and synaesthesia, and continue to believe that more, or better, training could perhaps one day bridge the gap between synaesthetes and non-synaesthetes. Most of them have, either explicitly or implicitly, suggested that behavioral results were sufficient to subsume artificially induced cases within the category of synaesthesia—be it as "synaesthesia" (Gebuis et al., 2009a), "pseudo-synaesthesia" (Howells, 1944; Verhagen and Engelen, 2006; Colizoli et al., 2013a), or "implicit synaesthesia" (e.g., Knoch et al., 2005). We would like to question the relevance of attaching the label "synaesthesia," in any of its guises, to the results that have been obtained to date here. On the contrary, we argue that, if anything, it is the failure to induce a conscious synaesthetic concurrent which makes these studies interesting, as these examples help to build a contrast with synaesthesia. Indeed, what this failure suggests is that conscious concurrents come from a very rare predisposition and will not be found in the general population, and/or that there is a critical period (e.g., Daw, 2003) during which associative learning can result in the involuntary elicitation of conscious synaesthetic concurrents. We then move to a critical evaluation of a number of other ways in which, over the years, researchers have claimed that synaesthesia can be trained, namely, as a result of hypnotic suggestion or by introducing drugs during training. Here, although the occurrence of conscious concurrents is better evidenced, it is important to remember not to assimilate any apparently arbitrarily-induced conscious experiences to synaesthesia—if the latter is supposed to be distinct from involuntary crossmodal mental imagery and/or hallucinations (Spence and Deroy, 2013a).

In conclusion, we stress that the inflation of new kinds of "synaesthetic" phenomena witnessed in the last decade is most probably not the right way in which to establish the scientific importance of the condition. On the contrary, we believe that this ambivalent extension potentially distracts researchers from singling out what is supposedly special about the developmental condition initially and centrally targeted by the term.

# **EIGHT DECADES OF FAILURE RECENT FINDINGS**

Several recent behavioral studies have attempted to induce synaesthesia (or, as we shall see later, "synaesthesia-like behavior") in non-synaesthetic participants as a result of training. It is worth describing these studies in some detail here.

In the first of the recent spate of acquisition studies, Meier and Rothen (2009) repeatedly exposed 20 non-synaesthetic participants to four letters (A, B, C, and D) each presented in one of four distinctive colors (red, green, yellow, and blue, respectively). The participants were presented with a total of 480 letters during training: In half of these presentations, the letters were presented in the "associated" color, while, in the remainder of the trials, they appeared in one of the three other colors. The participants had to make a speeded YES/NO discrimination response to indicate whether the color of each letter was appropriate or not.


**Table 1 | Summary of studies that have tried to induce an association between sensory features or dimensions comparable to synaesthesia.**

*N* = *Number of participants taking part in study.*

This training took place for 10 min a day over 7 consecutive days. On the eighth day, Meier and Rothen were able to demonstrate what they called a "synaesthetic Stroop effect"—that is, a significant difference in the reaction time (RT) and accuracy of participants' responses when trying to identify the color of a letter presented in a manner that was either congruent or incongruent with the intramodal associations that they had been exposed to over the course of the preceding week. Interestingly, no evidence of a conditioned synaesthetic response was demonstrated. This was measured in a synaesthetic conditioning test in which a grapheme that had been reliably associated with a particular color during training was tested to see whether or not it would, by itself, elicit a significant galvanic skin response (GSR) following a period of conditioning in which a color patch (of that particular color) had repeatedly been associated with the presentation of a loud and startling sound. By contrast, a group of thirteen graphemecolor synaesthetes exposed to a very similar training regimen in an earlier study did indeed exhibit a robust conditioning response (Meier and Rothen, 2007).

In a second study, Kusnir and Thut (2012) attempted to teach their participants a number of specific letter-color associations under somewhat less contrived experimental conditions: Specifically, the participants were not instructed about the goal of the study; What is more, the training phase of Kusnir and Thut's experiment consisted of participants performing a visual search task in which they had to make speeded left/right spatial discrimination responses to indicate the side of the screen on which a particular target letter had been presented. There were seven possible letters in total and three were presented symmetrically on either side of the screen on each trial. Unbeknownst to the participants, however, certain of the letters appeared somewhat more frequently in a given color than in any of the others <sup>1</sup> . Evidence that the participants had learned (or internalized) the letter-color associations that they had been exposed to during the training phase of the experiment was provided by the demonstration that color-letter congruency once again affected participants' performance in a "synaesthetic Stroop task" similar to that used by Meier and Rothen (2009). Interestingly, Kusnir and Thut (2012) went on to report interference between the learned (associated) color and the real color in which the letter was actually presented that depended upon their relative positions in color space (i.e., whether they constituted opponent vs. non-opponent colors). According to the authors, this particular aspect of their results suggested that the automatic formation of grapheme-color associations had taken place on a perceptual rather than merely on a conceptual level (Kusnir and Thut, 2012). However, it should be remembered here that opponent colors might also be more easily conceptualized as distinct, and thus, help to build two poles to which to refer the curved vs. angular letters that were the

<sup>1</sup>In fact, the pairings between the letters "H" and "U" and the colours blue and red were not systematic but rather just statistically more frequent, and inserted among other colour-letter pairings that were varied randomly (i.e., the letters were uncorrelated with the colour in which they happened to be presented) on a trial-by-trial basis. Note also that the pairings varied from one participant to the next (H-blue or red, U-blue or red).

target of the study (see, for instance, the discussion in Spector and Maurer, 2011; Simner, 2012; see also Proctor and Cho, 2006).

Finally, the participants in the most recent acquisition study, this time conducted by Colizoli et al. (2013a), had to read a book (or, in a few enthusiastic cases, chose to read up to five books) in which the letters had been colored consistently (A, E, S, T in either red, orange, green, and blue; high frequency letters paired with high-frequency colors). On average, the participants read nearly 500,000 colored characters over a 2–4 week period. The specific color-letter matches used in this experiment were based on each participant's stated preference, as established at the start of the study. After having read at least one book, the participants once again exhibited behavioral congruency effects in a "synaesthetic" version of the Stroop task. They also failed to exhibit any synaesthetic release from crowding in another visual task, arguing against the existence of any low-level visual effects (cf. Cavanagh, 2001; Vatakis and Spence, 2006; Tyler and Likova, 2007).

Colizoli et al. (2013a) argued that they succeeded in inducing genuinely synaesthetic associations based on the replies given by participants to two of the 19 questions that they were asked relating to their experience of reading in color. However, it is important to note that participants' mean response to one of the questions ["*I am experiencing color when I see certain letters (in addition to the color of the text*)"] that perhaps comes closest to tapping the existence of a sensory concurrent elicited responses near the bottom of the response scale (*M* = 1*.*5), while their responses to the other ("*I am experiencing color when I think about certain letters*") fell close to the midpoint of the 5-point Likert scale (*M* = 2*.*5). What's more, in the absence of answers to the very same questions prior to training (cf. Finke and Schmidt, 1977; Broerse and Crassini, 1984), or measures of any individual differences in the vividness of visual imagery (e.g., Marks, 1973; Craver-Lemley and Reeves, 2013; Lacey and Lawson, 2013), it is difficult to know how much these responses were actually a result of the training regimen that the participants had been exposed to, vs. a baseline tendency in certain of the participants to imagine vividly.

# **OLDER STUDIES**

Given the long history of research on the topic of synaesthesia (see Galton, 1880, for what may well be the first popular scientific study on the condition), it is perhaps inevitable, that older references are, on occasion, simply forgotten. However, it is more than a little unfortunate that not one of the just-mentioned acquisition studies made anything more than merely a passing reference (and that only in Meier and Rothen, 2009) to the much older literature on the acquisition of "(pseudo)synaesthesia" as a result of conditioning or associative learning where more training (than in these recent studies) was, on occasion, given. Indeed, a number of researchers attempted to induce synaesthesia in non-synaesthetes back in the 1930's and 40's: Here, we are thinking particularly of the work of Kelly (1934) and Howells (1944). These early researchers attempted to establish (cross-sensory) synaesthesia, described by Howells (1944, p. 316) as a "tendency of auditory stimuli to arouse simultaneous sensations or images of color as well as of sound" in participants using a methodology that is conceptually similar to that used in the more recent studies.

So, for instance, one group of participants in Kelly's (1934) study was repeatedly exposed to a consistent mapping between each of the 8 notes constituting the C-major diatonic scale with a different color (C—White; D—Red; E—Orange; F—Yellow; G—Green; A—Blue; B—Violet; C1—White). The majority of participants were then exposed to 320 repetitions of these stimulus pairs over a period of 7 weeks. The participants were subsequently presented with the notes and asked about any spontaneous colors that they saw, and which color matched each of the notes. Kelly (1934) argued that he had not been able to induce synaesthesia (what he called artificial chromaesthesia) because of the lack of stable conscious sensory concurrents on the part of his participants—although some of those taking the mescal reported having other kinds of conscious experiences. As he put it: "Without question, the results turned out negative. They are so distinctly so that the writer has no hesitancy in concluding that it is impossible to produce chromoaesthesia in normally non-synaesthetic adult subjects by the technique of conditioned response" (Kelly, 1934, p. 336).

Now Kelly (1934) cannot be faulted for not trying here: There can't be many published studies in the field of experimental psychology that include both the unexpected firing of a pistol (blank) from close to the participant during the course of the experiment (in order to see whether putting them into a nervous state, by giving the participant "bad fright," would increase the likelihood of synaesthesia) and the use of peyote(Lophophora williamsii, a known hallucinogen) to try to induce the condition, albeit temporarily! Now while the firing of the pistol cartridge may well have given rise to "a most admirable emotional disturbance" (p. 332), and the peyote to "gorgeous arrays of colored visions" (p. 335), not to mention severe nausea, there was absolutely no evidence of any stable conscious sensory concurrents being elicited by the presentation of specific sounds. Kelly's (1934, p. 334) conclusion on this point was unequivocal: "Not one of the eighteen subjects acquired any tendency to see colors upon the presentation of the auditory stimuli."

A decade later, Howells (1944) pointed to the problems associated with the use of introspective report in Kelly's (1934) original study (something that Kelly himself happily admitted to). Howells, for his part, presented his participants with up to 25,000 trials of an arbitrary mapping between color and the pitch of a sound at around the same time. The participants had to discriminate a stimulus patch as being either one of two complementary colors (red vs. green) under conditions where the saturation of the colors could be varied in order to make the participants' task either easier or harder. Performance in this task was facilitated by the presentation of a sound paired with each visual stimulus that onset just before the color patch and continued during its presentation (261 Hz middle C accompanied red, and higher tone 392 Hz, the G above with green stimulus).

To minimize the risk of response bias (i.e., to avoid the possibility of participants simply responding to the tone rather than to the light) a few trials were introduced into the experimental design whereby the tone that was presented was incorrect (thus, putatively inducing errors in participants' performance in relation to the strength of the crossmodal association that had been formed). After all that training, it should come as little surprise that participants responded more slowly, and made additional errors, on those trials in which the sound playing in the background was incongruent with the crossmodal association that had been trained. In order to try to tap into any more perceptual (as opposed to response-related) effects, Howells (1944) subsequently had a couple of his participants adjust a color patch to a neutral point between red and green, or else adjust it to get the best possible white. Under such conditions, a noticeable influence of the tone playing in the background was observed, with the color being adjusted by the participants as if to make up for any color attributable to the presentation of the sound.

Here, in contrast to Kelly's (1934) earlier study, we find subjective reports suggestive of what Howells (1944, p. 101) tentatively calls pseudo-synaesthesia: "The same S also volunteered that he saw a clear image of the screen of the chromatoscope, and of the color normally associated with a given tone, when this tone was sounded while the eyes were closed or in the darkened room." That said, the author also goes on to question "Whether or not the development of this experiment are representative of true synaesthesia, or merely what one writer has called pseudo-synaesthesia." Indeed, what is clear from a careful reading of Howell's original study is that there was no evidence of systematic rich sensory concurrents having been consistently induced across all (or even a majority) of his participants.

The question that one might want to ask at this point is why it is that these older studies are seemingly ignored by contemporary researchers interested in inducing synaesthesia, and why it is that prophecies concerning the acquisition of genuine synaesthesia (where the presentation of a given inducer reliably gives rise to a conscious sensory concurrent) being just round the corner are still being expressed, when negative results also show up nowadays? There are certainly grounds for taking coloredgraphemes as a test case for the acquisition of novel (intramodal) correspondences between sensory and/or conceptual attributes. Furthermore, it would seem logical to take the association that is by far the most common in the case of synaesthesia, present in almost 70% of self-reported cases (see Day, 2005). By contrast, trying to train a perceptual association between the pitch of sounds and different colors (Kelly, 1934; Howells, 1944) might be expected to be harder to establish, given that researchers currently believe that its' occurrence is much rarer amongst developmental synaesthetes. However, note that this line of reasoning assumes that there should be a relation between the prevalence of synaesthesia in the general population and the ease with which non-synaesthetic adults trained for a relatively brief time, typically in a laboratory study, should be able to pick-up (that is, to internalize) an association between two features/dimensions of their experience. As far as we are aware, this assumption has yet to be tested empirically. Indeed, there are certainly grounds for doubting whether the vividness of the concurrent necessarily has anything to do with the strength of the inducer-concurrent mapping (see Rader and Tellegen, 1987).

Going one step further, though, it can certainly be questioned as to whether the phenomenon of synaesthesia actually has anything at all to do with the results of these recent (and older) acquisition studies. It can be argued that all that has actually been demonstrated by these various training protocols is the learning of a novel association (either explicitly or implicitly) between a relatively small number of pairs of co-occurring sensory/conceptual features.

# **WHY CHOOSE THE SYNAESTHETIC LABEL?**

It is important to note that none of the acquisition studies just mentioned provided any strong evidence for the occurrence of a conscious sensory concurrent. Most of these studies hesitate to claim that they have induced "full-blown" synaesthesia in any of their non-synaesthetic participants. However, our concern here is that these authors' extensive use of terms such as "pseudo-synaesthesia," "implicit synaesthesia," "synaesthesialike" or synaesthetic interference is likely to confuse rather than to clarify future debate in this area. The question that we wish to pose here is what added value accrues by relating these behavioral results to synaesthesia, never mind by the addition of a range of other qualifiers?

Researchers who adopt this convoluted terminology implicitly or explicitly appeal to two arguments: First, they consider that similar behavioral aspects are sufficient to characterize a phenomenon as synaesthetic, non-withstanding the occurrence of a conscious concurrent. Second, they consider that the results of their studies do not preclude the hope that conscious synaesthesia can be induced in non-synaesthetes. However, both arguments face severe objections.

# **WHY BEHAVIOR IS NOT SUFFICIENT**

One other way in which to defend the appeal to synaesthesia, would simply be to drop the necessity of there having to be a conscious concurrent. Recent debates over the most appropriate definition of synaesthesia and the key criteria to apply to an increasing variety of kinds and cases have led many researchers in the field to become very permissive with the use of the term (see Marks, 2011; Eagleman, 2012; Simner, 2012, for similar worries). However, of all the controversies surrounding the very existence, or definition, of synaesthesia that have been documented over the last 130 years or so (see Galton, 1880, for one of the first scientific reports on the phenomenon; see also Jewanski et al., 2009), none have questioned that the fundamental characteristic of synaesthesia is the elicitation of a conscious concurrent. It is the supplementary conscious aspect which has led to the coining of the term, which justifies the use of "aesthesia" (which means *perceiving, experiencing*). Synaesthetes are those who experience additional conscious sensory or emotional concurrents, by comparison with the experience of other individuals, in response to the actual presentation (or, on occasion, the mere imagination) of particular sensory, or conceptual, inducers. For instance, Grossenbacher and Lovelace (2001, p. 36) define synaesthesia as an "involuntary concrete sensory experience"; Simner (2012, p. 2) stresses that "These sensations are explicitly experienced in that synaesthetes are consciously aware of them in daily life"; Ward (2007, p. 429) explains that "Synaesthesia is the automatic elicitation of conscious perceptual experiences by stimuli not normally associated with such experiences"; and these are only a few of the many researchers who have stressed the central importance of the conscious concurrent to the definition of synaesthesia.

When non-conscious cases, such as crossmodal matchings or associations between apparently unrelated sensory features or dimensions of experience have been discussed in relation to synaesthesia, they have typically been recognized within their own category. Martino and Marks (2001), for example, classified them as a kind of "weak synaesthesia"—whose relation to conscious (or full-blown) synaesthesia remained an open question (see also Marks, 2011; Spence, 2011; Deroy and Spence, 2013b). It is important to emphasize here, anyway, that when researchers talk about "inducing" synaesthesia they certainly do not mean merely inducing weak synaesthesia.

The possibility of non-conscious cases of synaesthesia has also been mentioned in connection with several recent published cases of implicit bidirectional synaesthesia (see Knoch et al., 2005; Cohen Kadosh and Henik, 2006; Cohen Kadosh et al., 2007; Johnson et al., 2007; Gebuis et al., 2009a,b; Weiss et al., 2009). These studies have demonstrated that certain of the synaesthetes who consciously experience colors when seeing numbers or letters can also show specific patterns of behavioral priming for—or interference with—the processing of a letter or number when a colored stimulus is presented that happens to match the synaesthetic concurrent usually experienced in response to that number or letter. Elsewhere, Rothen et al. (2010) have reported that transcranial magnetic stimulation (TMS) over parieto-occipital brain areas can eliminate the behavioral effects of implicit bidirectional synaesthesia, as it does for the conscious cases in the reverse direction (see Muggleton et al., 2007). There is therefore, good empirical evidence that certain of the neural substrates responsible for the conscious elicitation of a concurrent also underlie the unconscious effects that have now been documented in the opposite direction. That said, while these studies may well have successfully established a non-conscious correlate of conscious synaesthesia, they have certainly not demonstrated the existence of a fully nonconscious form of synaesthesia. Note also that in the very rare, and possibly unique, case of explicit bidirectional synaesthesia studied by Cohen Kadosh et al. (2007), distinct patterns of neural activation, together with different timecourses were observed for the neural activations associated with grapheme-color and colorgrapheme variants—this despite the fact that similar congruency effects were reported in both directions at the behavioral level (cf. Gebuis et al., 2009a).

The occurrence of a conscious concurrent therefore, plays a key role in the diagnosis of synaesthesia, be it established by means of questionnaire data (see http://www*.*synaesthesia*.* uwaterloo*.*ca/ColorAssessment*.*htm), or through the now seemingly well-accepted battery of tests developed by Eagleman et al. (2007). In these tests, participants have, among other things, to report on their conscious experiences. The test of consistency, for instance, often remains the standard to identify someone as a synaesthete. Although the minimal duration of consistency required to be a synaesthete can be questioned (Simner, 2012), consistency over short or long periods of time (years or series of trials, 180, for instance, in Eagleman et al., 2007) is, in the first place, a way to check the occurrence of a conscious concurrent. It is usually combined with other behavioral tests, the latter designed to confirm that a positive result on the former cannot simply be explained in terms of learned associations (e.g., Calkins, 1893; Eagleman et al., 2007), such as have been evidenced to affect participants' performance in a variety of Stroop-like interference, priming, and speeded congruency tasks (see Elias et al., 2003; Hancock, 2006; Eagleman et al., 2007). The majority of training experiments did not evidence the long-term consistency of the effects, therefore, failing to pass this test. This said, it is important to remember that even passing this test (for instance, with the 6 month positive re-test conducted by Colizoli et al., 2013a) does not demonstrate that a conscious concurrent experience necessarily occurs: Positive results might, in principle, be obtained, because of an individual's memories or non-conscious associations.

A more robust test, according to Rothen et al. (2013), comes from the transfer of a conditioning response formed with the concurrent (e.g., the association between a color and a startling sound) to the inducer (e.g., the presentation of a letter). This test establishes a strong distinction between synaesthesia and trained associations, and helps to establish the genuineness of synaesthetic reports, for instance in rare forms such as swimming style synaesthesia. As shown by this physiological test, and argued elsewhere (see Deroy and Spence, 2013b; see also Marks, 2011, for discussion), the systematic presence of a conscious concurrent in synaesthetes connects to other relevant differences in terms of how the association evolves over time.

### **WHY SYNAESTHESIA IS NOT THE ONLY AVAILABLE LABEL**

Couldn't the behavioral effects and implicit components present in the acquisition (or teaching) cases be better conceptualized, then, in terms of the notion of cross-correspondences? Crossmodal correspondences have been defined in terms of matchings between features, or dimensions of experience across distinct senses, due to weak correlations in the environment (e.g., Spence, 2011). In contrast to what is the case in synaesthesia, crossmodal correspondences typically do not give rise to a conscious sensory concurrent (except, perhaps, in the case of those individuals with especially vivid crossmodal mental imagery; see Spence and Deroy, 2013a). Crossmodal correspondences tend to be shared amongst those individuals exposed to a particular set of environmental conditions, and those, like certain audiovisual correspondences, that are linked to robust physical or phonological regularities appear to be universal (e.g., Bremner et al., 2013).

The fact that crossmodal correspondences do not seem to immediately correlate with an obvious environmental regularity has been taken to show that they reflect innate or given predispositions (Gibson, 1969; Marks, 1978; Maurer and Mondloch, 2005; see Deroy and Spence, 2013a, for discussion). However, a better understanding of the statistics of the environment and extended models of associative learning are proving capable of explaining a large number of the cases of crossmodal correspondences that have been documented to date (e.g., see Deroy et al., in press; Parise and Spence, in press). If that is the case, why not simply say that the sort of associations learned (or strengthened) in the studies of Meier and Rothen (2009); Kusnir and Thut (2012), and Colizoli et al. (2013a) demonstrate that adults can acquire intramodal (or crossmodal, if shape and color are considered as independent channels) correspondences, instead of training them in terms of the induction of synaesthesia-like phenomena? The availability of this alternative explanation questions the fact that synaesthesia is the only existing category to enlist and study the kind of phenomena documented above.

# **WHY LEARNING ASSOCIATIONS DIFFERS FROM DEVELOPING SYNAESTHESIA**

Even when they acknowledge the difference between the outcomes of training and synaesthesia, the authors of the recent training studies remain ambivalent in their conclusions: While two of the groups of researchers explicitly conclude that their results do not demonstrate that synaesthesia can be acquired (Meier and Rothen, 2009; Colizoli et al., 2013a)), all three research groups remain surprisingly upbeat about the possible acquisition of synaesthesia in adulthood—stressing that perhaps the amount of training in their studies may simply have been insufficient to elicit the full-blown condition in any of their participants. Take, for example the following quote from Meier and Rothen (2009, p. 1210): "the synaesthetic Stroop test is very useful to assess the strength of a semantic association. However, it seems to fail to assess what is unique about synaesthesia, namely the experience of the synaesthetic color"; or this from Colizoli et al. (2013a, p. 6) "these results alone are not enough to conclude the presence of synesthesia, since over-learned associations produce a Stroop effect." However, Meier and Rothen (2009, p. 1211) concluded their study by stating that: "*it will be interesting to test whether it is possible to induce a synaesthetic experience with more training*." Later, Rothen et al. (2011) also conclude: "*Future research using more extensive adapting training must reveal the boundaries of training synaesthetic experiences*" (Rothen et al., 2011, p. 1250).

It may be instructive to contrast those recent studies claiming to have induced synaesthesia (or one of its variants), as a result of training with the growing number of studies that have started to investigate the establishment of other kinds of correspondences (of both the intramodal and crossmodal variety) between erstwhile unassociated objects or dimensions of experience. These studies remain largely absent from the discussion initiated by the authors committed to studying the possibility of artificially inducing synaesthesia. They originate in the growing interest in trying to understand the ways in which the brain continuously updates its priors in response to changes in the statistics of the environment (e.g., Wozny and Shams, 2011; Xu et al., 2012; see also Baier et al., 2006; Van Wanrooij et al., 2010; Zangenehpour and Zatorre, 2010). Ernst (2007), for example, has demonstrated that people exposed to an arbitrary crossmodal associations between the luminance of a visual stimulus and its felt stiffness, a hapticallyascertained stimulus property (manipulated by means of a force feedback device) that is not (as far as anyone is aware) correlated with luminance in the natural environment, can give rise to a change in the strength of the coupling prior.

The participants in Ernst's (2007) study were trained with pairs of multisensory (visual-haptic) stimuli where an artificial correlation had been introduced between the two unisensory stimulus dimensions: For one group of participants, the brighter the object, the stiffer it was, while this mapping was reversed for the remainder of the participants. The results highlighted a significant change in participants' discrimination performance when their responses to congruent and incongruent pairs of haptic stimuli were compared before and after training. Ernst attributed these findings to changes in the distribution of the coupling prior. Given the very different theoretical background to this study (as compared to the acquisition studies mentioned earlier), it is perhaps unsurprising that he did not ask his participants whether the presence of one stimulus gave rise to any kind of concurrent in the other modality. Hence, at one level, we cannot say for certain whether or not Ernst's participants acquired any synaesthesia-like abilities following training.

Now, clearly, the experimental paradigm utilized by Ernst (2007) exhibits some important differences from the other acquisition-by-training studies that were mentioned earlier. Namely, in contrast to the other acquisition studies, where particular stimuli were associated (i.e., without there being any alignment of dimensions of experience, except for Kelly (1934, pp. 323–324) who did try to align his auditory and visual stimuli but did so using what is known as the Newtonian parallel: "*in order to take advantage of any possible existent relationship between color and tone (generally known as the Newtonian parallel) it was decided to pair each note of the octave with the color occupying the same relative position in the spectral series as the note occupies in the musical scale.*") Ernst, by contrast, used correlated dimensions of experience. Hence, his participants presumably learned the appropriate alignment (at a perceptual level) between two already familiar dimensions and, in so doing, likely demonstrated a generalization of learning beyond the particular set of stimuli that participants had been exposed to during training.

In other words, training studies interested in synaesthesia rely on associative learning to form a one-to-one matching between two limited sets of dimensions or objects, or what one might call a bijection: Each letter should be associated to a different color, and each color to a different letter. By contrast, studies interested in new binding priors rely on associative learning to recalibrate an existing mapping, where each and every value on one dimension, even beyond the learned ones, will correspond to at least one value on the other dimension (with the possibility of two different values on one dimension corresponding to a single value on the other, because of differences in accuracy). This contrast helps pointing out that training studies for synaesthesia are—at least—implicitly committed to synaesthetic associations being of the matching rather than the mapping form.

Now, if one adopts the view that color-grapheme synaesthesia only comes as a late specification growing out of what can be considered as a much more general mapping from degrees of angularity to degree of brightness (Spector and Maurer, 2008, 2011), this commitment might need to be revisited. Would it, for instance, be better to gradually train participants with more and more specific associations between shapes and colors, up to specific associations between graphemes and individual shades? None of the studies which have claimed to have induced some association related to synaesthesia has been particularly interested in testing this specific prediction, because they all choose to start with the synaesthetic inducers known in adults, that is, with graphemes. Notice here that synaesthetes can report having the same color for different letters, suggesting also that the one-to-one correspondence is not necessarily the right way to think about the determination of associations.

Finally, in all of the training cases, we wish to argue that what these training studies miss is the resistance of synaesthesia to further exposure: Once graphemes have been associated with a specific color, synaesthetes cannot simply be retrained to associate them to new colors, nor does their synaesthesia seem to be distorted by novel exposure. This point is particularly important as it highlights, in our sense, what might be special about the induction of a conscious concurrent in developmental synaesthetes, that is, that the association is constantly reinforced every time a grapheme is seen, and appears with the concurrent color (whereas the association will fade in non-synaesthetes, who will just see the grapheme in its actual color).

### **WHAT WE WIN FROM RESISTING THE SYNAESTHETIC LABEL**

Instead of looking for—or continuing to prophesy about—the possible occurrence of a conscious concurrent after training a particular association, the decision to qualify the effects of training as being "synaesthetic" at all should impinge on empirically testable hypotheses.

Besides the conceptual issues, the question also has methodological implications regarding the choice of stimuli and associations to be used in training studies. Noticeably, in order to decide that idiosyncratic synaesthetic concurrent-inducer relations can be quickly trained, future studies should presumably avoid using those stimuli that happen to follow existing intraor crossmodal correspondences (which, in this case, would only be strengthened through training). One reason to raise this worry comes from Colizoli et al. (2013b) who mentioned the existence of a "synaesthetic" Stroop effect even before training. Out of the four color-letter associations tested by Meier and Rothen (2009), three correspond to the implicit associations shared by many synaesthetes (see Simner et al., 2005): A's are commonly red, C's yellow, and D's blue for non-synaesthetes as well as for synaesthetes. One way in which to see whether the strength of the training effect reported in Rothen and Meier's study might have come from the reinforcing of pre-existing correspondences in the non-synaesthetic participants would be, then, first to test participants for their letter-color associations prior to testing, and at least to compare the results for these three letters with the non-common association of B and green (B's are usually associated with blue). This would not directly solve the issue of whether a conscious concurrent can be elicited after training, but would at least help to determine whether crossmodal correspondences have something to do with the results.

A more directly testable empirical prediction to draw out here is that if a synaesthesia-like phenomenon is at stake in the training studies, rather than an intramodal correspondence, then it should show a high degree of automaticity. Contrary to what appears to be the case for synaesthesia, it is indeed questionable whether or not crossmodal correspondences are automatic, i.e., goal-independent and load-insensitive (see Spence and Deroy, 2013b, for a review; see also Mattingley, 2009). Differences in reaction-time (RT) in Stroop-like tasks or other behavioral paradigms could be used to compare the kind of "trained" relations that are labeled as "synaesthetic" and cross- or intra-modal correspondences, but this should be done with equally trained participants (some reporting visual concurrents vs. others who do not) given independent evidence that the amount of training increases the RT differences between congruent and incongruent conditions in Stroop-like tasks (MacLeod and Dunbar, 1988; Colizoli et al., 2013b).

As a second test, one might consider whether the behavioral results from these (or future) acquisition studies shouldn't be combined with neurological data: Although both synaesthesia and crossmodal correspondences have been shown to be disturbed by TMS over the appropriate brain areas, the disturbance of synaesthesia has been shown to result from the disruption of neural activity in the right parieto-occipital (PO) junction (Muggleton et al., 2007). By contrast, the disturbance of the crossmodal correspondence between pitch and brightness in Bien et al's. (2012) study was targeted at the intraparietal sulcus (IPS) instead. Crossmodal correspondences present relevant differences with synaesthesia that are now progressively being documented. Studies highlighting differences between synaesthetes and non-synaesthetic participants trained with similar associations (e.g., Nunn et al., 2002; Elias et al., 2003) again argue in favor of the non-assimilation of artificially induced cases with synaesthesia, and thus, encourage more systematic comparison between naturally-occurring crossmodal and intramodal correspondences, evidenced in non-synaesthetic populations, and the more ephemeral cases of trained pairings of formerly uncorrelated stimuli (or, as we have seen, the strengthening of weakly associated stimuli). Based on the findings of animal neurophysiology (e.g., Furster et al., 2000), one might predict that the acquisition of novel cross-modal associations between formerly independent auditory and visual features (e.g., a red light being paired with a high tone and green light with a low tone) might recruit/require the involvement of the dorsolateral prefrontal cortex. That said, different areas may be involved in the establishing vs. retention of correspondences. Once firmly established, correspondences may be "represented" at more posterior locations, such as in parietal areas (see Bien et al., 2012).

### **INTERIM SUMMARY**

In summary, despite the growing support for there being robust, most often non-conscious crossmodal correspondences in adults and implicit effects of the concurrent on responses to the inducer (especially in the case of intramodal grapheme-color synaesthesia), there is, as yet, no convincing evidence to support the claim that our definition of the condition (synaesthesia) ought to relax the necessity of a conscious concurrent. What is more, it is perhaps also worth reiterating that the recent acquisition studies that were reviewed earlier all agree that a consistent conscious concurrent is a key part of what it is to be synaesthetic. Take, for example, the following from Meier and Rothen (2009, p. 1210) in support for this view: "*we conclude that the synaesthetic Stroop test is very useful to assess the strength of a semantic association. However, it seems to fail to assess what is unique about synaesthesia, namely the experience of the synaesthetic color.*"

Next, we turn our attention to some of the other modes of training in which researchers have claimed that experiences of synaesthetic colors, and synaesthetic concurrents more broadly conceived, can be artificially induced, namely through the use of hypnotic suggestion and by complementing training with the administration of a drug.

# **HYPNOTIC SUGGESTION AND ADDITIONAL DRUGS: NEW PROSPECTS FOR ARTIFICIALLY INDUCED SYNAESTHESIA? HYPNOTICALLY INDUCED CONCURRENTS**

An influential recent study investigated whether it is possible to induce synaesthesia by simply instructing non-synaesthetes under hypnotic suggestion to associate certain colors with particular graphemes. Cohen Kadosh et al. (2009) conducted an experiment on a small group (*N* = 4) of highly hypnotically suggestible non-synaesthetes (these participants had been screened, and had to have obtained the maximum score on the Stanford Hypnotic Susceptibility Scale prior to their taking part in the main part of the study). These participants were hypnotized by Cohen Kadosh et al., and while under hypnosis, instructed to associate specific digits with specific colors (just as done by grapheme-color synaesthetes—here 1-red; 2-yellow; 3-green; 4-turquoise; 5-blue; and 6-purple). Then, during post-hypnotic suggestion, various tests were administered, including a digit detection task in which a sequence of achromatic digits was presented against one of a number of colored backgrounds. On half of the trials, a digit was presented, and the participants had to make a YES/NO discrimination response regarding its presence vs. absence. Using such a task, the authors calculated a measure of perceptual sensitivity (*d'*) using signal detection theory: Crucially, a significant drop in participants' performance was observed when the letters were presented against a background that matched (that is, was congruent with) the color that had been associated with the digit while the participants were under hypnosis than when presented against an incongruently colored background. By contrast, no such drop in performance was reported in any of the control groups tested in this study.

Subjective (i.e., phenomenological) reports were also obtained from the participants. As Cohen Kadosh et al. (2009 p. 263) report: "*At the phenomenological level, participants' reports after the posthypnotic suggestion matched those observed in congenital synaesthetes ... The cross-modal experience was consistent and involuntary, and occurred in their everyday life. For example, one participant reported seeing the digit-color associations when looking at cars' license plates or watching television.*" This despite the fact that the post-hypnotic participants didn't remember having been instructed concerning any digit-color associations. It is, however, unclear from the above quote, or from the rest of the text which aspect of consistency is being referred to here—is it that whenever a concurrent was induced it was consistent in its color and/or was some concurrent (whether or not it was the same one) always induced on seeing a given inducer?

We would argue that Cohen Kadosh et al.'s (2009) results provide what is perhaps the strongest evidence to date that nonsynaesthetes can be induced to exhibit behaviors that resemble those observed in synaesthetes, given the apparent elicitation of conscious sensory concurrents. Bear in mind, though, that the authors themselves stop short of claiming that they had been able to induce synaesthesia, even in their highly suggestible participants. Instead, they merely state that: "*Here we show that posthypnotic suggestion induces abnormal cross-modal experience similar to that in congenital grapheme-color synaesthesia*." (Cohen Kadosh et al., 2009, p. 258, see also p. 263). What is more, an alternative explanation for Cohen Kadosh et al.'s striking results exists; Namely, it is difficult to rule out the possibility that their participants were not simply engaging in a particularly vivid form of visual mental imagery rather than necessarily experiencing synaesthetic concurrents. The instructions that were given to the participants could equally well be associated with the hypnotic induction of mental imagery as with the hypnotic induction of some sort of projector synaesthesia. So, for example, take Cohen Kadosh et al.'s (2009, p. 260) instructions to their participants: "*Look at the color; this is the color of the digit \_, and whenever you see, think, or imagine that digit, you will always perceive it in that color.*" Remember here that highly suggestible individuals also tend to have more vivid visual mental imagery (e.g., Crawford, 1982; Rader and Tellegen, 1987), and that visual imagery has been shown to affect visual perceptual sensitivity in a number of studies (e.g., Ishai and Sagi, 1995).

Again, something conceptually similar to Cohen Kadosh et al. (2009) had been reported previously. In particular, Leuba (1940) paired the rubbing of a participant's arm while under hypnosis with the smell of creosote. Later, the participant had an olfactory image of creosote whenever his arm was rubbed, although he too had no memory of making this crossmodal association. Interestingly, Leuba describes this as an example of conditioned mental imagery (see also Cytowic, 1989/2002, pp. 67–68).

What's more, it is not clear, from Cohen Kadosh et al.'s (2009) report whether any of the highly hypnotically suggestible participants might not also have had similar unusual phenomenology prior to their hypnotic suggestion (cf. Finke and Schmidt, 1977; Broerse and Crassini, 1984, for a very different example of where apparently experiment-induced color percepts, actually turned out to be present prior to experimental manipulation of interest, thus, perhaps acting as something of a cautionary note here). Ever since its appearance in the scientific literature, it has proven particularly difficult to distinguish between synaesthetes and those with especially vivid mental imagery (see Galton, 1880; Vernon, 1937; Craver-Lemley and Reeves, 2013; Price, in press; Spence and Deroy, 2013a). Nevertheless, despite the difficulty of drawing this distinction, many researchers clearly feel it is important to do so.

Future neuroimaging studies might be able to distinguish between these two accounts in the case of color concurrents/color imagery induced by the presentation of a given stimulus, given that color imagery seems to activate an area of the brain that lies substantially anterior to the classic V4/V8 region thought to be involved in color perception and often active in synaesthetes when experiencing conscious color concurrents (see Howard et al., 1998; Nunn et al., 2002; Rich et al., 2006; Niccolai et al., 2012b).

# **DRUG-INDUCED CONCURRENTS**

Drugs have recently been used as another example, and possibly a distinct route, to the induction of synaesthesia, following perhaps the long tradition in writers and artists to consider drugs, a form of sensory mixing and creativity (e.g., Dailey et al., 1997; Sitton and Pierce, 2004). Importantly, the induction of synaesthesia by drugs (often taken as a fact) has been used to constrain possible accounts of the neural underpinnings of synaesthesia (Grossenbacher, 1997; Grossenbacher and Lovelace, 2001). Here, though, we only want to consider the use of drugs in the light of the training studies (see Terhune et al., in press, for a recent review).

In Kelly's (1934) early study, peyote was used after training to see whether the predicted occurrence of hallucinatory experience while the drug was active would follow the conditioned association (trained in the absence of the drug). But, of the five participants who volunteered to take 15 g of peyote, none reported experiencing the learned color when the sounds used for training were played (although Kelly, 1934, p. 335, does say that his participants were "*rewarded with gorgeous arrays of colored visions which compared favorably with the ones described by previous investigators of the so-called divine plant*"). This failed attempt to combine training and induction of conscious concurrents contrasts with a more difficult to assess, but certainly more up to current ethical standards, survey regarding whether people might have synaesthetic experience while under the influence of certain pharmacological agents, and this independently of any preliminary short-term training with certain specific associations (see Luke et al., 2012). These researchers documented correlations between the frequency of drug consumption, the kinds of drugs consumed, and subjective reports of abnormal crossmodal experiences in recreational drug users. On the basis of their survey, Luke et al. argued that: "*synaesthesia is frequently experienced following the consumption of serotonergic agonists such as LSD and psilocybin and that these same drugs appear to augment synaesthesia in congenital synaesthetes*." (Luke et al., 2012, p. 74). As serotonin (5-HT) is a neurotransmitter with mostly inhibitory effects, and yet some documented excitatory effects as well, it is difficult to use this correlation to decide between competing models of synaesthetic induction—in particular, between Grossenbacher and Lovelace's (2001) disinhibited feedback account and Ramachandran and Hubbard's (2001; Hubbard et al., 2011) direct cross-activation account. One important test here would be to investigate the combined effect of serotonin and training in the induction of conscious concurrents.

This said, if the use of drugs were to open new prospects for the elicitation of conscious experiences in otherwise nonsynaesthetic participants, that would raise important questions regarding the kind of conscious concurrents that should be considered as synaesthetic. Sinke et al. (2012) have recently listed a number phenomenological differences between the experiences occurring in developmental synaesthesia and those reported in what they still consider as a separate type of synaesthesia, induced by drugs. The main lesson of their review, in our sense, is that the unreliability of the reports and/or the variability of drug experiences in terms of temporal dynamics and contextual or individual differences (including the quantity and quality of drugs taken) might not even authorize a systematic comparison, even within a single class of drugs. In this respect, we would recommend to stick to behaviorally measurable elements, or neurological data.

Furthermore, unless a form of consistency, through behavioral tests such as Eagleman et al. (2007) speeded congruency test, is established, the connection with synaesthesia remains terminological, and no continuity or relation between drug-induced and non-drug induced cases can be firmly established. How such tests could be reliably performed with participants under the influence of a given pharmacological agent is, however, a source of both ethical and practical concern. The alternative description for such results is that taking one of these hallucinogenic drugs leads to nothing more than vivid hallucinations (see MacDougal, 1898; Aghajanian and Marek, 1999). The evidence here could come from resting-state fMRI, which shows an increased intrinsic functional connectivity in synaesthetes similar to that which is observed in schizophrenic patients who are subject to hallucinations (Jafri et al., 2008).

# **THE BENEFITS OF RESISTING THE SYNAESTHETIC LABEL**

As the occurrence of conscious concurrents remains definitional and mysterious in synaesthetes, and as synaesthesia is a naturally occurring phenomenon that is not known to correlate with drug usage or hypnotic training, it can be argued that the former studies do not really shed any light on the core of synaesthesia. Is it however, possible that the brain mechanisms at stake in the hypnotic and drug induction of conscious concurrents are similar to—or at least serve as some sort of proxy for—the mechanisms explaining the development of synaesthesia?

As far as this suggestion is concerned, it is unclear whether the results of the hypnotic and drug studies have actually shed any light on the fixation of the synaesthetic repertoire, given the lack of any documented consistency over time for the pairs of inducers and concurrents. In Cohen Kadosh et al.'s (2009) study, the effect only lasted for 3 weeks. The same is true, notice, of Colizoli et al.'s (2013a,b) more controversial results, as they reported that the associations that their participants had "learned" had disappeared within 6 months of training having terminated. This, once again, obviously contrasts with claims regarding the persistence of synaesthetic pairings in developmental synaesthesia, which usually lasts a lifetime (e.g., Ward, 2013).

Besides consistency, we should not miss some important differences between the kinds of conscious experiences obtained after hypnosis or drug consumption and the kinds of conscious concurrents reported in synaesthesia. For instance, a vast majority of synaesthetes report experiencing the concurrents "in their mind's eye" and as vividly as perceptual experience—two reports that they do not use to characterize their mental imagery. In the case of the induced concurrents experienced after drug consumption, it is unclear whether the participants were in a state where they could reliably assess the vividness of the experience, independently of the heightened emotional state also induced by the drug. No data is given regarding the peculiar location of the concurrent (external, internal, or "in the mind's eye"). In the case of the induced concurrents following hypnosis, assessment of the vividness of the experience is missing, and the concurrents seem to be experienced as external—a case of projection which is rare or even controversial (Hupé et al., 2012) in the case of synaesthesia. Another difference between the phenomenology of mental imagery and the most common features of developmental synaesthetic cases, is to see whether the experienced form can be zoomed in upon (a signature feature of spatial mental imagery; Finke and Shepard, 1986), whereas synaesthetic concurrents can usually not be changed volitionally (see Price, 2009, for a similar suggestion).

What this suggests is the need to distinguish between the elicitation of certain forms of mental imagery, including crossmodal imagery and synaesthesia. Crossmodal mental imagery is defined as mental imagery occurring in one sensory modality as the result of the presentation of a stimulus in another sensory modality. The occurrence of auditory mental imagery when watching silent speech, or the occurrence of visual imagery when walking in the dark e.g., Sathian and Zangaladze, 2001; Zhang et al., 2004; for neuroimaging evidence, see Lacey et al., 2010; Lacey and Sathian, 2013, provide good examples of crossmodal imagery. It often serves a role of crossmodal completion, by filling-in the missing features of a stimulus that is physically present (e.g., Pessoa and De Weerd, 2003; Gallace and Spence, 2011; see Spence and Deroy, 2013a, for a review) but it can also occur in less specific ways as shown in the elicitation of visual or proprioceptive imagery in musical experience.

Here, we want to side with Price (2009) in suggesting that the effects of developmental synaesthesia should be kept distinct from the mere effects of mental imagery. Although Galton (1880) became interested in colored-hearing as a case of mental imagery, most definitions since have considered synaesthesia to be something different from highly consistent, highly automatic mental imagery. Even Barnett and Newell (2008) who document higher self-rated visual imagery in color-grapheme synaesthetes stress that synaesthesia should be kept distinct from mental imagery. The underlying mechanisms of synaesthesia are distinct from the ones involved in classical mental imagery tasks (see Nunn et al., 2002; Rich et al., 2006; Hupé et al., 2012) and the resulting concurrent usually considered as perceptual, rather than imagistic (Auvray and Deroy, in press for a review).

We recommend, first, that results of hypnosis and reports from past drug experiences—as well as future training studies should be analyzed with respect to participants' mental imagery scores. Taking number-space as an example, there is growing evidence that some—or all—individuals classified as synaesthetes might just have a particularly vivid and consistent form of spatial or visual imagery, which can be tested independently (Price, 2009). Participants identified with a possible form of synaesthesia, should also be tested behaviorally (for instance, using Strooptests) while being instructed to imagine a congruent vs. incongruent color (or concurrent). In so doing, Price, for instance, has shown that many participants identified as "number-space" synaesthetes because of their spontaneous subjective reports of experiencing numbers in space would perform in a way that is similar to other participants with high visual mental imagery in congruent and incongruent imagery instructions. Price goes on to suggest that instead of synaesthesia, odd case reports of seeing numbers in space could be explained in terms of "*the interaction between (1) a predisposition for exaggerated and sometimes distorted spatial coding of numerical or temporal sequences, and (2) the strong visual imagery needed to make these representations salient and explicit*" (Price, 2009, p. 1239).

# **CONCLUSIONS**

In conclusion, in this article, we have questioned the utility (not to mention the validity) of talking about learned associations between erstwhile independent sensory features (or dimensions of experience) in terms of "pseudo-synaesthesia," "synaesthesialike," or "acquired synaesthesia" that has become increasingly popular over the last few years (e.g., see Meier and Rothen, 2009; Kusnir and Thut, 2012; Colizoli et al., 2013a). It is our position that one can avoid the terminological confusion [not to mention the implications of adding to the problematic unity (see Marks, 2011), and etiology of synaesthesia (see Maurer and Mondloch, 2005; Deroy and Spence, 2013a)]. The effects of training can be more easily accounted for within the category of crossmodal and intramodal correspondences, and explained in terms of coupling priors in Bayesian Decision Theory (e.g., Ernst, 2007).

The expansion of different qualifiers for, or divisions between kinds of, synaesthetes over the last decade or so has tended to blur the definition of the condition—while most of these divisions remain controversial, as shown for "associator" and "projector" synaesthetes (Dixon et al., 2004; Rouw and Scholte, 2007; Ward et al., 2007; though see also Gebuis et al., 2009a), "lower" and "higher" synaesthetes (Ramachandran and Hubbard, 2001; Hubbard et al., 2005; Gebuis et al., 2009a), "explicit" and "implicit" synaesthetes (Knoch et al., 2005), and "weak" and "strong" synaesthetes (Martino and Marks, 2001; but see Deroy and Spence, 2013b). Ultimately, despite many researchers having written over the years as if artificially induced synaesthesia is a well-established condition, as compared to the much more common (albeit still rare) developmental (sometimes called congenital or idiopathic synaesthesia) variety, it is possible to raise legitimate grounds for pursuing the argument that there may actually be no such thing as "becoming a synaesthete" in neurotypical adults (see also the evidence for a genetic basis of synaesthesia, Asher et al., 2006, and distinctive neurological profile, Rouw and Scholte, 2010). In other words, it may be impossible to acquire synaesthesia unless one is *predisposed* to be a synaesthete and hence develops it during a critical period of development.

A claim of impossibility seems difficult to support empirically. After all, three possibilities exist here: (1) It is impossible to acquire synaesthesia in adulthood, at least if synaesthesia is defined with respect to the criteria established on the basis of studies of developmental synaesthesia (and hence robust supporting evidence will simply never be forthcoming); (2) It is possible to acquire synaesthesia in adulthood, but it is just that robust empirical evidence has not as yet been established to support this claim; (3) It is evident that non-synaesthetes can become synaesthetes, and we simply need to change our definition in order to take account of the kinds of expressions of the condition that are seen in acquired cases. This third option is the one which seems to encourage the premature labeling of the effects of training as synaesthetic. In the absence of robust evidence of a successful artificial induction of synaesthesia, the idea behind the premature label is that training studies are nonetheless already teaching us something about synaesthesia—for instance, that Stroop interference between apparently unrelated features is not a defining characteristic of synaesthesia. This seems to us a weak argument given the long literature on training and Stroop interference written independently of any concern for synaesthesia. On the contrary, these training studies highlight how different synaesthetic pairings are from learned associations, noticeably that they resist further learning through exposure (Deroy and Spence, 2013a) and can lead to associatied conditioned responses (Meier and Rothen, 2009).

The first claim, that training will never bridge the gap with synaesthesia, is certainly a controversial one, but the one that the accumulated evidence seems to favor. What about the


**Table 2 | Illustration of the various kinds of phenomena that have been labeled as synaesthetic in the literature—and how they depart from the characteristics of developmental/canonical synaesthesia.**

*This table shows that, even in the broad category of "acquired synaesthesia," what we suggest calling "artificially induced synaesthesia" presents more differences with canonical synaesthesia than other novel conditions sometimes labeled as emergent forms of synaesthesia.*

possibility (2) that future hypnosis, or carefully-controlled drug studies might provide evidence of consistent, idiosyncratic, automatic, and crucially conscious concurrents being induced by specific sensory (or conceptual) inducers? As we have stressed in this piece, the evidence in support of such a claim is, at present, remarkably thin on the ground. Regarding the most promising example to date of acquired synaesthesia (Cohen Kadosh et al., 2009), it remains to be seen whether this is anything more than merely hypnotically-induced mental imagery (cf. Leuba, 1940), though the distinction here is, admittedly difficult to draw out (Craver-Lemley and Reeves, 2013). What would nonetheless be the consequences if robust cases of artificially induced synaesthesia were to be evidenced? One possible consequence would be to consider that the idea of a neurological structural grounding for synaesthesia (e.g., Banissy et al., 2012) does not hold, as individuals without the neurological (qua genetic) disposition to develop synaesthesia in childhood could acquire it later in life. The evidence of a polygenetic origin for synaesthesia—still weak (Brang and Ramachandran, 2011)—could point toward a second possibility, that the disposition to become a synaesthete is not equally distributed in the population and in time.

For the moment though, the inclusion of the results of training, drugs, or hypnosis adds to the fact that the word synaesthesia is acting as something of an umbrella term, and losing its scientific robustness (see **Table 2**). The way to resist this dissolution is to insist on the occurrence of a conscious concurrent being a necessary condition for meriting the label of synaesthesia. A form of consistency should be present, not as an *a priori* definitional demand (which can be seen as a dogmatic threshold, see Simner, 2012), but as a way to give weight to the occurrence of the conscious concurrent (see Eagleman et al., 2007).

Now, if it is impossible to acquire synaesthesia in adulthood, how can we explain the fact that many if not all of the inducer-concurrent mappings exhibited by so-called developmental synaesthetes must have been acquired at some stage during development? Think only of the acquisition of colored graphemes, or the childhood tastes (rather than, say, the tastes associated with baby food), that can act as inducers/or concurrents in certain synaesthetes. Here perhaps the appropriate thing to do is to suggest that there may simply be a critical period in human development for the acquisition, or expression, of developmental synaesthesia (see Daw, 2003, on the notion of critical periods for visual development)—probably only in those with a genetic disposition to develop it (Brang and Ramachandran, 2011). There is only one documented case of discordant monozygotic female twins, where one sister has conscious graphemecolor synesthesia while the other does not (Smilek et al., 2005) But this is not a counterexample showing that individuals could have the genetic disposition and not develop synaesthesia: As the authors themselves point out, the absence of development must also have a genetic (and not environmental) basis (i.e., an epigenetic event, X chromosome inactivation, or a mutation of a synaesthesia gene).

For us, perhaps the more interesting issues in this area revolve around the question of if, and why, it may be easier to acquire certain associations than it is to acquire others, and whether there is any relation between the associations that appear commonly in developmental synaesthetes and those that non-synaesthetes find it easier to acquire (or else to strengthen). One other question that seems particularly interesting in terms of future study relates to individual differences in the ability of people to learn new associations (be they intramodal or crossmodal). This issue has come up in passing in a number of the studies that have been reviewed here (e.g., Kusnir and Thut, 2012; see also Cohen Kadosh et al., 2009). It might well turn out to be the case that certain associations are just easier to learn that others (cf. Baeyens et al., 1990)—such as, for example, the association between rounded shapes or higher pitch sounds and brighter colors vs. angular shapes and lower pitch sounds and darker colors. This would then explain why these higher-order are resemblances found between synaesthetes, and between synaesthetes and non-synaesthetes (Simner et al., 2005).

As far as the explanation of the development of synaesthesia is concerned, we would argue the real challenge is to understand the specificity of the learning curve of synaesthesia (including its resistance to further training and its high degree of stability over time once established) as well as the specific nature of the concurrent experience.

# **REFERENCES**


synesthete. *Neuroscience* 137, 3–5. doi: 10.1016/j.neuroscience.2005. 08.057


Examining the neonatal synaesthesia hypothesis. *Neurosci. Biobehav. Rev.* 37, 1240–1253. doi: 10.1016/j. neubiorev.2013.04.001


synaesthetes: brain-behaviour correlations. *Neuron* 45, 975–985. doi: 10.1016/j.neuron.2005.02.008


Object familiarity modulates the relationship between visual object imagery and haptic shape perception. *Neuroimage* 49, 1977–1990. doi: 10.1016/j.neuroimage.2009. 10.081


back": bidirectional cross-activation creates learning opportunities in synesthesia. *Neuroscience* 147, 569–572. doi: 10.1016/j.neuro science.2007.04.008


*Psychol. Bull.* 132, 416–442. doi: 10.1037/0033-2909.132.3.416


Martinez, J.-M. Alonso, and P. U. Tse (London: Elsevier Science), 263–275.


evidence for the bidirectionality of grapheme-color synesthesia. *J. Cogn. Neurosci.* 21, 2019–2026. doi: 10.1162/jocn.2008.21166


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 26 June 2013; accepted: 04 September 2013; published online: 14 October 2013.*

*Citation: Deroy O and Spence C (2013) Training, hypnosis, and drugs: artificial synaesthesia, or artificial paradises?. Front. Psychol. 4:660. doi: 10.3389/fpsyg. 2013.00660*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Deroy and Spence. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Mirror-touch and ticker tape experiences in synesthesia

# *Charlotte A. Chun\* and Jean-Michel Hupé*

*Centre de Recherche Cerveau et Cognition, Université de Toulouse and Centre National de la Recherche Scientifique, Toulouse, France*

### *Edited by:*

*Michael Banissy, Goldsmiths University of London, UK*

### *Reviewed by:*

*University, UK*

*Henning Holle, University of Hull, UK Monika Sobczak-Edmans, Brunel*

# *\*Correspondence:*

*Charlotte A. Chun, Department of Psychology, University of North Carolina-Greensboro, Eberhart Building, PO Box 26170, Greensboro, NC 27402-6170, USA e-mail: cagates2@uncg.edu*

A fundamental question in the field of synesthesia is whether it is associated with other cognitive phenomena. The current study examined synesthesia's connections with phenomenal traits of mirror-touch and ticker tape experiences, as well as the representation of the three phenomena in the population, across gender and domain of work/study. Mirror-touch is the automatic, involuntary experience of tactile sensation on one's own body when others are being touched. For example, seeing another person's arm being stroked can evoke physical touch sensation on one's own arm. Ticker tape is the automatic visualization of spoken words or thoughts, such as a teleprompter. For example, when spoken to, a ticker taper might see mentally the spoken words displayed in front of his face or as coming out of the speaker's mouth. To explore synesthesia's associations with these phenomena, a diverse group (*n* = 3743) was systematically recruited from eight universities and one public museum in France to complete an online screening. Of the 1017 eligible respondents, synesthetes (across all subtypes) reported higher rates of mirror-touch and ticker tape than non-synesthetes, suggesting that synesthesia is associated with these phenomenal traits. However, effect sizes were small and we could not rule out that response bias influenced these associations. Mirror-touch and ticker tape were independent. No differences were found across gender or domain of work and study in prevalence of synesthesia, mirror-touch or ticker tape. The prevalence of ticker tape, unknown so far, was estimated at about 7%, an intermediate rate between estimates of grapheme-color (2–4%) and sequence-space synesthesia (9–14%). Within synesthesia, grapheme-personification, also called ordinal-linguistic personification (OLP) was the most common subtype and was estimated around 12%. Co-occurences of the different types of synesthesia were higher than chance, though at the level of small effect sizes.

**Keywords: synaesthesia, subjective experience, phenomenology, grapheme color, number lines, spatial forms**

# **INTRODUCTION**

When probing atypical subjective experiences, for example when asking people questions such as, "Do numbers have colors?" the most typical reaction from people who do not have such experience is puzzlement. Those who do may also be puzzled, either by the idea that not everyone shares this experience or, on the contrary, by the discovery that they are not unique. The more we ask questions about the intimacy of subjective experience, the greater diversity of responses we seem to get. Is there any "normal" or at least common subjective experience? Synesthesia, which at first seemed a very rare and extraordinary condition, now seems to be shared by a large fraction of the population. As soon as researchers started considering so-called atypical subjective experiences, the social demand for numbers has been high, and quite legitimately: people suddenly either discover that they are "different" or may take comfort from not being that "weird." So the first question is how normative this experience is. Only a few large-scale, systematic studies have been able to provide prevalence estimates so far. The present study aims to contribute to this endeavor by including many subtypes of synesthesia as well as two other, possibly related, subjective phenomena: mirror-touch and ticker tape.

We consider synesthesia as the subjective phenomenon of additional experiences that sometimes, but not always, involves mixing sensory modalities: perceptual, emotional, or imaginary stimulation evokes sensory, representational, cognitive, or affective "synesthetic" experiences. These associations are supplementary, automatic, idiosyncratic, arbitrary, and involuntary (Hupé et al., 2012; Simner, 2012). Though some common trends in synesthetic pairing have been identified (e.g., light colors with high-pitched notes; Ward et al., 2006; common letter-color combinations; Rich et al., 2005; Simner et al., 2005), specific synesthetic associations are distinctive to an individual. A common example is grapheme-color synesthesia, in which letters or numbers evoke color associations (i.e., 7 is green).

Synesthesia runs in families (e.g., Barnett et al., 2008) and there is evidence of genetic influence on its development (Asher et al., 2009; Tomson et al., 2011). However, environmental factors also play a role in the expression of synesthesia, evidenced by: (1) variation in synesthetic subtypes and specific associations within families (Barnett et al., 2008) and (2) examples such as lexical-gustatory synesthetes associating words with foods they ate during childhood (i.e., British synesthetes tend to associate words with flavors like jam and not with chili pepper or wine, which are rarely consumed during childhood; Ward and Simner, 2003) or some grapheme-color synesthetes whose associations correspond to the colored letters from their childhood toys (Witthoft and Winawer, 2013).

Estimates of the prevalence of synesthesia vary depending on the methodology and criteria employed. A large-scale, systematic study including letter-color, number-color, monthcolor, day-color, word-color, person-color, person-smell, tasteshape, and music-color indicated a prevalence of 4.4% in the Scottish population (*n* = 500; Simner et al., 2006). However, this study did not include the two other most common forms of synesthesia (according to Flournoy, 1893), sequence-space synesthesia, a visuospatial representation of sequences, such as numbers ("number forms": Galton, 1880a,b), and graphemepersonification synesthesia, also named ordinal-linguistic personification (OLP): the association of characteristics, such as gender and personality, with linguistic sequences (Simner and Holenstein, 2007). Moreover, this prevalence value was based on the number of synesthetes confirmed with objective measures (Simner et al., 2006), aiming to estimate the lower bound, not the upper bound of the proportion of synesthetes.

Mirror-touch is the automatic, involuntary experience of tactile sensation on one's own body when others are being touched (Blakemore et al., 2005). For example, watching another person's arm being stroked can evoke a physical sensation of the touch on one's own arm. This phenomenon is proposed to arise in part from atypical representations of self-other discrimination (Banissy and Ward, 2007, 2013). Banissy et al. (2009) distinguished between specular subtype (mirrored sensations) and anatomical subtype (non-mirrored sensations, felt on the same side of the body as the true touch), finding that specular was more common (*n* = 17/21).

About 10.8% of an undergraduate British sample (*n* = 567) reported experiencing mirror-touch. Further interview of these 61 subjects inquiring about the location and description of tactile sensations during video observation of touch reduced the number of subjects with potential mirror-touch to 2.5% (*n* = 14). The prevalence of mirror-touch was further estimated from this sample, identifying only 9 subjects who showed Stroop-like effects stronger than controls in a tactile-congruency paradigm (Banissy et al., 2009). However, synesthetic Stroop-like effects can be elicited in non-synesthetes trained to learn grapheme-color associations (e.g., Elias et al., 2003; Meier and Rothen, 2009) and can be mild or absent in synesthetes verified with consistency tests (Hupé et al., 2012; Ruiz and Hupé, under review). Therefore, Stroop interferences likely measure the strength more than the authenticity of phenomenal associations. Nonetheless, the conservative prevalence estimate of 1.6% using this paradigm suggests that mirror-touch is at least as common as grapheme-color synesthesia in the British population, also using stringent criteria (Simner et al., 2006). The intermediate estimate of 2.5% highlights the potential for misunderstanding or false report inherent in brief self-report measures (Banissy et al., 2009).

Ticker tape experiences are the automatic visualization of words as they are thought or spoken, often seen in the mind's eye as static subtitles or a dynamic teleprompter (Galton, 18831 ; Day, 2005). For example, when being spoken to, a ticker taper might see mentally the words as they exit the speaker's mouth. To our knowledge, there are no prevalence estimates available for ticker tape experiences, so the present study may be the first one to report on this prevalence.

Mirror-touch and ticker tape experiences share some commonalities with synesthesia, and could therefore be considered as subtypes of synesthesia (e.g., Serino et al., 2008; Fitzgibbon et al., 2010; Banissy et al., 2011): namely, they involve supplementary, automatic, involuntary associations between an inducer and a concurrent. In mirror-touch, visual or imaginary stimulation evokes somatosensory experience; in ticker tape, auditory or imaginary stimulation evokes visual experience. However, mirror-touch and ticker tape are minimally idiosyncratic and not arbitrary (Hupé et al., 2012; Rothen and Meier, 2013). Whether these phenomena should be considered a subtype of synesthesia largely depends on the criteria employed but there is preliminary evidence that mirror-touch and synesthesia may co-occur: in a mixed group of systematically-recruited (*n* = 9) and self-referred (*n* = 12) participants, nine (43%) individuals with mirror-touch reported grapheme-personification associations and seven (33%) reported grapheme-color associations (Banissy et al., 2009), wellabove the estimates for the general population.

Knowledge of the co-occurrences of mirror-touch and ticker tape with synesthesia could suggest whether these phenomena have similar genetic or neurological underpinnings. As an example, Gregersen et al. (2013) showed that colored-hearing synesthesia was positively associated with absolute pitch (which is not in itself considered a form of synesthesia): out of 768 subjects showing robust evidence of absolute pitch, 20% reported synesthesia, mostly between pitch and color (17% of this population, much higher than estimated in the general population—see Discussion). Combined linkage analysis of multiplex families with synesthesia or absolute pitch suggested that both phenomena were genetically closely related, likely reflecting an underlying commonality of neurodevelopmental mechanisms (Gregersen et al., 2013).

Possible co-occurrence of mirror-touch and ticker tape with synesthesia may, however, be expressed in a complex or subtle manner. Indeed, the very large-scale study by Novich et al. (2011) on about 19,000 self-reported synesthetes suggested that synesthesia may not be a single phenomenon since it appeared to be composed of five independent subgroups: colored sequences, musical colors, colored sensation, non-visual sequelae, and spatial sequence synesthesias. This result could indicate independent neural or genetic mechanisms for these different types of synesthesia (Novich et al., 2011). Co-occurrences of mirror-touch and ticker tape should therefore be searched for at the level of synesthesia subtypes.

The current study had five main goals (1) to examine whether mirror-touch and ticker tape associations are more prevalent in synesthetes than non-synesthetes, (2) to examine whether mirrortouch and ticker tape are associated with specific subtypes of synesthesia, (3) to examine gender differences in the proportions of synesthesia, mirror-touch, and ticker tape experiences, (4) to

<sup>1</sup>Galton F. Inquiries into human faculty and its development. London: MacMillan, 1883, p. 67: "Some few persons see mentally in print every word that is uttered; they attend to the visual equivalent and not to the sound of the words, and they read them off usually as from a long imaginary strip of paper,

such as is unwound from telegraphic instruments. The experiences differ in detail as to size and kind of type, colour of paper, and so forth, but are always the same in the same person."

determine whether proportions of synesthesia, mirror-touch, and ticker tape experiences differ across domain of career and education, and (5) to provide prevalence estimates of phenomenal traits in the French population.

# **METHODS**

# **RECRUITMENT**

A focal point of this project was its ambition to employ methods for participant recruitment unbiased by self-referral. An effort was made to systematically recruit participants from a large and diverse group. Presentations were given to individuals at eight universities and one museum in Toulouse, southern France, in which a short description of the project was provided (a 5-minute oral presentation in the classroom or a quick explanation of the flyer for the museum). Flyers were then distributed with the internet address for a short online survey, "Interior Experiences."

This study was conducted across 2 years. The first year involved recruitment at both universities and a museum; due to administrative restraints, recruitment presentations were different for universities and for the general public. In the first year of the study, university presentations included a definition and specific example of synesthesia as one of many different kinds of thought and perception. Flyers given to the general public explained that everyone has a different way of thinking, yet without any reference to synesthesia (note that in France, synesthesia is still unknown by the vast majority of the population, unlike in the United States and the United Kingdom). The proportion of respondents who reported synesthesia was very similar (less than 1% difference) between university and general public samples, so the explicit reference to synesthesia in the first case did not seem to induce more synesthetes to complete the survey. In the second year of the study, only university students were recruited and no reference to synesthesia was made in the presentation. A unique code was given to each person, allowing us to evaluate the response rate for every class and museum group, but the respondents could remain anonymous if desired. Students were recruited from the domains of economics, political science, law, engineering, agronomy, applied science, veterinary, medicine, psychology, and biology. Members of the general public were systematically recruited from conferences at the local Natural History Museum and during city-wide "Brain Week" events. We distributed a total of 3743 flyers.

# **MATERIAL**

# *Interior experiences survey*

This 5-minute online survey (whose translation is provided in the Appendix) involved questions concerning general demographic information, career and education, and the following types of synesthesia: grapheme-color (letters and/or numbers evoking colors/forms), temporal-color (numbers and/or time sequences: days, months, centuries, *etc*. evoking colors/forms), sequencespace (numbers and/or time sequences being organized in space), grapheme-personification (letters and/or numbers associated with gender/personality), person-color (colors associated with people), and audition-color/form (sounds/voices/music evoking colors/forms). Since audition-color/form synesthesia may be easily confused with normal multisensory experience, a comment box was provided for explanation and examples of this subtype. An additional "other" comment box was provided (for other multisensory experiences in year one of the study and other types of unique thought/perception in year two of the study), as well as a final comment box where participants were instructed to list any doubts or explanations about earlier questions. We added three more questions in the second year in order to implicate the contribution of those without phenomenal traits (see Results and Appendix).

To assess mirror-touch, participants were asked "When you observe a person being touched on a place on his/her body by someone or something, do you feel the sensation on your own body on the place where the person was touched?" Unlike a previous study that asked participants to rate the degree to which they experience mirror-touch on a five-point scale (Banissy et al., 2009), our participants responded dichotomously (yes/no) and were asked to describe their experiences in a comment box, including whether or not the sensations occurred in a mirroredfashion (an example was provided). To assess ticker tape, participants were asked two questions: (1) "When you listen to someone speaking, do you automatically visualize the words that he/she is saying (like a "teleprompter" in a way that scrolls in your head)?" and (2) "When you speak (or think verbally), do you automatically visualize the words you are saying?" To reduce the length of the questionnaire, individuals were not asked for a description of their ticker tape experiences, as this phenomenon may be easier to discern than mirror-touch.

# **CLASSIFICATION CRITERIA**

Consistent with the criteria of synesthesia being arbitrary and idiosyncratic, participants were counted as non-synesthetes if they marked "yes" to questions about synesthesia yet gave only common examples in the audition-color/form, "other," or final comment box, such as smells triggering tastes or stimulation eliciting emotions and memories: for example, a taste or odor bringing to mind a precise visual memory. Participants were also counted as non-synesthetes if their only descriptions were clearly cultural or metaphorical associations; for example, spring associated with a floral ambiance or red, green, and yellow associated with reggae music. Individuals who gave these types of examples in addition to other valid synesthetic examples were still counted as synesthetes for their other subtypes. Participants were counted as non-mirror-touch if their descriptions only mentioned empathy or emotion without physical experience. Because no specific comment box was provided for ticker tape or other subtypes of synesthesia, anyone who answered "yes" to these questions was counted as a synesthete; furthermore, individuals were classified as ticker tapers whether their visual experiences occurred for words that were heard, spoken/thought verbally, or both.

Individuals' career or education domain was classified into three different groups, according to the French education system: (1) Scientific (S), (2) Economic and Social (ES), and (3) Literary (L). The following career and education areas were coded as Scientific: medicine, veterinary, biology, agronomy, applied science, and engineering. The following areas were coded as Economic and Social: political science, economics, and law. The following areas were coded as Literary: psychology, literature, and language.

# **ANALYSES**

*Chi-squared* tests were conducted to examine the following relationships: (1) differences in ticker tape and mirror-touch proportions between groups of self-reported synesthetes and non-synesthetes, (2) differences in ticker tape and mirrortouch proportions across subtypes of synesthesia, (3) differences between men and women in proportions of self-reported ticker tape, mirror-touch, and synesthesia (any synesthesia and across subtypes), and (4) differences among career/education domains in proportions of self-reported ticker tape, mirror-touch, and synesthesia (any synesthesia and across subtypes). Analyses were corrected for multiple comparisons using Bonferroni corrections to maintain 5% family-wise error rates. The pattern of results provoked an investigation of the general tendency to endorse items. To examine this, the relationships among responses to some survey questions were tested *post-hoc*, using point-biserial correlations (Pearson correlations in which one variable is dichotomous) and multiple linear regression.

# **RESULTS**

# **RESPONSE RATES**

Response rates from students and the general population were ∼30 and 16%, respectively. Forty-two individuals who began but did not finish the survey and 38 individuals whose maternal language was not French were removed (i.e., not used in the study), providing usable data from a total of 1017 respondents (university: *n* = 900, museum: *n* = 117). Analyses were first performed independently on the data obtained in the 2 years of the study (345 and 672 respondents, respectively). None of the measures appreciably changed between the 2 years so the data were combined.

Of these respondents, ∼70% reported at least one type of synesthetic association. Such a high proportion indicates an obvious response bias, as well as potential false-positive reports. We decided to hypothesize a very strong response bias, assuming that those who did not complete the survey had neither synesthesia nor other phenomenal traits. In other words, we considered that all people who thought that their inner experience may be special had the motivation to check the online questionnaire and complete the full survey. Such an assumption is of course very conservative. But without verification using consistency tests, we had no way to detect potential false-reports so our initial criteria were certainly too liberal. We hoped that our conservative assumption would balance our liberal criteria. The comparison of our prevalence estimates with those from the few other studies available (see the Discussion section) indicates that these assumptions put us in the right ballpark.

# **PREVALENCE ESTIMATES**

Prevalence estimates of phenomenal traits in the population were estimated based on the full recruitment pool receiving flyers (**Table 1**).

# **CO-OCCURRENCES (***n* **= 1017 RESPONDENTS)**

To examine whether phenomenal traits are more frequent in synesthetes, we computed Pearson χ<sup>2</sup> values to test whether the co-occurrences of phenomenal traits with subtypes of synesthesia

### **Table 1 | Prevalence estimates.**


were higher than chance (**Table 2**, rows 1 and 2). For example, under the assumption of independence between mirrortouch and grapheme-color, we would expect that 57 people with grapheme-color would also have mirror-touch (152 × 383/1017, see **Table 1**), while the other 95 grapheme-color synesthetes would not experience any mirror-touch. The Pearson χ<sup>2</sup> value is calculated by comparing the observed values (75 grapheme-color synesthetes who also have mirror-touch and 77 grapheme-color synesthetes without mirror-touch) to these expected values.

Mirror-touch was associated with all six subtypes of synesthesia (association with temporal sequence-color was marginally significant, depending on the level of statistical correction) and ticker tape was associated with every subtype except temporal sequence-color, and only marginally with grapheme-color. Though these associations were significant, the *phi* statistics indicated small effect sizes at best. Considering the correlations with any type of synesthesia, effect sizes were still small (mirror-touch and any synesthesia, <sup>χ</sup><sup>2</sup> <sup>=</sup> <sup>31</sup>.7, - = 0.18; ticker tape and any synesthesia, <sup>χ</sup><sup>2</sup> <sup>=</sup> <sup>13</sup>.0, - = 0.11). Mirror-touch and ticker tape did not significantly co-occur.

The majority of respondents did not indicate whether their mirror-touch experiences were felt in a mirrored fashion. Of those who did provide this information (*n* = 98), 43% were of the specular (mirrored) subtype and 57% were of the anatomical (non-mirrored) subtype. Similar rates were found among individuals who reported ticker tape experiences for both heard and spoken/verbally thought words (47% of ticker tapers) as for those who reported just one type (53% of ticker tapers). Among those with only one type, it was slightly more common to experience ticker tape for listening (59%) than for speaking/thinking verbally (41%).

We performed similar analyses to evaluate co-occurrences among subtypes of synesthesia (**Table 2**, rows 3–8). All types of synesthesia were significantly and positively correlated with each other (we observed no co-occurrence lower than chance) but most were at the level of a small effect size. Only the co-occurrence between grapheme-color and temporal sequence-color reached the level of a medium effect size.

# **CAREER/EDUCATION DOMAIN (***n* **= 1,017 RESPONDENTS)**

In the first year of the study, we compared ES, S, and L domains. We found no differences among the three domains. In the second


**Table 2 | Co-occurences among subtypes of synesthesia and phenomenal traits, displayed using Pearson χ2 values; phi (***-***) effect sizes in parentheses.**

*According to usual conventions (Cohen, 1988), effect sizes can be considered as small (*- = *0.10–0.29), medium (*- = *0.30–0.49), and large (over 0.5). Effect sizes smaller than 0.10 are likely to reflect spurious correlations and in our study did not reach our statistical criterion correcting for multiple comparisons (Bonferroni correction for 28 analyses: p* < *0.0018,* χ<sup>2</sup> > *9.7. An uncorrected p* < *0.05 is obtained for* χ<sup>2</sup> > *3.84). Small effects are shown in italics. Only one analysis reached a medium effect size, shown in bold and italics.*

year of the study, recruitment was only conducted at S and ES universities. We summed the responses across both years using the common S (*n* = 526) and ES (*n* = 368) domains and found no significant difference for mirror-touch (χ<sup>2</sup> <sup>=</sup> <sup>0</sup>.9, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.34), ticker tape (χ<sup>2</sup> <sup>=</sup> <sup>0</sup>.13, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.72), or synesthetic subtypes (χ<sup>2</sup> values ranged from 0.001 to 4.0, all *p* > 0.047, uncorrected).

# **GENDER COMPARISONS (***n* **= 1,017 RESPONDENTS)**

No significant differences were found between men (*n* = 321) and women (*<sup>n</sup>* <sup>=</sup> 696) for rates of mirror-touch (χ<sup>2</sup> <sup>=</sup> <sup>2</sup>.7, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.10), ticker tape (χ<sup>2</sup> <sup>=</sup> <sup>1</sup>.2, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.28), or synesthetic subtypes (χ<sup>2</sup> values ranged from 0.23 to 2.6, all *p* > 0.08, uncorrected), when summing across both years of the study. The same pattern of results was found in both years of the study. Note that more women filled out the online questionnaire than men. However, we do not know whether this difference reflects a response bias or a sampling bias, since we do not know the male/female ratio of the population to which we distributed the flyers.

### **ACQUIESCENCE**

Our pattern of results (significant, positive correlations between most items) suggested that some individuals might be more likely to endorse items in general. In order to evaluate possible acquiescence effects, we conducted *post-hoc* analyses to examine responses to three items unrelated to the study. Note that these questions were not originally designed for the purpose of examining acquiescence but were added during the second year of the survey so that individuals without phenomenal traits would still feel implicated: "How often do you remember your dreams?" (always, often, sometimes, never/rarely), "Do you have memories before the age of 5?" (yes, no, I don't know), "How often do you have a song stuck in your head?" (often, sometimes, never/rarely). The question on dreams was scored from 1 to 4 and the question on songs was scored from 1 to 3. The question on memories was scored dichotomously, with a response of "I don't know" scored as zero, representing a lack of acquiescence. It is unknown whether these items might be related to phenomenal traits: having a song stuck in one's head could presumably be associated with subtypes of synesthesia that have auditory inducers but there is no strong argument for the other questions being associated with phenomenal traits. Therefore, the correlations of these items with mirror-touch, ticker tape, any synesthesia, and the six subtypes of synesthesia were examined for possible effects of acquiescence.

Twenty-seven point-biserial correlations were conducted so the family-wise error rate was set to *p* < 0.0019. The frequency of remembering one's dreams correlated significantly with global synesthesia (*r*<sup>2</sup> <sup>=</sup> <sup>0</sup>.017), sequence-space (*r*<sup>2</sup> <sup>=</sup> <sup>0</sup>.026), and OLP (*r*<sup>2</sup> <sup>=</sup> <sup>0</sup>.014). A multiple linear regression analysis showed that global synesthesia did not explain any meaningful variance in the endorsement of remembering one's dreams, over-and-above that explained by sequence-space and OLP (*r*<sup>2</sup> change <sup>=</sup> 0.001, *<sup>F</sup>* change = 0.61, *ns*), suggesting that this correlation was specific to the two subtypes. Though significant, the correlations were wellbelow the usual criterion to even qualify as weak (see Cohen's criteria for effect sizes in the legend of **Table 2**). The four other subtypes of synesthesia and phenomenal traits were unrelated to general items. Moreover, the weak correlation for sequencespace and OLP was only found for one out of the three questions. Individuals with synesthesia, mirror-touch, and ticker tape are in fact not more likely to acquiesce on the majority of general items. The relations within subtypes of synesthesia and to phenomenal traits were therefore unlikely due to over-endorsement of items.

# **DISCUSSION**

Few systematic studies exist to date on the prevalence of synesthesia, certain synesthetic subtypes, and mirror-touch; to our knowledge, no previous study has tried to evaluate the prevalence of ticker tape. Knowledge of the frequency of synesthesia and phenomenal traits is important both for informing the general public and to guide future research efforts (e.g., sample size and recruitment requirements). High prevalence rates of certain subtypes may also be a concern for studies not interested in synesthesia a priori but in general cognitive traits, since undisclosed synesthetic experiences may interfere with other measures, as in the example of sequence-space synesthesia (e.g., Price and Mentzoni, 2008; Price and Mattingley, 2013, for a review) for the SNARC effect (spatial-numerical association of response codes; Dehaene et al., 1993). Synesthetic associations between letters and colors may also promote cognitive and memorization strategies and bias the results of certain tests (Rothen et al., 2012). The presence (or absence) of co-occurrences between subtypes of synesthesia and phenomenal traits may suggest possible common (or independent) genetic origins and neuronal mechanisms for the development and expression of these traits. The current study brings new information to these questions, though exact figures should be interpreted with caution due to methodological limitations. In this discussion, we will weigh the arguments for and against the validity of our findings.

Important limitations of our study are the brevity of the screening questionnaire and the absence of verification using consistency tests, so we had no way to detect potential false-reports. However, the free reports provided in the comment boxes, as well as a follow-up study with a subset of participants recruited from this screening (Chun and Hupé, 2013 [Abstract]; see Anecdotal Reports, below) yielded rich information, supporting the valid recruitment of authentic synesthetes.

Another strong limitation of our study is that less than a third of the people to whom we distributed flyers filled out the online questionnaire. The very high prevalence rate of synesthesia that we measured among those who did respond suggested a strong bias presiding upon the choice to fill out the questionnaire. Our prevalence numbers (**Table 1**) are based on the hypothesis of this strong response bias, assuming that those who did not complete the survey had neither synesthesia nor other phenomenal traits. This hypothesis is obviously too conservative, but it seemed to balance out our overly liberal inclusion criteria (without verification of experiences). Indeed, when comparing our estimated prevalence rates with those obtained with stronger methodology, when available, we found in most cases a similar order of magnitude (see Prevalence Comparisons, below). This allows us to hypothesize that our relative rates for subtypes of synesthesia are fairly accurate and our novel prevalence rates provide an adequate first approximation.

Our measures of co-occurrences between subtypes of synesthesia and phenomenal traits could also be contaminated by response bias, if people with some specific traits were for any reason more (or less) motivated to fill out the online questionnaire. Without completely ruling out this possibility, several observations argue for a limited influence of such a bias. First, we measured similar rates of synesthesia and phenomenal traits in men and women. Previous gender differences reported in synesthesia (e.g., Baron-Cohen et al., 1993) are now thought to be due to disparity in self-disclosure (Ward and Simner, 2005). The finding of equal gender proportions in the current study thus diminishes the likelihood of self-disclosure biases in our sample, as equal rates of synesthesia in males and females were found in large-scale studies that verified authentic associations in systematically recruited samples (Sagiv et al., 2006; Simner et al., 2006) and a mixed systematic and self-referred sample (Seron et al., 1992). A second, incidental validation of our recruitment method was provided by the results of year one. As indicated in the Methods section, the University and Museum groups received different instructions, with reference to synesthesia only in the first group. Yet the results were highly similar in both groups, suggesting that the response bias of completing the survey was not specific to synesthesia. A third argument in favor of the validity of our results on co-occurrence comes from the comparison with the few numbers available in the literature, based either on systematic recruitment or large-scale self-reports (see *Co-occurrence Comparisons*, below).

# **ANECDOTAL REPORTS**

There was considerable variety in individuals' experience of phenomenal traits. Mirror-touch was described for many different sensations, including: pain, general pleasure, sexual pleasure, kissing, temperature, tickling, pinches, etc. We even received reports of mirror-touch experiences in response to observation of very specific activities, such as clipping fingernails or putting on lotion. This is consistent with reports that mere observation or imagination of motor activity can induce synesthetic associations, as seen in swimming-style synesthesia (Nikolic et al., 2011; Mroczko-Wasowicz and Werning, 2012). Almost all reports of mirror-touch described direct reciprocation of the localization of touch (whether specular or anatomical). We received less common reports from people (*n* = 3) who always experienced tactile perceptions in the same place, regardless of localization of observed touch; for example, "the inner thigh," "the spinal cord," or "a shiver of pain that scrapes from the left armpit to the forearm." Intensity of perception was also differentially experienced: some reported that observed pain was directly related to perceived pain, even to the point that it became "handicapping and unbearable." For others, perceived intensity was more or less independent from the strength of observed pain, felt as more of a tightening or a twinge.

Banissy et al. (2009) previously reported that almost 20% of individuals with mirror-touch also experienced personal tactile sensations when observing a lamp being touched. Three participants in our study (two grapheme-color synesthetes and one number-space synesthete) reported similar object-tactile associations, in which someone touching their personal belongings led to experience of touch on their own body (e.g., "a prickling sensation on the back of my neck that is both painful and pleasurable"). Note that these participants offered this information in a comment box even though they were not directly asked about these perceptions, so the occurrence is undoubtedly higher than what we found. Unlike perceptions in response to lamps (Banissy et al., 2009), each of our three participants reported this experience specifically for their personal possessions. This suggests that emotion may play a role in the experience of tactile phenomena such as mirror-touch and synesthesia. While some participants' tactile experiences generalized to strangers and fictional characters, many reported that their mirror-touch responses were enhanced for—or even limited to—people with whom they feel close. Such reports are consistent with previous findings, like those showing that mirror-touch perceptions are stronger for observed touch of real bodies than of dummy bodies (*n* = 14; Holle et al., 2013).

Ticker tape experiences also showed a wealth of individual differences, as reported in semi-structured interviews of participants recruited from our sample for another study (Chun and Hupé, 2013 [Abstract]; ticker tape: 7 men, 11 women). Most participants reported a constant size and font for visualized letters; however, some individuals reported experiencing a change in letter size depending on the volume with which words are spoken. The way in which ticker tape perceptions were "displayed" varied as well: we received reports of both static display, on a screen inside the head or in front of the body, and dynamic display, with words that stream out through the mouth or from behind the head. One ticker taper reported that during a verbal fluency task, ideas "stacked up" visually behind her head before streaming through her mouth as she said them aloud. When too many ideas were being held there, some would disappear before she could say them and thus disappeared from memory. A subset of ticker tapers described visualizing noises spelled out onomatopoetically ("as in a comic book"), whereas others did not. Likewise, some ticker tapers reported spelling out words phonetically from an unknown language while for others, ticker tape seemed directly linked to comprehension: they reported that hearing a language they do not understand would fail to elicit ticker tape.

# **PREVALENCE COMPARISONS**

**Table 3** shows a comparison of prevalence estimates between the current study—employing systematic recruitment without verification of associations—and previous systematic recruitment studies that were able to verify subjects' associations. Due to

### **Table 3 | Prevalence comparisons.**

the use of different populations, different recruitment and sampling strategies, and different diagnostic criteria among studies, their comparability is arguably limited. However, prevalence estimates in the current study are not significantly different from those previously reported in the literature for grapheme-color <sup>2</sup> and sequence-space associations, as well as for initial self-report of mirror-touch. Our estimates are slightly higher than previous reports for person-color and temporal sequence-color and are much higher than previous prevalence estimates for OLP; hypotheses to explain such discrepancies are proposed below. Though the estimated prevalence of audition-color in the current study appears elevated compared to a previous report, this difference could be due to the questions we asked (see Appendix, Interior Experiences Survey): we asked participants whether they associated colors with sounds and voices, in addition to music (Simner et al., 2006).

2When combining the results of both studies by Simner and colleagues that reported the proportions for color associations to letters or numbers, the proportion of synesthetes was 18/719 (=2.5%), which is only marginally different from 4.1% (152 synesthetes among 3473 individuals; <sup>χ</sup><sup>2</sup> <sup>=</sup> <sup>3</sup>.99, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.046).


*aSimner and Holenstein, 2007: 78 out of 219 individuals initially reported "some type of OLP and/or grapheme-color synesthesia." From this group, 21 (9.6%) scored higher than controls (2–3 week retest) 5 weeks later and 10 (4.6%) continued to score higher 1 year later. The strict estimates are derived from these 10 individuals: 2 with OLP, 7 with grapheme-color, and 1 with both.*

*bSimner et al., 2006: For comparability with the current study, the proportion of individuals with letter- and/or number-color associations are reported from Simner et al.'s data; therefore this figure is slightly different than the 1.4% typically cited from the university study, referring to individuals with both letter- and number-color associations.*

*cSeron et al., 1992: 194 individuals were recruited systematically to take a brief questionnaire. From this group, 27 individuals gave a positive response regarding "some particular number representation;" no breakdown was provided for group composition from the three measured types of associations: sequence-space, number-color/form, and simple analogical representations ("the quantity was directly represented by patterns of dots or other things such as alignment of apples, parts of a bar of chocolate, etc."). An additional non-systematic, informal inquiry yielded 22 more positive responses. From the mixed group of 49 individuals, 26 agreed to answer a more detailed questionnaire; however, it is unknown how many of these verified associations came from the systematically-recruited group. The sequence-space prevalence is therefore less than or equal to 27/194. It should be noted that sequence-space composed 74% of positive responses from the detailed questionnaire and 68% of positive responses from the brief questionnaire. If the frequency found from the detailed questionnaire is accurate, one might speculate the sequence-space prevalence to be* ∼*10% (0.74* × *13.9%).*

# *Phenomenal traits*

To our knowledge, this is the first study to present systematic data on ticker tape experiences. Prevalence rates are estimated at 7% for ticker tape and 10% for mirror-touch. In a previous mirrortouch study, detailed interview and examination of response to videos of tactile stimuli reduced the number of potential mirrortouch subjects by a factor of over 4 (Banissy et al., 2009). Though specific elimination criteria were not provided, this yields two possible implications for the current study: (1) genuine mirrortouch is consistent and our prevalence estimate is too high, or (2) mirror-touch lacks the consistency of synesthesia (Rothen and Meier, 2013).

In contrast with the previously found preponderance of specular mapping in individuals with mirror-touch (*n* = 17 specular vs. 3 anatomical; Banissy et al., 2009), the current study found relatively equal rates of mirror-touch subtypes, favoring anatomical mapping (*n* = 42 specular vs. 56 anatomical). Though this large disparity in subtypes could mean that anatomical mapping is more prevalent in the French population, it seems more likely that the associations of those reporting anatomical mapping have lower consistency, as they were less frequently identified with the use of stringent criteria (Banissy et al., 2009).

# *Ordinal-linguistic personification*

OLP synesthesia may be more prevalent in Francophone (12%) than in Anglophone (1.4%) populations. This would be logical given the masculine-feminine categorization built into the structure of the French language. In French, grammatical gender exists only for words (which we did not specifically inquire about) but personification associations are seen at the level of numbers and letters. It has already been shown that childhood cultural experience can shape the expression of specific associations within synesthesia (Ward and Simner, 2003) but it is an empirical question whether culture and/or maternal language may affect the actual development and prevalence of synesthesia within a population. The idea that grammatical gender may shape thought specifically related to personification attribution has already been proposed (Amin et al., 2011).

The potential role of culture and maternal language on the development and expression of synesthesia remains speculative for several reasons: (1) the current study lacked verification of associations, (2) Simner and Holenstein's (2007)'s study may have had an insufficient sample size to make a stable prevalence estimate (3 synesthetes from a group of 219), and (3) Simner and Holenstein (2007) used a very conservative procedure (see **Table 3**, footnote 1) aimed at specifying the lower bound of this estimate.

# *Person-color*

One possible cause of the discrepancy in observed prevalence rates for person-color associations (6.6% in our study vs. 2% by Simner et al., 2006) could be related to cultural differences in the desire to conceal these associations, due to the stigma related to mystical aura-reading. Non-idiographic, synesthetic-like personcolor associations (i.e., associating a person with a frequentlyworn color or with a physical attribute, such as hair/eye color) may be more common than synesthetic-like associations for other subtypes, such as grapheme-color; therefore it is also possible that these non-idiographic associations were more easily identified and eliminated with face-to-face screening compared with online screening.

# **CO-OCCURRENCE COMPARISONS**

**Table 4** shows a comparison of co-occurrence rates between the current study and previous studies that used at least partial systematic recruitment. The same general trends in co-occurrence patterns lend validity to the current examination. Banissy et al. (2009) observed a high incidence of both grapheme-color and grapheme-personifications in their small sample of verified mirror-touch individuals, indeed suggesting co-occurrence of mirror-touch with synesthesia. Simner et al. (2006)'s systematic examination showed that grapheme-color and temporal sequence-color were highly correlated, in agreement with our largest observed effect size. Unlike what was found in the current study, however, they found graphemecolor and temporal sequence-color to be completely independent from person-color and audition-color, with zero cases of co-occurrence.

Sagiv et al. (2006) examined the occurrence of number forms in both grapheme-color synesthetes and non-synesthetes (that is, not including number forms in the definition of synesthesia). They found a higher proportion of number form cases in grapheme-color synesthetes. The greater rate of co-occurrence found in their study compared to our study could be due to their different recruitment procedures for grapheme-color synesthetes (no systematic recruitment) and non-grapheme-color synesthetes (systematic recruitment). Seron et al. (1992) reported the number of grapheme-color synesthetes among individuals with sequencespace. This time the number of co-occurrences was lower than observed in our study but here as well, recruitment was not homogeneous. Simner and Holenstein (2007) measured both grapheme-color and OLP, but their strict criterion for inclusion restricted their sample to only three people with OLP (see **Table 3**, footnote 1), precluding meaningful statistical comparisons.

Novich et al. (2011) conducted the largest study to date on co-occurrences between subtypes of synesthesia, on the basis of about 19,000 self-referred reports. However, like in our study, most subtypes could not be verified. Prevalence estimates were not possible since only potential synesthetes filled out their online questionnaire. Relative prevalence rates of the different subtypes were also not possible to calculate, since grapheme-color synesthetes were apparently more motivated to visit the "synaesthesia battery" website (probably due to research interests and media coverage). This bias is expressed in their high proportion of grapheme-color synesthetes (about 40%) compared to sequencespace synesthetes (31%), while systematic recruitment studies have found a much higher prevalence of sequence-space than grapheme-color, comparing both within (Seron et al., 1992) and across populations (i.e., Sagiv et al., 2006 vs. Simner et al., 2006). This strong bias means that their observed rates of co-occurrences could not be extrapolated to the general population, as demonstrated by the following thought experiment: if only graphemecolor synesthetes visited the synaesthesia battery website, then all sequence-space synesthetes would also report grapheme-color


### **Table 4 | Co-occurrence comparisons.**

*GC, Grapheme-color; MT, Mirror-touch; OLP, Ordinal-linguistic personification; SS, sequence space; TSC, Temporal sequence-color.*

*aBanissy et al., 2009: 9 individuals were recruited systematically and 12 individuals were recruited by self-referral.*

*bSagiv et al., 2006: Non-grapheme-color synesthetes were recruited systematically (n* <sup>=</sup> *311) but grapheme-color synesthetes (n* <sup>=</sup> *100) were self-referred online. cSeron et al., 1992: From a mixed recruitment group (see Table 3, footnote 2 for a full explanation), detailed questionnaires showed 1 out of 20 SS who had GC as well; brief questionnaires showed 1 out of 13 SS who had GC as well.*

synesthesia. In spite of such a bias, the main result of that study a clustering of subtypes of synesthesia—is probably valid, and in that case very informative. Continuing the thought experiment, if only grapheme-color synesthetes visited the synaesthesia battery website, that alone would not lead to a higher proportion of those also experiencing colors for temporal sequences than those also experiencing sequence-space (as observed by Novich et al., 2011). Such strong bias would predict the same proportion of graphemecolor synesthetes (that is, 100% in this extreme case) among their whole sample and the subset of synesthetes with sequencespace (as observed by Novich et al.), but with no influence on the proportions of synesthetes with sound-color associations, for example, in the whole sample and among sequence-space synesthetes. Therefore we have no reason to suspect that their recruitment bias questions their observed clustering of subtypes of synesthesia within five groups. Such clustering leads to precise predictions for our study. Among the five subtypes included in both Novich and our study, four types belonged to different groups. Only grapheme-color and temporal sequence-color belonged to the same group. In agreement with Novich et al. (2011), co-occurrence between these two types was the only one in our study that reached a medium effect size.

Novich and colleagues emphasized the relative independence between subtypes of synesthesia, showing, for example that the proportion of people having each type of synesthesia was very similar for synesthetes with or without sequence-space synesthesia. Our results do not contradict this observation: sequencespace synesthesia was significantly correlated with every other subtype, not any subtype in particular (all small effect sizes, *phi* between 0.11 and 0.20—see **Table 2**). Novich and colleagues could not measure such a correlation because they had no control group without synesthesia.

Our results therefore show that, even if synesthetic subtypes cluster in different groups, as shown by Novich et al. (2011), synesthetes tend to experience several subtypes of synesthesia, an important argument for inclusion within a unique phenotype. Following such logic, one may argue for including mirror-touch and ticker tape also within the synesthesia phenotype. However, co-occurrence should not be the sole criterion considered, as exemplified by the co-occurrence of absolute pitch and synesthesia (Gregersen et al., 2013). Moreover, the average effect sizes of co-occurrences between phenomenal traits and synesthesia were weak (0.13 for mirror-touch and 0.10 for ticker tape), even weaker than between subgroups of synesthesia (0.19). Given the high uncertainty surrounding these numbers (due to our methodological limitations), further research will be necessary before reaching any strong conclusion. At this stage, we would like to conclude that genetic and/or neurological links between synesthesia, mirror-touch and (but to a lesser degree) ticker tape, are plausible.

# **CONCLUSIONS**

Our study had five main goals. First, to examine whether mirrortouch and ticker tape associations are more prevalent in synesthetes than non-synesthetes. The answer is yes (which may indicate common genetic or neural mechanisms), though only to a weak degree in our study, and we cannot exclude that the elevated frequency of these phenomenal traits in synesthetes resulted from our recruitment bias. Our second goal was to examine whether mirror-touch and ticker tape are associated with specific subtypes of synesthesia. The answer is no: co-occurrences, if real, were distributed across all subtypes. Our third aim was to examine gender differences in proportions of synesthesia, mirrortouch, and ticker tape experiences; no differences were found. The fourth goal was to determine whether proportions of synesthesia, mirror-touch, and ticker tape experiences differ across domain of career and education; no differences were found. Finally, we aimed to provide prevalence estimates of phenomenal traits in the French population. We estimated ticker tape at 7% and mirror-touch at 10%. These numbers place the prevalence of these phenomena within the range of those of grapheme-color (4%) and sequence-space (9%), the most studied subtypes of synesthesia. We observed frequent associations of people with colors (7%) and graphemes with gender or personality (12%). These proportions are higher than previously presumed, based indirectly on sampling of Anglo-Saxon populations. We suggest that grapheme-personifications may be more frequent in the French population. If confirmed, this cultural difference would show that culture and maternal language play an important role in the development and/or expression of synesthesia.

The main strength of this study was its systematic recruitment, though the sample was still biased toward scholarly individuals.

# **REFERENCES**


Newell, F. N., et al. (2008). Familial patterns and the origins of individual differences in synaesthesia. *Cognition* 106, 871–893. doi: 10.1016/j.cognition.2007.05.003


The use of a brief online questionnaire yielded a sizeable sample but introduced greater ambiguity than face-to-face studies. The study's main limitation was our inability to test the authenticity and consistency of participants' perceptions. For example, report of synesthetic-like experiences resulting from drug use or neurological conditions reflect possible sources of error. In light of these limitations, the authors made every effort to provide conservative estimates of synesthesia and phenomenal traits. However, without verification of the consistency and number of synesthetic associations, the group of synesthetes may be better described as "individuals with synesthetic-like experiences." Considering these shortcomings, evidence of a higher prevalence of mirrortouch and ticker tape associations in the synesthetic population is tentative.

# **ACKNOWLEDGMENTS**

Research funded by Agence Nationale de Recherche ANR-11-BSH2-010. Charlotte Chun was sponsored by a Fulbright fellowship.

investigations. *Brain Res. Cogn. Brain Res*. 16, 232–237. doi: 10.1016/S0926-6410(02)00278-1


associations produces a synaesthetic Stroop effect, but not a conditioned synaesthetic response. *Neuropsychologia* 27, 1208–1211. doi: 10.1016/j.neuropsychologia. 2009.01.009


blue". *Cognition* 44, 159–196. doi: 10.1016/0010-0277(92)90053-K

Simner, J. (2012). Defining synaesthesia. *Br. J. Psychol*. 103, 1–15. doi: 10.1348/000712610X528305

Simner, J., and Holenstein, E. (2007). Ordinal linguistic personification as a variant of synaesthesia. *J. Cogn. Neurosci*. 19, 694–703. doi: 10.1111/j.1748-6653.2011.02009.x


synesthesia: suggestive evidence of linkage to 16q and genetic heterogeneity for the condition. *Behav. Brain Res*. 223, 48–52. doi: 10.1016/j.bbr.2011.03.071


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 05 July 2013; accepted: 02 October 2013; published online: 07 November 2013.*

*Citation: Chun CA and Hupé J-M (2013) Mirror-touch and ticker tape experiences in synesthesia. Front. Psychol. 4:776. doi: 10.3389/fpsyg. 2013.00776*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Chun and Hupé. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# **APPENDIX**

# **INTERIOR EXPERIENCES SURVEY**

*Author*'*s Note:* The survey questions were completed in sequential order, such that subsequent questions were not visible until the previous question had been filled out.

# *(A) Introduction*

Thank you very much for accepting to participate in this study, led by the Brain and Cognition Research Center, laboratory of the University of Toulouse and of CNRS (http://cerco.ups-tlse. fr). Please respond to all the questions below concerning your activities and your personal experiences, as well as some general information. It shouldn't take much more than 5 minutes.

Know that your responses and personal information—if you provide us with them—will be kept confidential and evidently won't be distributed to exterior parties. You will have the possibility to give us your email address at the end of the questionnaire. In this case, we will be able to ask you to participate in the next step of our study. We will select a varied sample, representative from this questionnaire if possible. So it is important that you complete it carefully even if you do not wish to participate in the next step of the study.

If you are willing and selected, we will later ask you to respond to other questions (via internet), then to come in to the laboratory to take several playful tests, using your creative and imaginative capacities. We will then explain in more detail what that consists of and of course you will always have the choice to continue or stop your participation at any moment. You will be compensated for the time spent at the lab.

If you have questions, do not hesitate to contact us.

# *(B) General information*


Yes/No

If yes, indicate which languages you learned before starting school

7. Gender

Male/Female

	- Left/Right/Ambidextrous

# *(C) Your activities*

1. Do you have a regular artistic activity?

(We consider all types of artistic practices—whether in fine arts, photography, music, dance, theatre, writing, etc.—from the moment that you produce a "piece or work," whether in an institutional or private setting).

Yes/No

If yes, explain which type and an approximate frequency If yes, do you ever present your pieces in public (or have representations in public)?

# *(D) Your subjective experiences*

It's possible that certain questions are difficult to understand in this case, there is a chance that it concerns a particularity that you don't have and you should answer "no" to the question. If you have doubts, you can indicate and explain them in the space provided for this later.

1. When you think, would you say that it's in verbal form (with an interior dialog)?

Always/Often/Sometimes/Very rarely or never

2. When you think, would you say that it's in the form of images?

Always/Often/Sometimes/Very rarely or never

If this happens, can you explain the dominant nature of these images? (for example, purely visual, auditory, olfactory, audio-visual, etc.)

B3. Do you remember your dreams?

Always/Often/Sometimes/Very rarely or never

B4. Do you have memories before the age of 5? Yes/No/I don't know

You may explain your earliest memory, if you would like.


Yes/No


Yes/No

If yes, explain whether it is systematic. Indicate whether your sensation is mirrored (for example, if a person across from you is touched on their right arm—that is therefore on your left side—do you feel the sensation in your own right arm or your left arm, therefore mirrored). If your experience is close to this but is different than what is described, please explain as well.

	- Yes/No

If yes, please explain (examples are welcome)


If yes, please explain (examples are welcome)

A18. Do you know or think you have other immediate family members (siblings, parents, children, cousins, nieces/nephews, or aunts and uncles) that would have responded yes to one of the questions in this section?

If yes, please explain and tell how many (number of brothers, sisters, etc.)


If yes, please explain (examples are welcome)

If you responded "yes" to at least one of the questions 9–17, you are most likely what we call a "synesthete." You can find more information at the following internet address:

http://cerco.ups-tlse.fr/∼hupe/synesthesie.html

If you have doubts or explanations to give, please indicate them here, with the number to which they correspond.


Thank you for your time and participation! You may find the description of our research project et follow its development at:

http://cerco.ups-tlse.fr/∼hupe/experienceinterieure.html

# *Author's Note*:

AOnly asked in year one of the study

BOnly asked in year two of the study

∗Touch-color experiences were asked about in the second year of the study as part of a recruitment effort for interview-based research on touch-color and orgasm-color synesthesia. It was not included in the current analyses due to its different sample size, as well as the already large number of variables and time constraints in the current study.

# Veridical mapping in savant abilities, absolute pitch, and synesthesia: an autism case study

# *Lucie Bouvet 1,2 \*, Sophie Donnadieu1,3 , Sylviane Valdois1,4 , Chantal Caron5 , Michelle Dawson5 and Laurent Mottron5*

<sup>1</sup> Laboratoire de Psychologie et Neurocognition, CNRS UMR 5105, Grenoble, France

<sup>2</sup> Université Lille 3 - Charles de Gaulle, Lille, France

<sup>3</sup> Université de Savoie, Chambéry, France

<sup>4</sup> Centre National de la Recherche Scientifique, Paris, France

<sup>5</sup> Centre d'Excellence en Troubles Envahissants du Développement de l'Université de Montréal, Hôpital Rivière-des-Prairies, Montréal, Canada

### *Edited by:*

Michael Banissy, Goldsmiths University of London, UK

### *Reviewed by:*

Michael Banissy, Goldsmiths University of London, UK Clare Jonas, University of East London, UK

### *\*Correspondence:*

Lucie Bouvet, Université Lille 3 - Charles de Gaulle, UFR Psychologie, Domaine Universitaire "Pont de Bois", 59 653 Villeneuve d'Ascq, France e-mail: lucie.bouvet@gmail.com

An enhanced role and autonomy of perception are prominent in autism. Furthermore, savant abilities, absolute pitch, and synesthesia are all more commonly found in autistic individuals than in the typical population. The mechanism of veridical mapping has been proposed to account for how enhanced perception in autism leads to the high prevalence of these three phenomena and their structural similarity. Veridical mapping entails functional rededication of perceptual brain regions to higher order cognitive operations, allowing the enhanced detection and memorization of isomorphisms between perceptual and nonperceptual structures across multiple scales. In this paper, we present FC, an autistic individual who possesses several savant abilities in addition to both absolute pitch and synesthesia-like associations. The co-occurrence in FC of abilities, some of them rare, which share the same structure, as well as FC's own accounts of their development, together suggest the importance of veridical mapping in the atypical range and nature of abilities displayed by autistic people.

**Keywords: autism, synesthesia, savant abilities, cognition, veridical mapping**

# **INTRODUCTION**

Savant abilities involve a marked contrast within the same individual, in whom apparent intellectual or developmental disabilities co-exist with strong, sometimes outstanding, specific talents. The prevalence of savant or exceptional abilities in autism is understudied and likely has been underestimated, with popularly reported figures such as 1 in 200 (Hermelin, 2001) falling well short of current findings [approximately 1 in 3; (Howlin et al., 2009)]. To date the savant literature has concentrated on abilities in a few general areas thought to be characteristic, for example drawing (Mottron and Belleville, 1993; Wallace et al., 2009), musical skills (Sloboda et al., 1985; Young and Nettelbeck, 1995; Heaton et al., 2008), and calendrical calculation or other types of memory (Mottron et al., 2006b; Thioux et al., 2006; Neumann et al., 2010), as well as hyperlexia (Nation, 1999; Newman et al., 2007). However, it should be noted that many other savant abilities or entire areas of savant ability may exist but are yet to be adequately studied (e.g., estimation; Soulières et al., 2010). Given the high occurrence of savant ability in autism, and reciprocally the high prevalence of autism or autistic traits among savants (Heaton and Wallace, 2004), the emergence of savant abilities is linked with some specificity to the autistic mind and brain.

Using the weak central coherence theory framework (Happé and Frith, 2006), Happé and Vital (2009) proposed that an autistic processing bias toward local information may predispose these people to talent. Based on parental report only, they observed that items considered related to a detail-focused cognitive style were more pronounced among children with "special abilities" than

among those without. However, this cannot explain the acquisition of savant abilities in autism, and the concept of local bias itself is ill-defined and intrinsically multi-level. Baron-Cohen et al. (2009) proposed that local bias or detail-focus associated with another concept, hyper-systemizing, might account for the predisposition of autistic1 people to talent. Systemizing is defined as "the drive to analyze or construct systems"; systems in turn are sets of information following rigid, predictable "if *p*, then *q*" rules. For instance, systemizing can account for superior performances in some folk physics tests (Baron-Cohen et al., 2001; Binnie and Williams, 2003). However, while rules may govern common areas of savant abilities, such as 3-D drawing, music, calculation, and the decoding of written language, these areas may also be characterized by irregularities and unpredictability, as in English orthography or musical improvisation.

The enhanced perceptual functioning (EPF) model (Mottron et al., 2006a) provides another trend of thought, in which an enhanced role and autonomy of perception may be prominent in what is specific to the autistic mind, and particularly, in the acquisition of savant abilities. Existing evidence for superior perceptual processes in autism comes primarily from results of visual and auditory tasks (for reviews, see Simmons et al., 2009; O'Connor, 2012), including those which are not performed perceptually by typical participants (i.e., working memory task, Koshino et al.,

<sup>1</sup> In order to reduce unhelpful biases, and in keeping with the current consensus on language in autism research (see Pellicano and Stears, 2011), we prefer the respectful, accurate term "autistic" rather than "person with autism" (see also Sinclair, 1999).

2005) and/or are considered "high-level" (i.e., matrix reasoning Soulières et al., 2009). Enhanced perception and manipulation of specific materials might predispose autistic individuals to develop savant abilities, given the opportunity, as in 3-D drawing or musical improvisation. However, as with proposals based on detailed-focused cognitive styles and/or hyper-systemizing, there is insufficient evidence as to how enhanced perception leads to the development of savant abilities (Dawson et al., 2008).

To address this shortcoming, the EPF model has recently been extended via a proposed veridical mapping (VM) mechanism, which represents an attempt to address how and why savant and other related unusual abilities develop in autism (Mottron et al., 2009; Mottron et al., 2013). VM is a capacity to detect regularities within and across isomorphic structures (i.e., structures sharing perceptual or structural similarity), at multiple scales. The materials involved in savant abilities are often structured human codes (e.g., written, arithmetical, and musical structures) which are also multi-level and redundant (i.e., sentences composed of words composed of letters, songs composed of melodies composed of notes, years composed of months composed of days). These materials exist across multiple scales, from very low-level or simple to very high-level or complex, and can be seen as highly isomorphic.

Veridical mapping in autism may arisefrom the combined effect of enhanced low-level perceptual abilities and superior mid-level ability in manipulating complex patterns. As a mechanism, VM would allow autistics to flexibly detect complex repetitive occurrences within human codes, while operating a parallel mapping of their constituents with other structures sharing some perceptual or structural similarity. It allows the memorization of the coupling between homologous elements of these structures, as in graphemephoneme coupling evident in hyperlexia. A basic example of VM in autism can be found in the autobiographical account of GT, a 9 year-old autistic boy, who could make outstanding weight, height, and distance estimations. GT reported that he used the recurrent mapping of a weight of a cereal bar of 35 g to estimate weights under 10 kg, which was confirmed by his contrasting accuracy in estimation of weights superior versus inferior to this amount (Soulières et al., 2010).

We have also proposed (Mottron et al., 2013) that the same VM mechanism might also account for the plausibly superior prevalence of synesthesia (Baron-Cohen et al., 2013; Neufeld et al., 2013) and absolute pitch (DePape et al., 2012) in autism. This proposal draws on multiple observed similarities or overlaps connecting the two abilities. Synesthesia is condition where "attribute of a stimulus (e.g., its sound, shape, or meaning) may inevitably lead to the conscious experience of an additional attribute" (Ward, 2013, p.50). This definition has been quite useful to describe the most studied forms of synesthesia, such as colored perception of alphabet letters or musical notes. Recently this definition has been questioned due to the rising number of different forms of synesthesia (61 according to Cytowic et al., 2009), some of which involve more elaborate cognitive traits. For example, some people associate a specific personality with a certain number or letter (Simner and Holenstein, 2007; Smilek et al., 2007), which is called ordinal linguistic personification (OLP). Thus, Simner (2012) recently proposed a new

definition: "Synaesthesia is characterized by the pairing of a particular triggering stimulus with a particular resultant experience." (p. 12).

Absolute pitch is defined as the ability to name or otherwise indicate notes without reference to an external standard. Synesthesia and absolute pitch share some common neural mechanism (Loui et al., 2012). The association between note and label in absolute pitch possessors is automatic (Akiva-Kabiri and Henik, 2012; Schulze et al., 2012), as is the case with associations in synesthesia (Ward, 2013). Genetic linkage and co segregation between absolute pitch and synesthesia has also been observed (Gregersen et al., 2013). Synesthesia possessors display characteristics neighboring those observed in autism, such as superior perceptual capacities (Banissy et al., 2009; Goller et al., 2009) and superior mental rotation in time-space synesthetes (Simner et al., 2009; Brang et al., 2010, 2013) who also show memory benefits that could be linked to savant abilities (Simner et al., 2009). Further, the cross modal retrieval of a concurrent element (i.e., the resultant synesthetic experience) when its inducer (i.e., the element that elicits synesthesia) is perceived in synesthesia has similarities with the possible role in savant abilities of redintegration, which is the non-strategic recall of a missing element in the presence of its homolog (Mottron et al., 2013). First-hand accounts document the co-occurrence of synesthesia and savant syndrome, as in the case of Daniel Tammet (Bor et al., 2007). It has been suggested by some authors that when autism and synesthesia co-occur, the probability of savant syndrome is increased (Baron-Cohen et al., 2007; Simner et al., 2009). These authors proposed that an over-rehearsal related to the repetitive nature of interest in the autistic population can lead to the development of savant abilities.

In this paper we present a case study of FC, a savant autistic adult, which may cast a new light on the nature of this link. FC is exceptional at several levels. First, he possesses multiple perceptual and non-perceptual savant abilities, among which some (e.g., calendar calculation) are in classical savant areas but others (e.g., absolute pitch) are marginally if at all considered to be savant abilities. Second, despite limited verbal abilities, FC is able to provide spoken accounts of some of his methods, providing exceptional information on the way they progress with time. Third, FC also possesses some synesthesia-like associations. We will report a description and empirical study of FC's abilities as well as his own accounts of the acquisition of these abilities.

# **CASE REPORT**

# **DEVELOPMENTAL HISTORY**

FC was born by cesarean section to a 23-year-old mother following an uncomplicated 42-week pregnancy. He weighed 2.62 kg and measured 47 cm at birth. He has one older brother and one younger sister who are both typically developing. His mother is the youngest of a family of three and his father is the 9th child from a family of 10. One cousin on his father's side presents an intellectual disability and another suffers from major thalassemia; there was nothing relevant to report on his mother's side. Both parents operate a restaurant, although his father is trained as an electrician.

FC's first year of life was marked by repeated ear infections with bilateral myringotomy and, retrospectively, atypical calmness and hypo-activity. Motor milestones were unremarkable. At the age of two, his parents suspected atypical development based on absence of speech, as well as presence of swaying and excessive quietness. A 2-week stay in a neurological hospital revealed normal physical and auditory functions, but a suspected childhood psychosis, a frequent diagnosis for French autistic toddlers at the time. FC started attending a nursery school at the age of three. By 5 years of age, FC still did not utter a single word but used his parents' hands to communicate his needs, or emitted small noises. His first word was his brother's name and by age six, his vocabulary was only 10 words. At this age, immediate echolalia appeared and FC's spoken language then evolved from echolalia to standard use of speech. Between 5 and 10 years, he received regular speech therapy support, attended a mainstream school, and followed a specific program in an outpatient clinic for children with learning disability. FC was in this period able to read without apparent understanding of what he was reading, but by the age of 10 was able to read, write, and count typically. He used to cover his ears with his hands or run away when he was confronted with unknown or loud sounds.

The diagnosis of autism was given at 11 years and 7 months by a professional clinician, on the basis of above-threshold past and current ADI (Le Couteur et al., 1989) algorithm scores in the four areas of social interaction, communication, restricted interests and repetitive behaviors and age of concern. His CARS score (Schopler et al., 1980) at this age was 31, above the threshold for autism. As a teen, he was integrated in a specialized school for neurodevelopmental disorders.

FC was 21 years old at the beginning of testing and 26 years old at the time of writing. He is now working in an establishment for disabled people, doing repetitive work. He gives brief eye contact and socially oriented smiles, but in an atypical way, and also experiences tics (hissing like a snake or looking at his watch). When he speaks, under a certain insistence, he uses stereotyped verbal expressions with some verbal apraxia. An intellectual assessment using WAIS-III at the age of 22.2 years (see **Table 1**) indicated verbal comprehension and processing speed index scores in the intellectual disability range (under the 2nd percentile), but normal-range perceptual organization and working memory index scores. In contrast, FC obtained a raw score of 55 (95th percentile, 1985 norms), in the range of superior intelligence, on Raven's Progressive Matrices, a major test of fluid intelligence which may provide the best estimate of autistic intelligence.

During interviews, FC was able to answer questions about how he acquired his unusual abilities and the mechanisms that underlie them. We video-recorded these interviews and transcribe here the explanations he gave us, along with ecological descriptions of behaviors related to his abilities, and test results in four areas: absolute pitch, memory skills, computational ability, and synesthesia. Because FC possesses tics and has difficulty in understanding rapid instructions, tests were developed to evaluate the stability of the described associations, using discrimination tasks. Therefore, while we provide his reaction times (RTs) on different tests, as they may to some extent reflect how he processes information, they cannot be taken as unambiguous or definitive in this respect.

### **Table 1 | FC's IQ profile from theWAIS-III.**


### **ABSOLUTE PITCH** *Related behaviors*

FC possesses absolute pitch, that is, he is able to name a note without external reference. Despite never playing the domestic organ before the age of nine, FC's parents noticed that at this age he was able to reproduce a song on the organ immediately after hearing it on the radio. After this age, FC displayed numerous music- and sound-related occupations and activities. For example, he possessed a small electronic piano that he brought everywhere. He also recorded his voice and what he was playing on a portable recorder and played it repetitively. He tried different instruments, including the guitar, flute, accordion, violin, and drums, but the piano was his favorite instrument. FC plays both classical and modern songs, such as Bach's Toccata and Fugue in D minor BWV 565 and the John Lennon song "Imagine." He also composed a melody, mostly of consonant third intervals, which he named "Repezik" and played over and over – "because that made me very pleased," he reported. FC took a few piano lessons at the age of 10, but at the time was overtly more interested in the sound of the piano than in playing the instrument. FC is now taking piano lessons and is attracted by composers like Bach or Handel. According his teacher, he possesses superior abilities to an adult amateur player, particularly his memory for and immediate reproduction of melodies. He plays with his two hands independently but has more difficulty with rhythm.

# *Self-report*

We asked FC how he knew that the note C corresponds to the C key. FC showed us the keyboard. Starting with the C key, he explained "C is Monday, D is Tuesday, E is Wednesday, F is Thursday, G is Friday, A is Saturday, and B is Sunday." Then he added, while pointing to the right proximal C key on the keyboard "Next C is next Monday" and pointing to the next C key on the keyboard, said "next C and next Monday". Then he pointed to the left proximal C key and explained "last Monday," and the next left C key, indicating "Monday before."We then asked him to which days the black keys correspond. Starting with the C key, and while pointing to each of the 12 keys composing an octave, he said "January, February, March, April, May, June, July, August, September, October, November, December. And everything from there (the C key) until there (the next right B flat key), is a year. And from there (C) until there (the next right B flat key), these are two years. And from there (C key three octaves higher) until there (next B flat key), it makes five years."When we asked him whether somebody taught him or he developed this by himself, he replied, "Me, because it's in my head." An illustration of this association is provided in **Figure 1**.

# *Test results*

FC's receptive absolute pitch was assessed through the identification of 60 musical notes without reference note (Vangenot, 2000). Each note lasted 1000 ms, with an ISI of 2000 ms. In order to prevent the use of relative pitch, there was more than one octave between each note. He orally gave the name of the note. Absolute pitch was also tested in production, by asking him to sing 12 notes when provided with their names. His production was recorded and compared to reference musical notes. Both FC's identification and production were 100% correct. His capacity to identify notes embedded in chords was also tested. There were three trials per chord of different numbers of notes, starting by three and up to six notes (12 different trials). His identification of notes within chords up to four notes was 100% correct. For chords composed of more than 4 notes, FC did not make any false identifications but reported only a subsample of the notes composing the chord.

FC's idiosyncratic note-calendar mapping was tested receptively using 14 1000 ms notes ranging from B3 flat to D5 sharp, produced with Final® software, and a computer screen presentation of days of the week and months of the year. In each trial, a day (or a month) was visually presented, together with a note presented aurally. The association could be either congruent or incongruent per FC's system of mapping. Days and months were visually presented (in 24-point Arial font) in the center of the screen and remained presented until a response was given. Trials were interleaved with a blank screen for 2000 ms.

There were 168 trials in the notes-days experiment. According to FC's mapping, half (84) were congruent and half were incongruent. In the incongruent trials, days were presented with a note that could be higher or lower by one tone compared to the congruent note. For example, Thursday, associated with the F note in FC's mapping, was associated with the E note and the G note in incongruent trials. Similarly, there were 192 trials in the notes-months experiment, half (96) congruent and half incongruent according to FC's mapping. In the incongruent trials, months were presented with a note either one semitone higher or lower than the congruent note. For example, March was associated with the D note in FC's mapping and therefore in congruent trials, while March was presented with a C sharp or D sharp note in incongruent trials. Days and months were pseudo-randomly selected in the material list to avoid the consecutive presentation of the same day or month. FC was asked if the sound presented corresponded to the correct day or month and responded by pressing with his right hand a green button for accurate (congruent) correspondences and a red button for the inaccurate (incongruent) correspondences. The experiment was run with PsychoPy software (Peirce, 2007).

FC was able to correctly identify 100% of the congruent and incongruent day-note associations (see **Table 2**). His RT for congruent trials (after removing 3SD outliers, which represented 4.76% of the trials) was 2.369 s (SD = 0.446). His mean RT for incongruent trials (after removing 3SD outliers which represented 1.19 % of the trials) was 2.309 s (SD = 0.475). There was no difference between congruent and incongruent conditions on mean RT (*p* > 0.4). In the month-note experiment, FC made 6 mistakes and was able to identify correctly 96.8% of trials in the congruent condition, and 96.8% of trials in the incongruent condition. His mean RT for congruent trials (after removing

### **Table 2 | Summary of FC's performance.**


errors and 3SD outliers, which represented 14.58% of the trials) was 2.397 s (SD = 0.428). His mean RT for incongruent trials (after removing errors and three SD outliers, which represented 9.37% of the trials) was 2.369 s (SD = 0.418). There was no difference between congruent and incongruent conditions on mean RT (*p* > 0.5). FC can therefore redintegrate the missing element in pairs of homolog elements with a speed and accuracy poorly compatible with a strategic or algorithmic computation.

# **CALENDAR CALCULATION**

# *Related behaviors*

FC is a calendar calculator: he can tell which day of the week corresponds to a date (day, month, year). His parents were unaware of this ability before our study, but they reported that FC had been interested in calendars, dates, and time since the age of six.

# *Self report*

We asked FC how he performed calendar calculation. "I'm doing like this," he said and then he turned toward a piano, pointed to the keys, and said "I'm thinking with musical notes. C is Monday, D is Tuesday, E is Wednesday, F is Thursday, G is Friday, A is Saturday, and B is Sunday. And these seven notes are one week." We asked him if he thinks using the sound of the note or the name of the note. He said "I think it's like musical note, with the name of the musical note. Like this it's easier." It seems therefore that he uses a synesthetic correspondence as a support for his calendrical computation ability.

# *Test results*

In 2011, we questioned FC on the weekday of 10 past (from year 1993 to year 2011) and 10 future (from year 2013 to year 2031) dates. Questions did not involve the same month more than twice, and correct answers did not fall on the same weekday more than twice. FC made one mistake for the past dates (90% accuracy) and one mistake (90% accuracy) for the future dates. We also asked him 10 reversed questions for past dates (from year 1991 to year 2009) of the type "what are the months having Monday the fifth in 2007?" There were a total of 20 correct answers possible; FC made one mistake and three omissions (80% accuracy).

# **COMPUTATION ABILITY**

# *Related behaviors*

FC can mentally add, subtract, and multiply numbers up to four digits quite rapidly, clearly above his verbal level. His parents reported that FC mastered multiplication tables at the age of seven and knew how to divide numbers at the age of eight. FC then asked for a calculator that he brought with him everywhere. At eight years old, he also showed a great interest in time; he possessed a watch, and knew how to read time.

# *Self report*

FC spontaneously explained his method of performing mental calculation. "I calculate with hours, minutes, and seconds, like this it's easier." He elaborated: "I did like this (for the addition 1728 + 2932 = 4660): 48 min and 52 s plus 28 min and 28 s equal 1 h 17 min and 40 s."

# *Test results*

Forty numbers were randomly chosen between 1200 and 14,000. The time correspondence of these numbers was then calculated (e.g., 3520 = 58 min and 42 s). For incongruent trials, the time correspondence was modified by ±10 min. (e.g., 3520 = 52 min and 10 s). There were 80 trials: in half (40 trials), numbers were presented with the correct time correspondence whereas in the other half, numbers were presented with an erroneous time correspondence. For each trial, one number and one time (in 24-point Arial font) were presented in the center of a computer screen until FC gave his response by pressing with his right hand the corresponding button "c" for the correct association and "n" for the incorrect association. The experiment was run with PsychoPy software (Peirce, 2007).

FC correctly identified congruent and incongruent associations in 100% of the trials (see **Table 2**). His mean RT for congruent trials (after removing 3SD outliers, which represented 2.56% of the trials) was 6.244 s (SD = 2.14). His mean RT for incongruent trials (after removing 3 SD outliers, which represented 4.87 % of the trials) was 6.725 s (SD = 2.01). There was no difference between congruent and incongruent conditions on RT (*p* > 0.3).

# **SYNESTHESIA-LIKE MANIFESTATIONS** *Related behaviors*

FC frequently referred to months of the year to verbalize his emotions or sensations. His parents reported FC saying, for instance, "it hurts like a month of March and half" or that he is happy "like the month of June." FC can describe his own feeling or emotion

with these associations, and asks his parents for an interpretation of certain situations using this month-emotion correspondence (e.g., "like which month are you happy?"). Some numbers can also provoke in FC pleasant sensations.

# *Self report*

When questioned about the month/emotion correspondence, FC explained that "January is very very bad, February is very bad, March is bad, April is good, May is very good, and June is very very good." When asked about the number/emotion correspondence, FC reported that some numbers can be "nice or not nice" or that they can "matter or not matter." In addition, the evocation (or the vision) of certain numbers cause him a physical sensation, as sometimes he reacted to certain numbers as though to a tickle, according to his parents and to our own observations. FC spontaneously provided a graphic representation of how a subset of numbers can be classified as a function of their "kindness" (nice / not nice) and their "mattering" (matters/does not matter; see **Figure 2**). Thus, four categories of emotional valence emerge from FC's classification. In order to establish a more systematic representation of FC's categorization of numbers, we questioned him on which emotional valence (i.e., category) each number is associated with. At the beginning, numbers were asked individually and in order. After number 10, only numbers close to the categories' boundaries were asked. A representation of the architecture of FC's categorization is given in **Table 3**. FC's synesthesia-like categories are structured by certain rules. Number of digits in each successive category is multiplied by two (between number 3–4 there are 2 digits, between number 5–8 there are 4 digits, between number 9– 16 there are 8 digits, from number17–32 there are 16 digits, and so on...). Categories are presented systematically: *not nice/not matter, nice/not matter, not nice/matter*, *nice/matter*. When FC arrived at the number 8,388,608, he declared he wanted to stop.

# *Test results*

Thirty-two numbers of one, two, or three digits were selected at the boundaries of FC's categories. Numbers were pseudo-randomly selected in the material list in order to avoid the presentation of

the same number or same category twice in a row. Each number was presented three times in association with the correct category and three times with the category of the closest number outside the category, for a total of 96 congruent and 96 incongruent trials. For example, in the congruent condition *15* was presented with the correct category *not nice and not matter* and *17* was presented with the correct category *nice and not matter*; whereas in the incongruent condition *15* was presented with the category *nice and not matter* and *17* was presented with the category *not nice and not matter*. The experiment was run with PsychoPy software (Peirce, 2007). In each trial, a number was visually presented with the written name of the congruent or an incongruent category. Numbers and categories were visually presented in the center of the screen and remained presented until a response was given. Number and categories were written in black in 20-point Arial font on a white background, with a 2000 ms blank screen ISI. FC was asked to press a green button for the correct or congruent category and a red button for incongruent categories with his right hand.

FC was 97.9% accurate according to his own classification (see **Table 2**). His four errors were in the incongruent condition (95.8% of correct identification). RTs were also analyzed. Errors and RTs over 3 SD were excluded from analyses (13.5% of trials). A T-test was performed, finding with marginal significance longer RTs for congruent (*M* = 2.331 s, SD = 0.752) compared to incongruent (*M* = 2.041 s, SD = 0.471) trials, *t*(31) = −2.03, *p* = 0.051. However, as noted above, RTs might not straightforwardly reflect FC's information processing because of his tics and his difficulty in understanding rapid instructions.

# **DISCUSSION**

This paper provides a naturalistic, empirical, and autobiographical report of a multi-talented autistic savant who is willing and able to comment on his skills. We will now discuss to what extent his savant abilities are related to what is called synesthesia in nonautistic people, how this is beneficial to his performance, and how it supports the role of VM in the acquisition of savant abilities in autism.

# **IS FC ACTUALLY A SYNESTHETE?**

FC's abilities can be linked to synesthesia. Indeed, FC's number/valence attribution bears some resemblance with OLP synesthesia (Simner and Holenstein, 2007). In OLP synesthesia, the inducer is a grapheme, generally a number or a letter, and the concurrent is a specific personality. In some cases, objects can also be associated with a specific personality (Smilek et al., 2007). Persons reporting OLP synesthesia have provided detailed descriptions of the personality associated with graphemes. For example, TE, a 17 year-old woman, reported "Three is such a jerk! He only thinks of himself. He does not care about any other numbers or anything" (Smilek et al., 2007, p. 981). Despite obvious differences in verbal complexity that may be related to differences in verbal IQ, TE's description is similar to FC's number/valence associations.

Moreover, FC's accounts provide evidence that the correspondence between notes and days or months emerged after encountering a specific material during childhood. Synesthesia, as with absolute pitch, is acknowledged to be influenced by

### **Table 3 | FC's categorization of numbers.**


childhood experience (Chin, 2003; Simner et al., 2005). This is well-established in the case of the influence of early manipulation of colored alphabets on further grapheme-color synesthesia (Witthoft and Winawer, 2006, 2013). In the case of OLP synesthesia, learning and previous childhood experience might also have shaped the mapping between grapheme and personality types (Simner and Holenstein, 2007; Smilek et al., 2007). Witthoft and Winawer (2013, p. 6) recently argued that synesthesia involves not only perception but also learning and memory: "associative learning and the perceptual experiences of synaesthetes are not only compatible, but also lie on a continuum with ordinary experience."

Because of FC's tics and difficulty in understanding rapid instructions, we were unable to provide evidence for the automaticity of his associations, this being one specific characteristic of synesthesia (Simner and Holenstein, 2007). However, we were able to provide evidence that FC's associations are stable. FC's associations could still reflect cross-modal correspondences (Spence, 2011), which share certain similarities with synesthesia and pertain to the same continuum (Martino and Marks, 2001). One main point of dissociation is that, unlike synesthetic associations, cross-modal correspondences are not believed to be idiosyncratic. FC's associations are clearly idiosyncratic. Moreover, automaticity

might not be specific to synesthesia, as it has also been observed in cross-modal correspondences (Deroy and Spence, 2013) as well as in induction of verbal notes labels when hearing notes in some absolute pitch possessors. It should be also emphasized that characteristics such as automaticity have been defined in the context of synesthesia occurring in non-autistic people, and therefore that some kind of modification of their nature might be expected when synesthesia occurs in an autistic context. Indeed, the extreme regularity of FC's associations is not usually observed in synesthesia. This regularity might be the expression of FC's autism, or just the case of an extreme synesthetic manifestation.

# **HOW PERCEPTION HELPS INTELLIGENCE**

FC's way of computing mental additions or multiplications may not appear to be the most economical way, according to typical standards. However, FC declared that this way was easier for him, and it results in accurate operations. This suggests that the correspondence between time units in base 60 up to the level of the second, as well as reciprocal transcoding between base 10 and base 60, is for FC effortless, fast, and stable. A similar gain through fast, cross modal correspondences is found in synesthesia. The ability to map numbers onto a complex combinatorial lexicon of

colors and emotions, although out of reach for most people, may result in exceptional memory performances. Rothen et al. (2012) report some cases of exceptional memory in which elements are remembered using variants of the "loci" method. This is the case for Daniel Tammet, an Asperger synesthete with unique learning abilities. Tammet reported that numbers up to an integer of 10,000 have unique shapes, colors, textures, and feels. A list of numbers creates the experience of a complex landscape (Tammet, 2006). The access to such a multimodal experience appears to differ from types of memory facilitation in use in typical individuals. For instance, the chunking of verbal material in memory tasks helps typical individuals, but is not beneficial to Tammet (Bor et al., 2007). Rothen et al. (2012) assumes that Tammet possesses his own internal organization, which helps him to remember long series of digits without interference, but he cannot benefit from a strategy imposed by other minds.

That synesthesia reflects a functional combination of perception, learning, and memory is informative on its role in the acquisition of savant abilities and synesthesia in autism. This perceptual mapping of elements might reflect both an autistic way to create sense from the environment, and a way to feel positive emotions in doing so. Another related possibility would be that FC's synesthesia allows the coding and further manipulation of pattern-like information naturally presented in a non-discrete form, through its mapping onto mathematical or linguistic structures composed of discrete elements. In this case, synesthesia is used as an investigational tool, top-down or bottom-up or nonhierarchical as necessary, enhancing how an autistic person can manipulate, categorize, and make sense of the world. This is consistent with the enhanced role of perception in intelligence, which represents one of the main messages of neuroimaging studies of autistics (Soulières et al., 2009), and extends far beyond a bottomup, passive heuristic. It can also be manifested in situations where perceptual structures are used as a creative and investigative tool, for example to crack the codes of emotions and less obviously structured materials.

### **VERIDICAL MAPPING AND SAVANT ABILITIES**

This report of savant abilities in an autistic adult, FC, benefits from his capacity to document some psychological processes associated to his performances and interests. FC's performance, his history, and his own intriguing accounts are consistent with the plausible implication of VM, across large-scale isomorphic structures and applied to various materials (numbers, emotions, pitches, musical notation), in the genesis of savant abilities. VM appears to be spontaneous, non-strategic, and bidirectional (or even non-directional), for example between structures that are already represented in the person's mind, or as a way to collect, organize, understand, and to manipulate new material. Thus this mechanism may contribute to autistic learning, creativity, and everyday information processing.

The notion of veridicality implies that multiple and sufficiently similar occurrences of a structure, across levels and scales, allows the non-hierarchical and non-strategic emergence of a mapping between homologous elements. This is illustrated by FC's reports on his acquisition of absolute pitch and computation ability. He first associated the seven notes of the musical scale with the seven

weekdays, and later on, the 12 semi-tones of the chromatic musical scale with the 12 months of the year. While differing in their substrates, both types of structure share the same number of elements and an ordering constraint, and thereby are highly veridical one to the other. Regarding his mental computation ability, FC computes arithmetical operations initially expressed in base 10 by transposing their constituents into base 60, performing computations in this system, then accurately transposing the result back into base 10. He therefore makes use of isomorphisms between base 60, which structures time measurement, and base 10, which structures arithmetic. This mapping is veridical, in the sense that the same operations can be performed in the two bases, allowing an accurate transformation of an operation expressed in one base to another base. For these two exceptional abilities, the structural similarity is in the object (thereby, is veridical) and not in the eye of the beholder, even if the mapping may appear as an idiosyncratic selection of one structural similarity among multiple others possible.

Critically, FC seems to possess the three main types of abilities – savant syndrome, absolute pitch, and synesthesia – that all, according to our model, plausibly represent an expression of similar or equivalent neurocognitive mechanisms. The micro-structural alterations (Kéïta et al., 2011), enhanced or atypical connectivity (Uddin et al., 2013) involving perceptual areas, and cortical rededication (Samson et al., 2012) characterizing autism could plausibly support a VM mechanism, with various consequences – such as absolute pitch and musical talent, synesthesia, mental computation, calendar calculation – depending on individual variability, availability of materials, and opportunities.

# **ACKNOWLEDGMENTS**

We are greatly indebted to FC and his family. We also thank Rachel Zoubrinetzky for her contribution to this study.

# **REFERENCES**


Uddin, L. Q., Supekar, K., Lynch, C. J., Khouzam, A., Phillips, J., Feinstein, C., et al. (2013). Salience network–based classification and prediction of symptom severity in children with autism. *JAMA Psychiatry* 70, 869–879. doi: 10.1001/jamapsychiatry.2013.104

Vangenot, S. (2000). L'oreille absolue: une oreille plus fine? *Mus. Sci.* 4, 3–30.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 May 2013; accepted: 26 January 2014; published online: 18 February 2014.*

*Citation: Bouvet L, Donnadieu S, Valdois S, Caron C, Dawson M and Mottron L (2014) Veridical mapping in savant abilities, absolute pitch, and synesthesia: An autism case study. Front. Psychol. 5:106. doi: 10.3389/fpsyg.2014. 00106*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Bouvet, Donnadieu, Valdois, Caron, Dawson and Mottron. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Synaesthesia and sexuality: the influence of synaesthetic perceptions on sexual experience

### *Janina Nielsen1, Tillmann H. C. Kruger 1, Uwe Hartmann1, Torsten Passie1,2, Thorsten Fehr <sup>3</sup> and Markus Zedler <sup>1</sup> \**

*<sup>1</sup> Department of Clinical Psychiatry, Social Psychiatry, and Psychotherapy, Hannover Medical School, Hannover, Germany*

*<sup>2</sup> Department of Psychiatry, Harvard Medical School, Harvard University, Boston, MA, USA*

*<sup>3</sup> Department of Neuropsychology, Center for Advanced Imaging Bremen/Magdeburg, Center for Cognitive Sciences, University Bremen, Bremen, Germany*

### *Edited by:*

*Clare Jonas, University of East London, UK*

### *Reviewed by:*

*Noam Sagiv, Brunel University, London, UK Danko Nikolic, Max Planck Institute for Brain Research, Germany*

### *\*Correspondence:*

*Markus Zedler, Department Clinical Psychiatry, Social Psychiatry, and Psychotherapy, Hannover Medical School, Carl-Neuberg-Str. 1 (OE 7110), 30625 Hannover, Germany e-mail: zedler.markus@ mh-hannover.de*

**Introduction**: Synaesthesia is a phenomenon in which a certain stimulus induces a concurrent sensory perception; it has an estimated prevalence of 4%. Sexual arousal as an inducer for synaesthetic perceptions is rarely mentioned in the literature but can be found sometimes in case reports about subjective orgasmic experiences.

**Aims**: To examine whether synaesthetic perceptions during sexual intercourse have an impact on the sexual experience and the extent of sexual trance compared to non-synaesthetes.

**Methods**: In total, 19 synaesthetes with sexual forms of synaesthesia (17 female; 2 male) were included as well as corresponding control data of 36 non-synaesthetic subjects (*n* = 55). Two questionnaires were used to assess relevant aspects of sexual function and dysfunction (a German adaption of the Brief Index of Sexual Functioning, KFSP) as well as the occurrence and extent of sexual trance (German version of the Altered States of Consciousness Questionnaire, OAVAV). Additionally qualitative interviews were conducted in some subjects to further explore the nature of sexual experiences in synaesthetes.

**Main Outcome Measures**: Sexual experience and extent of sexual trance during intercourse.

**Results**: Synaesthetes depicted significantly better overall sexual function on the KFSP with increased scores for the subscale "sexual appetence" but coevally significant lower subscale scores for "sexual satisfaction." Sexual dysfunction was not detected in this sample. Synaesthetes depicted significantly higher levels of the subscales "oceanic boundlessness" and "visionary restructuralization" than controls using the OAVAV. Qualitative interviews revealed varying synaesthetic perceptions during the different states of arousal. Furthermore, synaesthetes reported an unsatisfactory feeling of isolation caused by the idiosyncratic perceptions.

**Conclusions**: Synaesthetes with sexual forms of synaesthesia seem to experience a deeper state of sexual trance without, however, enhanced satisfaction during sexual intercourse.

**Keywords: synaesthesia, sexuality, sexual satisfaction, sexual appetence, oceanic boundlessness, visionary restructuralization, sexual trance, OAVAV**

## **INTRODUCTION**

Synaesthesia is a neural condition in which a certain stimulus, called inducer, elicits a concurrent perception; for example, listening to music can induce the visual perception of colored patterns or letters and numbers are perceived in specific idiosyncratically associated colors. Current estimates consider synaesthesia to be more common than previously assumed with prevalence at 4% (Simner et al., 2006). According to Day (2004) the most common form is grapheme-color synaesthesia with prevalence at 68.8% among synaesthetes. Orgasms as inducers for colored visions are considered less frequent with a prevalence of 2.15%. In some cases flavors as synaesthetic concurrents induced by orgasms are mentioned as well. Sexual types of synaesthesia as described below do not only contain orgasms and sexual arousal but also more general tactile inducers such as touching, caressing, and petting (i.e., all forms of physical contact during sexual intercourse). Therefore, sexual synaesthesia actually contains several synaesthetic types. Touches as synaesthetic inducers are mentioned by Day as well. There are described many different concurrents such as colors (4.08%), flavors (1.07%), smell (0.43%), sounds (0.32%) or temperatures (0.11%). Generally, in literature sexual types of synaesthesia are rarely mentioned. Cytowic (1989) notes that in some cases "kissing and sexual intercourse is a reliable trigger (*...*), causing colored photisms, tactile shapes and textures and tastes"(p. 34). Altogether, this special form of synaesthesia has been neglected so far in scientific research, probably due to the low rate of prevalence and the difficulties in investigating this highly subjective phenomenon.

Outside synaesthesia research, synaesthetic experiences during sexual activity have also been described by sex researchers. Fisher (1970) conducted an interview study to investigate female orgasmic experiences. The words of one subject bear remarkable resemblance to the descriptions by synaesthetes: "(*...*) the only images I ever experienced at this time and during orgasm is a fuzzy blackness with red or white muted bursts coming through it (*...*)"(p. 207). The phenomenological description is similar to descriptions of audio-visual forms of synaesthesia as the visualized structures and colors are not static but dynamic and alternating (Haverkamp, 2009). Mosher (1980) refers to this citation as an example of highly extended sexual trance which is defined as "(*...*) an altered state of consciousness (*...*) representing a qualitative shift in the pattern of psychological functioning that differs from the norm of alert, waking consciousness" (p. 11). In his model "Three dimensions of depth of involvement in human sexual response" sexual trance is one of three postulated dimensions essential to a maximum extent of depth of involvement which in turn is a condition for efficient sexual stimulation. The maximum extent of sexual trance leads to a state of total absorption. The other dimensions are "role enactment" and "engagement with the partner." According to Mosher, an equilibrium between those three dimensions is a condition for a maximum extent of depth involvement and thus, for sexual satisfaction. According to Swartz (1994) absorbed states of consciousness—sometimes called altered states of consciousness (ASCs) or sexual trance are a common phenomenon during sexual arousal and play an important role in sexual responses. Regarding general, inducerindependent criteria for ASCs, Ludwig (1966) postulates synaesthesia may be an important feature of ASCs. Shanon (2003) also discusses the occurrence of synaesthesia in ASCs which seems to be a common phenomenon. Tellegen and Atkinson (1974) who investigates the "high absorption person" describes the inherent ability "to operate diverse representational modalities synergistically so that a full but unified experience is realized" (p. 275) which he claims is a cognitive accomplishment of the absorption trait. He calls this integrative phenomenon "syngnosia" (p. 275), a term used analogous to "synesthesia" which he suggests is one of its components.

Assuming that synaesthesia may be an important constituent of ASCs and that several parallels between synaesthesia and the ability to achieve sexual trance can be found the consequent hypothesis is that synaesthetes with sexual forms of synaesthesia (i.e., synaesthetic perceptions through any sexually experienced inducers) may experience a higher degree of sexual trance during intercourse. The present study investigated this hypothesis by comparing the sexual experiences of a group of synaesthetes with a control group. A further hypothesis concerns the sexual satisfaction of synaesthetes with sexual forms: According to Mosher's (1980) theory that sexual trance is one basic criterion for high sexual involvement, it is assumed that synaesthetes are able to experience a greater sexual satisfaction than a non-synaesthetic standardization sample.

# **MATERIALS AND METHODS**

# **STUDY PARTICIPANTS**

Nineteen synaesthetes (17 female, 2 male) were recruited via internet and through an already existing data pool of the synaesthesia work group at the Hannover Medical School (MHH). All subjects were assumed to be psychiatrically and physically healthy which was based on enquiry during acquisition phase. Only synaesthetes who reported synaesthetic perceptions during sexual experiences were included. Synaesthesiae included visual perceptions such as colored shapes. There didn't seem to be any essential differences between the synaesthetic experiences of the male and female subjects i.e., both types of gender described generally similar perceptions. Coexisting synaesthetic forms ranged from emotionally induced ones to more common forms such as grapheme-color synaesthesia which was confirmed via the consistency test. This test was developed to investigate the consistency of mapping across time as it has previously been shown that synaesthetes show greater matching consistency than non-synaesthetes (e.g., when asked to match colors to graphemes and then being re-tested after a delay). It was designed via MatLab (Matrix LABoratory, Version 7.1.) by the MHH synaesthesia research group and is based upon the "synesthesia battery" by Eagleman et al. (2007). The inclusion criterion of a sexual form of synaesthesia was simply confirmed by interviewing the participants, as this form of synaesthesia cannot be verified through any standardized tests. This lack of assurance will be discussed in detail below.

The control group data for the KFSP (Reynolds et al., 1988; Mazer et al., 2000) stem from a data pool of sexual healthy women generated by the Clinical Psychology Department of the MHH. Due to the insufficient number of men in the synaesthetic group, the male participants were excluded from the KFSP research. The mean age of the female synaesthetes (37.06; *SD* ± 11*.*31) again was not statistically different from that of the controls (36.12; *SD* ± 10*.*17).

The data of the control group for the OAVAV (Dittrich, 1975, 1998; Passie, 2007) investigation was obtained in a parallel MHH study investigating gender-specific differences in sexual experience. The control group was matched pairwise by gender and age. The mean age of the synaesthetic group (36.53; *SD* ± 11*.*7) did not differ from the control group (34*.*2 ± 10*.*0) as measured by *t*-test.

The study was approved by the MHH Ethics Committee. The subjects were assured that any private information they gave would be kept confidential. By carrying out the procedure via mail the participation was kept anonymous. Each questionnaire was labeled with a code number so that the response rate could be followed. Each participant gave written consent about the participation.

# **PSYCHOMETRIC ASSESSMENT**

Two questionnaires were employed. First, the KFSP ("Kurzfragebogen zur Erfassung sexueller Probleme") is an instrument for the investigation of most relevant aspects of sexual function and dysfunction. The KFSP consists of two different versions; one especially developed for women (KFSP-F) and one for men (KFSP-M) but only the KFSP-F was implemented. The KFSP-F is theoretically based on, but is not a direct translation of the BISF-W ("Brief Index of Sexual Functioning for Women") which in turn is based on the BSFQ ("Brief Sexual Function Questionnaire for Men") (Reynolds et al., 1988). The KFSP-F contains four subscales: "Sexual satisfaction," "sexual appetence," "arousal/lubrication," and "orgasm ability."

Furthermore, the OAVAV was applied to investigate general forms of ASCs. The original version APZ ("Abnorme Psychische Zustände") is a standardized psychometric assessment of ASCs in humans developed by Dittrich (1975) and is available in English and German (Dittrich, 1998). The APZ consists of three dimensions ("oceanic Boundlessness," "dread of ego dissolution," and "visionary restructuralization"). During the process of validation the APZ was revised and renamed the OAV; later it was completed by two more subscales. Thus, the name of the latest version, OAVAV, is a composition of the capital letters representing the five subscales in German: Oceanic boundlessness, dread of ego dissolution, visionary restructuralization, reduction of vigilance, and auditive alterations (Passie, 2007). Each one of the subscales contains specific characteristics of ASCs according to Ludwig (1966). The items have to be rated on a visual analog scale from 0 up to 10. Subjects were instructed to fill out the questionnaire about 10 min after sexual intercourse. The German version of the OAVAV is validated in comprehensive studies (Dittrich, 1998). Independent two-sample *t*-tests were conducted for statistic comparison of the synaesthetic group and the control groups; the statistic analysis was accomplished via PASW (Predictive Analytics Software, version 18). A *p*-value of *p <* 0*.*05 (α-level) was considered as statistically significant.

Additionally, qualitative semi-structured interviews were conducted with approximately half of the synaesthetic subjects (*n* = 7) who were willing to participate in this part of the study. The interviews were supported by guidelines which were adjusted to the sexual response cycle by Masters and Johnson (1966), in order to investigate perception during different phases of sexual experience. The transcriptions of the interviews were analyzed by the "Grounded Theory" by Glaser and Strauss (1998) which is a systematic methodology to analyze written data. Instead of beginning with a theory, the basic idea is to develop a superordinate theory by extracting key points from the text and then subsequently create codes, concepts, and categories. The results will be presented exemplarily to complement the quantitative results.

# **RESULTS**

## **KFSP-F: SEXUAL FUNCTION AND DYSFUNCTION**

The KFSP total score was significantly higher in synaesthetes than in the controls, which reflects better sexual function in general. Accordingly, the subscale score for "sexual appetence" was significantly higher (*p* = 0*.*003) for synaesthetes. Surprisingly, the subscale "sexual satisfaction" was significantly lower for synaesthetes than for control subjects (*p* = 0*.*036). The subscales "arousal/lubrication" and "orgasm ability" did not reveal any differences between synaesthetes and control group (see **Table 1** for details).

### **THE OAVAV: ALTERED STATES OF CONSCIOUSNESS**

Synaesthetes achieved significantly higher scores for total OAVAV, as well as for the subscales "oceanic boundlessness" and "visionary restructuralization" (see **Table 2** for details). Since the latter contains visual perceptions as patterns or colors, the high "visionary restructuralization" score for synaesthetes seems to directly reflect their visual synaesthetic perceptions during sexual activity.

# **PERSONAL INTERVIEWS: THE SEXUAL RESPONSE CYCLE IN SYNAESTHETES AND PERSONAL SIGNIFICANCE OF SYNAESTHETIC PERCEPTIONS**

A detailed qualitative analysis according to the "Grounded Theory" by Glaser and Strauss (1998) could only be accomplished for three of the interviewed subjects, because the other participants would not comply with being recorded on tape during the interview, due to the intimate subject. By analyzing the resulting transcriptions two main categories could be extracted: (1) stages of sexual experience and (2) significance of the synaesthetic perceptions.

Concerning the first category, the synaesthetes reported different stages of synaesthetic experiences analogous to the different stages of the sexual response cycle by Masters and Johnson (1966). Not all synaesthetes were capable of allocating their synaesthetic perceptions to one of the different stages but variation in the appearance of their perceptions over time was observed in every subject. **Table 3** shows the five different stages of the response cycle [including the desire stage added by Kaplan (1974)] with exemplary citations.

Concerning the second category, a significant emotional compound of sexual synaesthesia was detected: All of the synaesthetes questioned described themselves as "emotional synaesthetes" [which is based upon the term "Gefühlssynästhesie" by Emrich (2009)], generally experiencing synaesthetic perceptions induced by emotional states. The accompanying emotions during each state of sexual excitement seemed to influence the synaesthetic perceptions. The degree of emotional influence on the synaesthetic perceptions can be illustrated by the example of one female synaesthete who describes her desire stage as orange while claiming that she experiences orange itself as quite aversive. On further enquiry, the subject admitted that her aversive perception during the desire stage is based upon a more general aversion to sexuality which she describes as low interest in sexual activity. Sexuality never played an important role in her life and this previously used to concern her when she compared herself with others. It is noteworthy that the synaesthetes reported almost consistently that they consider the synaesthetic experiences as enriching for themselves, but that at the same time it makes them feel isolated from their environment during sexual intercourse.

# **DISCUSSION**

By studying synaesthesia we can learn a lot about consciousness in general (e.g., different types of synaesthesia can provide us with clues for theories concerning the nature of consciousness). ASCs—such as synaesthesia—are a part of human consciousness and as such, they need to be studied in order to understand its nature (Sagiv and Frith, 2013).

Based on the various indications that synaesthesia and sexual trance seem to share certain phenomenological characteristics,

### **Table 1 | Sexual function in synaesthetic and non-synaesthetic females.**


*Results of the short questionnaire for sexual problems for women (KFSP-F): Comparison between two independent samples by t-test in the appending subscales; significance: \*p <sup>&</sup>lt; 0.05; \*\*p <sup>&</sup>lt; 0.01; degrees of freedom (df)* <sup>=</sup> *32 for all t-tests; effect size according to Cohen's d: Around 0.2* <sup>→</sup> *small effect, around* <sup>0</sup>*.*<sup>5</sup> <sup>→</sup> *medium effect, around 0.8 and larger values* → *large effect.*


# *Results of the altered states of consciousness questionnaire (OAVAV): Comparison between two independent samples by t-test in the appending subscales; significance: \*p <sup>&</sup>lt; 0.05; \*\*p <sup>&</sup>lt; 0.01; degrees of freedom (df)* <sup>=</sup> *34 for all t-tests; effect size according to Cohen's d: Around 0.2* <sup>→</sup> *small effect, around* <sup>0</sup>*.*<sup>5</sup> <sup>→</sup> *medium effect, around 0.8 and larger values* → *large effect.*

we hypothesized an increased degree of sexual trance and sexual satisfaction in synaesthetes. Indeed, synaesthetes depicted significantly higher levels of "oceanic boundlessness" and "visionary restructuralization" reflecting sexual trance and visual synaesthesia during sexual intercourse. However, contrary to our hypotheses, synaesthetes were less satisfied with their sexuality, possibly due to a lack of partner involvement regarding synaesthetic perceptions.

Regarding the possible neurobiological mechanisms of synaesthetic perceptions during sexual intercourse, an imbalance between prefrontal and limbic brain function may be an important factor. Some ASCs can be accompanied by transient frontal hypoactivity as made evident by Dietrich (2003). This overlaps with a former theory by Cytowic (1989), who proposed a limbic basis for synaesthesia due to insufficient inhibition of subcortical activity by cortical processes, which was supported


by tentative neurophysiological data from Schiltz et al. (1999). In a more recent study by Cohen-Kadosh et al. (2009) it was possible to induce synaesthetic experiences through hypnotic states which are also accompanied by transient changes in prefrontal functioning. Passie et al. (2004) suggested that hyperventilation during sexual intercourse may be a mechanism that intensifies the sexual experience due to cortical, in particular frontal inhibition, which can be seen as congruent with the assumption that synaesthetes are able to experience a deeper state of trance. This possible neurobiological link between synaesthetic proneness and intensification of sexual trance may be an explanation for the higher values for synaesthetes in the ASCs questionnaire although the present study certainly remains hypothetical on this topic due to the lack of actual neurophysiological data.

Concerning the postulated increased sexual satisfaction of synaesthetes, the employment of the KFSP-F showed a significantly lower level of sexual satisfaction in synaesthetes and significantly higher values in the subscale "sexual appetence" and the total KFSP-F score. Apparently the female synaesthetes were less satisfied despite increased sexual appetence and a possibly heightened sexual trance. One possible explanation for these, at first sight, contradictory results can be found in the theory by Mosher (1980) who postulates sexual trance as one of the three dimensions essential to maximum extent of depth of sexual involvement. The model proposes that equilibrium between the three dimensions is mandatory for sexual satisfaction, whereas over-emphasis of one dimension develops an assimilating effect and averts satisfaction. In the case of over-emphasis of the dimension "sexual trance" the other dimensions—"role enactment" and "engagement with the partner"—are neglected. The person becomes more introverted and separated from the environment which may impair partner involvement. Thus, the high degree of sexual trance in synaesthetes may provoke an increased focus on sensual experiences on the inside, instead of partner-related aspects. This explanation is supported by the results of the personal interviews in which the synaesthetes reported an isolating effect of their synaesthetic experiences, while at the same time they experienced their perceptions as enriching and pleasant for themselves.

We would like to emphasize that the results are restricted by some methodological limitations. First of all—as has already been noted above—the presence of sexual synaesthesia could not be confirmed via consistency test [a test to investigate the consistency of mapping across time according to the "synaesthesia battery" by Eagleman et al. (2007)]. Therefore, the results presented should be treated with care.

This lack of evidence is due to the fact that sexual synaesthesia obviously cannot be tested in laboratory conditions by using the original inducers. However, it is still outstanding if there is a possibility of testing this form of synaesthesia without the original inducer which would imply a more semantic nature of the stimuli (i.e., ideaesthesia) (Jürgens and Nikolic, 2012). Concerning the question whether or not sexual synaesthesia could be considered as ideaesthesia, the reports of the interviewed synaesthetes show contradictory results. Thus, some of them reported more sensory based inducers which seem directly connected to perceptual stimuli (e.g., caressing). For them it was consequently difficult during the interview to remember the exact appearance of their concurrents by memory. Others, on the contrary, reported a more conceptual way in experiencing their synaesthetic concurrents, for example the female synaesthete who experiences her sexual desire stage in orange and who was easily able to visualize her synaesthetic experiences by memory. In summary, it still remains unclear whether or not sexual synaesthesia can be considered as ideaesthesia. Certainly, in case it could be confirmed, it would give an opportunity to prove in a laboratory setting that this form of synaesthesia really exists. This question should be topic of further enquiry.

However, it should be noted that, in our sample, other varieties of synaesthesia, such as grapheme-color or audio-visual forms, could be confirmed via consistency test which can be considered as clue for the credibility of the subjects. Another clue is that the "visionary restructuralization" scale showed significantly higher values for the synaesthetes than for controls, reflecting the general existence of visual synaesthesiae in our subjects.

Finally, it has to be pointed out that this study can be considered as a pilot project providing clues for further investigation. The descriptive character of some of the methods seems to be fairly adequate for this primary examination. Further research on this issue is still outstanding.

# **CONCLUSION**

The present investigation reveals novel aspects of ASCs in synaesthetes; synaesthetic experiences during sexual arousal seem to be accompanied by a higher degree of sexual trance

# **REFERENCES**


consisting of an experienced loss of environmental boundaries and varying visual perceptions. This increment in trance possibly leads to accelerated introversion and a shift in attentional resources focusing exclusively on inner perceptions. Whether or not this heightened introversion attenuates the partner involvement—and consequently the synaesthete's own sexual satisfaction—should be the subject of further research.

sexual-response. *J. Sex Res.* 16, 1–42. doi: 10.1080/00224498009551060


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 28 May 2013; accepted: 26 September 2013; published online: 16 October 2013.*

*Citation: Nielsen J, Kruger THC, Hartmann U, Passie T, Fehr T and Zedler M (2013) Synaesthesia and sexuality: the influence of synaesthetic perceptions on sexual experience. Front. Psychol. 4:751. doi: 10.3389/fpsyg. 2013.00751*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Nielsen, Kruger, Hartmann, Passie, Fehr and Zedler. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Enhanced recognition memory in grapheme-color synaesthesia for different categories of visual stimuli

# *Jamie Ward\*, Peter Hovard , Alicia Jones and Nicolas Rothen*

*Sackler Centre for Consciousness Science and School of Psychology, University of Sussex, Brighton, UK*

### *Edited by:*

*Roi C. Kadosh, University of Oxford, UK*

### *Reviewed by:*

*Christine Mohr, University of Lausanne, Switzerland Nathan Witthoft, Stanford University, USA*

### *\*Correspondence:*

*Jamie Ward, Sackler Centre for Consciousness Science and School of Psychology, University of Sussex, Falmer, Brighton, BN1 9QH, UK e-mail: jamiew@sussex.ac.uk*

Memory has been shown to be enhanced in grapheme-color synaesthesia, and this enhancement extends to certain visual stimuli (that don't induce synaesthesia) as well as stimuli comprised of graphemes (which do). Previous studies have used a variety of testing procedures to assess memory in synaesthesia (e.g., free recall, recognition, associative learning) making it hard to know the extent to which memory benefits are attributable to the stimulus properties themselves, the testing method, participant strategies, or some combination of these factors. In the first experiment, we use the same testing procedure (recognition memory) for a variety of stimuli (written words, non-words, scenes, and fractals) and also check which memorization strategies were used. We demonstrate that grapheme-color synaesthetes show enhanced memory across all these stimuli, but this is not found for a non-visual type of synaesthesia (lexical-gustatory). In the second experiment, the memory advantage for scenes is explored further by manipulating the properties of the old and new images (changing color, orientation, or object presence). Again, grapheme-color synaesthetes show a memory advantage for scenes across all manipulations. Although recognition memory is generally enhanced in this study, the largest effects were found for abstract visual images (fractals) and scenes for which color can be used to discriminate old/new status.

**Keywords: synaesthesia/synesthesia, recognition memory, scenes, color**

# **INTRODUCTION**

For people with grapheme-color synaesthesia, stimuli consisting of letters (including words) or digits are associated with experiences of color (Simner et al., 2006). These synaesthetes also show better memory for words, for instance in tests of free recall of lists (e.g., Radvansky et al., 2011). The most intuitive explanation for this is that it is the "extra" colors themselves that enable grapheme-color synaesthetes to create richer, and more robust, memory representations for words. In the case of spoken words, synaesthetes may encode these as visual objects enabling them to store them both as a verbal code and a visual code (Baron-Cohen et al., 1993). In the case of printed words, the presence of colors may render them as more distinct. (Although, of course, color cues could lead to memory confusions in certain circumstances for both written and spoken words). Whilst this may explain some of the memory advantage for synaesthetes it is unlikely to be the whole story. For instance, not all memory tests involving colorinducing stimuli have been linked to a memory advantage (e.g., associating digits to spatial positions; Rothen and Meier, 2009), moreover, some visual stimuli that do not induce any synaesthesia are linked to enhanced memory (Rothen et al., 2012). The current set of experiments aims to explore the latter more closely.

Certain visual stimuli that do not induce synaesthesia have been linked to enhanced memory in grapheme-color synaesthetes. Yaro and Ward (2007) found better memory for colors (but normal memory for a complex figure). Pritchard et al. (2013) also showed a memory advantage for color-based associative memory (relative to shape and location based associations). Rothen and Meier (2010) administered a standardized memory test (Wechsler Memory Scale–Revised) to a large group of grapheme-color synaesthetes and found that visual long-term memory was also enhanced and was, in fact, significantly better than verbal long-term memory (which was also enhanced relative to controls). The tests of visual memory in this battery all involved meaningless stimuli (e.g., figures, patterns) both colored and achromatic. Only a few studies have used meaningful visual stimuli (e.g., scenes, objects, faces) and have tended to use small samples of synaesthetes (Gross et al., 2011). As such, evidence for the memory enhancement for visual stimuli presently lacks breadth. The wide variety of testing methods used across different studies (associative memory, recall, recognition) and stimulus properties also make it hard to make direct comparisons across studies and stimuli.

Why do grapheme-color synaesthetes have enhanced visual memory for stimuli not (directly) linked to their synaesthesia? Rothen et al. (2012) suggested that grapheme-color synaesthetes have enhanced visual processing (in certain visual domains) that leads to benefits in both visual perception and visual memory. The latter would extend to certain auditory stimuli (such as spoken words) that are rendered by most grapheme-color synaesthetes as visual objects. This may explain why enhanced memory is not limited to synaesthesia-inducing material. There is evidence that color perception is behaviorally enhanced in grapheme-color synaesthesia (Banissy et al., 2013) but not synaesthesia involving other concurrent modalities such as touch (Banissy et al., 2009). Moreover, grapheme-color synaesthetes show enhanced visual-evoked potential, measured using EEG, to high contrast stimuli, to high spatial frequency gratings, and to chromatic stimuli (Barnett et al., 2008). This has led to the suggestion that the enhanced visual perception is not global but predominantly limited to the parvo-cellular system and/or the ventral visual stream (which depends heavily, but not exclusively, on parvo-cellular inputs). This stream is involved in recognition of visual features (e.g., color), objects, and scenes and is also implicated, in at least some accounts, in long-term storage of this information (Murray et al., 2007). The magno-cellular system is more specialized for motion perception (in addition to low spatial frequency processing and small luminance differences). There is evidence that motion perception is diminished in grapheme-color synaesthesia (Banissy et al., 2013) and that the motion-specialized region, V5/MT, has less gray matter (Banissy et al., 2012). The dorsal ventral stream, involved in processing of egocentric space, depends heavily on the magno-cellular system (e.g., Maunsell, 1987). How these perceptual differences may map onto memory processing in grapheme-color synaesthesia remains to be fully explored, but we speculate that enhanced memory will be more apparent when colors and objects are to be remembered and memory will be unaffected when memory tasks depend heavily on remembering spatial locations, object orientations, and so on.

The present set of experiments makes a significant new contribution to our understanding of memory processing in synaesthesia and addresses some of the limitations of previous research. We consistently use the same methodology for a variety of stimuli, namely a recognition memory paradigm in which a study phase of 30 items is followed by a test phase of 60 items in which participants must discriminate old from new. In Experiment 1, we used four kinds of stimuli presented visually and all achromatic: written words, written non-words, scenes, and fractals. These can be considered as a 2 × 2 design contrasting meaningfulness (words + scenes = meaningful, non-words + fractals = meaningless) and whether the stimulus is language-based (words, non-words) or image-based (scenes, fractals). For both synaesthetes and controls, we would expect better memory for meaningful stimuli because they can be linked to prior knowledge (Ericsson and Kintsch, 1995). If memory enhancement in synaesthesia is related to the presence of "extra" sensations then only the language-based stimuli should be enhanced relative to controls. The enhanced visual processing account would predict that recognition memory for visual stimuli should be generally enhanced (a main effect of group). The same test is administered on a group of lexical-gustatory synaesthetes (Ward and Simner, 2003; Ward et al., 2005) for whom words and, to a lesser extent, nonwords (Simner and Haywood, 2009) elicit flavor experiences (e.g., "Philip" tasting of sour oranges). Again, if memory enhancement were linked to extra sensations then we would predict that, for these synaesthetes, it should be specific to words (/non-words). If it is linked to enhanced visual perception then we would not predict any advantage in this sample as we assume (although it is an open question) that the changes in visual perception do not extend to this kind of synaesthesia.

The second study also uses the same recognition memory paradigm but involves scenes contrasted with words. The scenes are manipulated in one of three ways (by changing a color, orientation or object) and the study is loosely based on Pritchard et al. (2013). In that study, participants (synaesthetes and controls) were shown meaningless visual stimuli comprising of a conjunction of shape, color and location (e.g., shape A + color A + location A). In the test phase, they had to distinguish the initial conjunctions from those in which shape had been changed (e.g., shape B + color A + location A), color had been changed (e.g., shape A + color C + location A), or location had been changed (e.g., shape A + color A + location D). Synaesthetes did better, overall, on the test and showed a particularly enhanced ability to reject new items on the basis of color. Experiment 2 uses images of real world scenes in which the color of an object is changed between study and test (e.g., a chair is changed from red to green), or the orientation is changed (mirror reversal), or an object is added/removed from the scene. Our hypothesis is that grapheme-color synaesthetes will have enhanced memory on this test (a main effect of group), and particularly when color is a reliable cue to old/new status (an interaction between group and feature type).

# **EXPERIMENT 1: RECOGNITION MEMORY FOR WORDS, NON-WORDS, SCENES AND FRACTALS METHODS**

# *Participants*

Grapheme-color synaesthetes (*n* = 28) and non-synaesthete control subjects (*n* = 35) were recruited by email. Synaesthetes were tested for consistency using the Eagleman et al. (2007) battery and the cut-off score of 1.43 (from Rothen et al., 2013a). The groups were matched for age and sex, with a mean for synaesthetes of 31.6 years (*SD* = 14*.*9; 5 male), and 31.1 (*SD* = 12*.*43; 2 male) years for controls. They were also matched for level of formal education [*X*<sup>2</sup> *(*3*)* = 5*.*29, *p* = 0*.*15] which was reported on a 4-category scale (postgraduate, undergraduate, to age 18, to age 16). None of them reported any instances of lexical-gustatory synaesthesia.

A sample of lexical-gustatory synaesthetes (*n* = 18) and matched controls (*n* = 18) were additionally tested. The groups were matched for age and sex, with a mean for synaesthetes of 40.9 years (*SD* = 16*.*5; 4 male), and 36.6 (*SD* = 17*.*9; 4 male) years for controls. They were also matched for level of formal education [*X*<sup>2</sup> *(*3*)* = 3*.*70, *p* = 0*.*30] which was reported on a 4-category scale (postgraduate, undergraduate, to age 18, to age 16). Ten of them had taken part in previous published research and 8 were self-referred and not previously assessed in detail. None of them reported any instances of grapheme-color synaesthesia.

Approval was gained for this study through the University of Sussex Life Sciences and Psychology Cluster-based Research Ethics Committee.

### *Design*

A 2 × 2 × 2 mixed design was used, with an independent measures factor of group (synaesthete and control), and repeated measures factors of meaningfulness (meaningful and meaningless), and stimulus type (language-based and image-based). The two groups of synaesthetes were compared separately to their respective control groups.

# *Stimuli*

All words and nonsense words were four letters long, consisting of one syllable. Nonsense words were used only if they were considered pronounceable in spoken English. Thirty words and 30 nonsense words were selected for use and a further list of 30 words and 30 nonsense words was generated by changing one letter of the original item (e.g., FISH-FIST, SNET-SNEF). The position of the changed letter in the stimulus was varied across the four possible positions (appearing 7 or 8 times in each position). One item from each pair was randomly assigned to be either an old or new item (with the assignment fixed across participants). They varied in both frequency of usage (from 1 to 917; Kucera and Francis, 1967) and imageability. All words and non-words were presented in Calibri font using black upper case letters against a white window, 350 pixels wide by 263 pixels tall.

The scenes consisted of 30 pairs of grayscale images taken by one of the authors (PH). They consisted of rural and urban landscapes containing various objects (e.g., people, animals, vehicles). The pairs were perceptually (and conceptually) similar having been taken from somewhat different viewpoints or with certain objects present or absent. Fractal patterns were downloaded from the internet (Fantastic Fractals, 2011) and were all in grayscale. Similar fractal images were paired together. One item from each pair of scene or fractal images were randomly assigned as either an old or new item (with the assignment fixed across participants). The images were 350 pixels by 263 pixels in size and displayed against a white background. Examples are shown in **Figure 1**.

# *Procedure*

Cake PHP web application software was used to create a program for presenting stimuli to participants over the internet and recording their responses. After providing consent, participants were required to enter an email address so that the software could prevent multiple attempts at completing the test and so that synaesthetes (known to the researchers) could be grouped separately to controls and linked to their synaesthetic consistency score. They were also required to enter their age and sex. After the experiment participants were asked on screen to select their highest level of education achieved.

The main experiment consisted of four sets of study phases followed immediately by test phases (i.e., study-test, study-test, etc.) with the order of stimuli (words, non-words, scenes, fractals) counterbalanced across participants. Before each study phase participants were informed of the stimulus category and were asked to try to remember them for later on. Participants clicked to start the experiment once they had read the instructions. Items within a category were presented in a random order across participants. Each item was presented for 1 second against a plain white background.

After each study phase, participants were presented with another instruction screen, informing them that some items that had been presented and some items that they had not seen before would be displayed, and that they were required to respond "old" or "new" to items respectively. Participants were made aware that new items were similar to those presented before. This phase presented all 60 items from previously studied category and participants made a self-paced mouse-click on the

**FIGURE 1 | Examples of pairs of images used as old and new items in Experiment 1 (scenes on top, fractals on the bottom).**

old/new radio buttons. The item remained on screen until participants confirmed their choice, at which point the next item was displayed.

Once all four learning and test phases had been administered participants were asked on screen whether they had seen any colors for each of the stimulus types (with a brief definition of synaesthesia, for the benefit of the controls). Participants were also asked which statement best describes the strategy they had used for each type of stimulus from a list of "Associating with other things," "Verbalizing the item" or "Visualizing the item."

# **RESULTS**

The results for the grapheme-color synaesthetes are considered first, followed by the lexical-gustatory synaesthetes. A value of *p <* 0*.*05 was used for statistical significance, although we highlight possible non-significant trends (*p <* 0*.*10) and supplement our analyses with effect size calculations.

The results are summarized in **Figure 2** for overall accuracy (percentage correct). A 2 × 2 × 2 mixed ANOVA revealed a main effect of group [*F(*1*,* <sup>61</sup>*)* = 15*.*32, *p <* 0*.*001] with synaesthetes outperforming controls. There were also main effects of meaningfulness [*F(*1*,* <sup>61</sup>*)* = 207*.*23, *p <* 0*.*001] and stimulus type [*F(*1*,* <sup>61</sup>*)* = 6*.*99, *p <* 0*.*01] with meaningful stimuli (words, scenes) being more memorable than meaningless stimuli (nonwords, fractals) and linguistic stimuli being more memorable than images. However, neither of these factors interacted with group (*p >* 0*.*10) suggesting that synaesthetes benefit from these factors similarly to controls. No other interaction was significant but the three-way interaction approached significance [*F(*1*,* <sup>61</sup>*)* = 3*.*02, *p* = 0*.*087]. This is due to a trend for the synaesthetes to do particularly well in recognizing the fractal images. This is shown in **Figure 3**, considering the effect sizes for the different classes of stimuli. The same overall pattern is reproduced when the hits and false alarms are transformed to d-prime scores with synaesthetes performing better than controls on fractals (synaesthetes: mean = 1*.*46, *SD* = 0*.*73; controls: mean = 0*.*84, *SD* =

0*.*59), non-words (synaesthetes: mean = 1*.*41, *SD* = 0*.*73; controls: mean = 1*.*02, *SD* = 0*.*44), words (synaesthetes: mean = 2*.*81, *SD* = 1*.*19; controls: mean = 2*.*03, *SD* = 1*.*25), and scenes (synaesthetes: mean = 2*.*14, *SD* = 0*.*77; controls: mean = 1*.*64, *SD* = 0*.*74).

In the debriefing questions, the majority of synaesthetes reported noticing colors for words (82%) and non-words (82%) during the experiment. No control did so. However, the controls and synaesthetes generally did not differ in their self-reported strategies for memorizing the different stimuli. This is shown in **Table 1**. The groups did not differ in their reported strategy for fractals [*X*<sup>2</sup> *(*2*)* <sup>=</sup> <sup>0</sup>*.*09, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>*.*76], non-words [*X*<sup>2</sup> *(*2*)* = 1*.*96, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>*.*38] or scenes [*X*<sup>2</sup> *(*2*)* = 2*.*53, *p* = 0*.*28]. That is, the objective benefits in memory performance for these stimuli cannot be attributed to synaesthetes deliberately adopting a different strategy to controls. The reported strategies used for words did, however, differ significantly across groups [*X*<sup>2</sup> *(*2*)* = 11*.*72, *p* = 0*.*003] with controls tending to rely more heavily on a shallower "verbalizing" strategy (in comparison to the deeper "associating" strategy). This may contribute to the memory advantage for

**FIGURE 3 | Effect sizes (Cohen's d) for the various stimuli used in Experiments 1 and 2, for grapheme-color synaesthetes relative to controls.** These are calculated based on the memory discriminability estimates (i.e., d-prime, normalized hits minus false alarms).

**Table 1 | Encoding strategies reported by grapheme-color synaesthetes and controls for the various stimuli during the debriefing questionnaire (%).**


*The chi-square statistics are calculated on frequency counts but are shown here as percentages for ease of comparison. \*\*p < 0.01.*

synaesthetes in word recognition and this point is returned to in Experiment 2.

The results for the lexical-gustatory synaesthetes are summarized in **Figure 2B**.A2 × 2 × 2 mixed ANOVA revealed no main effect of group [*F(*1*,* <sup>34</sup>*)* = 1*.*83, *p* = 0*.*18]; i.e., these synaesthetes do not show the same memory advantage as documented for the grapheme-color synaesthetes. Most of the synaesthetes reported in the debriefing that they had experienced tastes for the words (94% of participants) and non-words (77%), so we may have predicted a group X verbal/image interaction (with the synaesthetes showing enhanced memory only for verbal stimuli) but this was not found [*F(*1*,* <sup>34</sup>*)* = 0*.*07, *p* = 0*.*799]. Nor were any other interactions significant (all *p*'s *>* 0*.*10). As before, there were main effects of meaningfulness [*F(*1*,* <sup>34</sup>*)* = 80*.*24, *p <* 0*.*001] and stimulus type [*F(*1*,* <sup>34</sup>*)* = 14*.*22, *p* = 0*.*001] with meaningful stimuli (words, scenes) being more memorable than meaningless stimuli (non-words, fractals) and linguistic stimuli being more memorable than images. However, a direct comparison between the two groups of synaesthetes (again employing a 2 × 2 × 2 ANOVA) failed to reveal a main effect of group [*F(*1*,* <sup>44</sup>*)* = 2*.*81, *p* = 0*.*101], or any interactions between stimuli and group (all *p >* 0*.*10). Thus, statistically speaking, it is not possible to reliably distinguish lexical-gustatory synaesthetes from either non-synaesthetes or from grapheme-color synaesthetes. Further studies will need to contrast different types of synaesthesia (e.g., sequence-space) in which sample sizes and demographics are better matched. As an exploratory first step, it is to be noted that the graphemecolor synaesthetes did significantly outperform lexical-gustatory synaesthetes on the fractal stimuli [*t(*44*)* = 2*.*26, *p* = 0*.*029] but not any of the other stimuli (all *p >* 0*.*10).

# **DISCUSSION**

Experiment 1 showed that grapheme-color synaesthetes have enhanced recognition memory for visually presented stimuli from a variety of categories. Whilst other studies (Yaro and Ward, 2007; Rothen and Meier, 2010) have contrasted different kinds of stimulus material (words, meaningless figures, etc.) they have tended to use a variety tasks to probe memory (e.g., verbal recall, drawing figures, associating several features/items together) thus making it uncertain whether the effects relate to the stimuli used, the task, or both. By standardizing the task we demonstrate that the memory advantage extends, approximately evenly, across all the stimuli tested. This is inconsistent with the notion that the memory advantage is due specifically to the presence of extra sensations. It is broadly consistent with the notion of enhanced (ventral) visual processing, although we initially predicted that memory for meaningless visual stimuli would be particularly enhanced in grapheme-color synaesthesia because these rely primarily on shallow visual encoding (assumed to be enhanced in these synaesthetes) whereas words/scenes afford opportunities for encoding deeply (we assume the structure of semantic memory to be unaffected by synaesthesia). Nevertheless, fractals showed the largest memory benefit for these synaesthetes which is intriguing given that these images consist of high contrast, high spatial frequency patterns which are known to elicit larger visual evoked potentials in these synaesthetes (Barnett et al., 2008). The memory advantage for these synaesthetes cannot be attributed to the strategy adopted (with the possible exception of words) because the reported strategies tended not to differ between groups. It is impossible to discount the suggestion that the groups differ in ways other than age, sex, and education—for instance in their motivation levels to do well. However, the fact that another group of synaesthetes (with lexical-gustatory synaesthesia) did *not* show a memory advantage, relative to their matched controls, speaks against this (for a similar discussion see Rothen et al., 2013b). It is also noteworthy that the fractal stimuli were associated with the largest group difference (relative to controls) for the grapheme-color synaesthetes but the smallest difference for the lexical-gustatory synaesthetes.

# **EXPERIMENT 2: RECOGNITION MEMORY FOR SCENES DIFFERING BY COLOR, ORIENTATION OR OBJECT PRESENCE**

This experiment explores the memory advantage for scenes that was found in Experiment 1. The scenes used in that experiment consisted of pairs of items (one old, one new) that were similar but differed in several ways such as camera angle, the presence/absence of certain objects (e.g., a passing car), as well as the overall pattern of luminance (e.g., presence of shadows). Experiment 2 aims to explore these different visual properties more systematically by using identical images but in which only

# **METHODS**

# *Participants*

Grapheme-color synaesthetes (*n* = 33) and non-synaesthete control subjects (*n* = 38) were recruited by email. These were a different sample to that used in Experiment 1. Synaesthetes were tested for consistency using the Eagleman et al. (2007) battery and the cut-off score of 1.43 (from Rothen et al., 2013a). The groups were matched for age and sex, with a mean for synaesthetes of 31.3 years (*SD* = 13*.*8; 3 male), and 32.9 years (*SD* = 17*.*61; 5 male) for controls. They were also matched for level of formal education [*X*<sup>2</sup> *(*3*)* = 5*.*33, *p* = 0*.*15] which was reported on a 4-point categorical scale (postgraduate, undergraduate, to age 18, to age 16). Approval was gained for this study through the University of Sussex Life Sciences and Psychology Cluster-based Research Ethics Committee.

# *Design*

The primary interest was in the different kinds of scene manipulations and therefore a 2 × 3 mixed design was used contrasting group (synaesthete and non-synaesthete) and scene manipulation (color change, orientation change, object change). The same word stimuli were included as in Experiment 1, as a fourth condition.

### *Stimuli*

The same set of 30 word pairs were used as in Experiment 1. In addition, 3 sets of 30 pairs of outdoor and indoor scene images were created from an initial set of images either taken by an author (AJ) or downloaded from "Google Images" in response to general searches such as "countryside scene" or "picnic." They were chosen on the basis of being a complex scene but also had a significant feature which could be manipulated without affecting any other aspect of the scene. The original images were manipulated using Adobe Photoshop CS5.1. Thirty pictures were a mirrorimage of the original picture, a further 30 were manipulated using an adjustment layer and altering the "hue/saturation" of object(s) in the scene and the final 30 were manipulated using the "clone stamp tool" and "spot healing brush" in order to remove one or more discrete objects from the scene (see **Figure 4**). Within each pair, one was randomly selected to appear as an old item and one as a new item (i.e., it was not the case the original photos were old items and manipulated items were new). This assignment was fixed across participants.

# *Procedure*

The basic procedure was identical to that used in Experiment 1. However, the instructions to participants were modified slightly. At the start of each study phase participants were told whether they would be shown words (1 block) or scenes (3 blocks) to remember but they were not told how the old/new scenes would subsequently differ. Thus, participants had no reason to deliberately try to attend to color, orientation or object presence at encoding. At the start of the *test* phase they were then instructed how the old/new items would differ from each other (e.g., "Click OLD if you have seen it exactly before. Click NEW if it is different in any way. Note the COLORS of some of the objects may have

changed"). This was added to ensure that the task was not too difficult and to ensure that participants had properly understood what was required of them. Pilot testing had revealed that participants have a strong tendency to endorse all items as 'old' in the absence of this clarification.

### **RESULTS**

**Experiment 2.**

The results are summarized in **Figure 5**. For the scenes, a 3 × 2 ANOVA revealed that synaesthetes performed significantly better than controls [main effect of group, *F(*1*,* <sup>69</sup>*)* = 10*.*62, *p* = 0*.*002]. There was a main effect of manipulation [*F(*2*,* <sup>138</sup>*)* = 3*.*24, *p* = 0*.*042] owing to the fact that the orientation changes tended to be easier to detect relative to color [*t(*70*)* = 2*.*72, *p* = 0*.*008; other pairwise comparisons not significant]. Contrary to our hypothesis there was no group X manipulation interaction [*F(*2*,* <sup>138</sup>*)* = 0*.*50, *p* = 0*.*60] suggesting that enhanced memory extends across all conditions. Nevertheless, there was a tendency for the color manipulation to produce the most reliable group difference, as is also evident from the effect sizes displayed in **Figure 3**.

The same broad pattern is found when the hits and false alarm rates are transformed into d-prime measures. Synaesthetes outperformed controls when orientation was manipulated [means with standard deviation in parentheses: synaesthetes = 2*.*28 *(*1*.*30*)*, controls = 1*.*58 *(*0*.*88*)*], when color was manipulated [synaesthetes = 1*.*72 *(*0*.*74*)*, controls = 1*.*15 *(*0*.*62*)*], and when object presence was manipulated [synaesthetes = 1*.*76 *(*0*.*94*)*, controls = 1*.*35 *(*0*.*62*)*].

The self-reported strategies for memorizing scenes did not differ between synaesthetes and controls [*X*<sup>2</sup> *(*2*)* = 0*.*42, *p* = 0*.*81]. For synaesthetes, 9% reported "associating to other things," 9% reported verbalizing, and 82% reported visualizing. For controls, 11% reported associating to other things, 5% reported verbalizing, and 84% reported visualizing. As such, the performance advantage is not attributable to the use of different mnemonic strategies.

As in Experiment 1, synaesthetes tended to have better memory for the word stimuli than controls (synaesthetes: mean d-prime = 2*.*63, *SD* = 1*.*52; mean % correct = 84*.*0, *SD* = 10*.*4; controls: mean d-prime = 2*.*13, *SD* = 1*.*26; mean % correct = 80*.*7, *SD* = 11*.*8). Unlike in Experiment 1, this did not

reach significance [% correct: *t(*69*)* = 1*.*25, *p* = 0*.*21; d-prime: *t(*69*)* = 1*.*51, *p* = 0*.*14]. Most synasthetes reported experiencing colors for the words during the experiment (76%). The self-reported strategies for memorizing words did not differ between synaesthetes and controls [*X*<sup>2</sup> *(*2*)* = 0*.*20, *p* = 0*.*90]. For synaesthetes, 39% reported associating to other things, 24% reported verbalizing, and 26% reported visualizing. For controls, 34% reported associating to other things, 26% reported verbalizing, and 39% reported visualizing. As such, one of the reasons for the discrepancy between Experiments 1 and 2 may lie in the somewhat different strategies employed (with synaesthetes in Experiment 1 more likely to employ an advantageous "associating" strategy and controls a weaker verbalizing strategy). To examine the effect of strategy use further, we combined the word recognition data across the two experiments and entered strategy as a separate grouping factor. The results are summarized in **Figure 6**.A3 × 2 betweensubjects ANOVA revealed a main effect of strategy [*F(*2*,* <sup>128</sup>*)* = 6*.*68, *p* = 0*.*002], a main effect of presence of synaesthesia [*F(*1*,* <sup>128</sup>*)* = 4*.*47, *p* = 0*.*036] and no interaction [*F(*2*,* <sup>128</sup>*)* = 0*.*48, *p* = 0*.*62]. In summary, although certain strategies tend to aid the recognition memory for words (notably associating to other things), the memory advantage of synaesthetes is independent of strategy.

Finally, not all of the synaesthetes reported noticing colors for the words (when asked at the end of the study) and the synaesthetes were therefore grouped according to this (collapsing across both experiments). There was no difference in recognition memory ability for words according to whether synaesthetes noticed the colors (mean % correct = 85*.*8, *SD* = 9*.*50) or not [mean % correct = 85*.*3, *SD* = 11*.*8; *t(*59*)* = 0*.*18, *p* = 0*.*86]. This provides further evidence that the memory enhancement is not directly tied to the deliberate use of synaesthetic color at encoding.

### **DISCUSSION**

The present study extends that of Experiment 1 by showing that grapheme-color synaesthetes have enhanced visual memory for images of scenes. In Experiment 1 the stimuli were achromatic and in Experiment 2 they were colored, but the memory advantage for grapheme-color synaesthetes were numerically similar for both. That is, the memory advantage for scenes is not dependent on the mere presence of color *per se*. In the present study, the scenes were manipulated in one of three ways: by changing orientation, by changing the color of a surface/object, or by adding/removing an entire object. We predicted, based on other research (Pritchard et al., 2013), that synaesthetes would be particularly good at rejecting distractors on the basis of color. Whilst this condition showed the greatest advantage, the interaction term itself was not significant. It may be that the *spatial distribution* of color (and perhaps other kinds on visual information such as luminance) is crucial, as this was affected by all three manipulations.

Finally, by pooling the data on word recognition memory across the two experiments (which used identical stimuli and testing conditions) we were able to show that the memory advantage for these stimuli, in grapheme-color synaesthesia, is not tied to the adoption of particular strategies and does not depend crucially on whether the synaesthete claimed to have been aware of the colors during the study phase.

# **GENERAL DISCUSSION**

The current experiments add to the growing body of evidence that memory is enhanced in grapheme-color synaesthesia and that this enhancement extends to stimuli that are not inducers of synaesthetic color. Most previous studies of visual memory in synaesthesia have used meaningless stimuli, but we extended this to (and directly contrast with) meaningful stimuli notably complex scenes. Based on the theory that enhanced visual memory in grapheme-color synaesthesia is linked to enhanced visual perception (notably of the visual ventral or parvo-cellular system) we predicted that the memory advantage may be greater for meaningless stimuli (fractals)—that tend to be remembered visually relative to meaningful visual stimuli (scenes) which can be more easily recoded verbally and semantically. This was not found, although there was a trend for the fractal images and color change to be linked to the largest memory benefit.

Within the literature on "superior memory" it has often been found that memory enhancement is strongly domain-specific and directly related to stimuli such as words, digits, and sequences which are more amenable to mnemonic strategies (Wilding and Valentine, 1997). In a difficult test of recognition memory of complex visual patterns (snowflakes) such memory experts typically fare no better than regular controls because their strategies are not helpful in that domain (Maguire et al., 2003). Our prediction is that grapheme-color synaesthetes would show enhanced memory on the snowflake test and, hence, be a convincing example of "true" memory ability that is independent of strategy use or memory training. We have shown that they show an enhancement for visually similar fractal stimuli that are hard to encode semantically and verbally and, indeed, our participants confirm that they tend not to use these strategies. Future studies should attempt to directly impose memorizing strategies (shallow vs. deep encoding) to determine whether the synaesthetic memory advantage is attenuated or increased by these influences (rather than considering strategy *post-hoc*).

With regards to enhanced memory in grapheme-color synaesthesia, it will be as much of a challenge for future research to find memory tests that are *not* linked to enhanced memory as it is to extend and replicate the positive evidence accumulated so far. One starting point may be to consider stimuli in the non-visual domain such as music or voices, as it is possible to devise similar testing protocols to those used in the visual domain. One could also consider non-visual spatial navigation (path integration) or motor learning.

It is also important to determine whether the memory benefits are related to synaesthesia in general or is specifically related to grapheme-color synaesthesia. Our present study failed to find a benefit of having lexical-gustatory synaesthesia. This observation rules out the possibility that enhanced memory in synaesthesia just reflects a motivation to do well (cf. Rothen et al., 2013a,b). It is consistent with the notion that enhanced visual perception [assumed to be limited to visual synaesthesia, (Banissy et al., 2009)] leads to enhanced visual memory (Rothen et al., 2012). Another type of visually-based synaesthesia is socalled sequence-space synaesthesia in which sequences such as months, years, and numbers are visualized in a spatial configuration (Eagleman, 2009). It is unclear whether this is linked to enhanced visual perception, but there is some preliminary evidence that they perform well in short-term retention of visual patterns (Simner et al., 2009). These synaesthetes, who visualize years spatially, also perform well at recalling the years in which events occurred and in recalling autobiographical events given a year cue (Simner et al., 2009). This suggests that they are using their synaesthesia as a deliberate memory aid to organize events and is a different kind of explanation to the one proposed here. There is little doubt that, under certain circumstances, synaesthesia can be used as a deliberate memory aid (e.g., Tammet, 2006). However, the current study also demonstrates that graphemecolor synaesthesia can lead to a more general boost in recognition memory for visual stimuli.

# **REFERENCES**


*Psychol. Rev.* 102, 211–245. doi: 10.1037/0033-295X.102.2.211


# **ACKNOWLEDGMENTS**

Vera Burckhardt, Daniel Coolbear and Stephanie Clayton provided assistance with data collection and analysis. The research was funded by an ESRC grant to Jamie Ward (ES/K006215/1) and Swiss National Science Foundation grant (PA00P1\_145370 / 1) to Nicolas Rothen.

*ONE* 4:e5037. doi: 10.1371/journal.pone.0005037


factors. *Cognition* 89, 237–261. doi: 10.1016/S0010-0277(03)00122-7


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 17 June 2013; paper pending published: 09 July 2013; accepted: 29 September 2013; published online: October 2013. 24*

*Citation: Ward J, Hovard P, Jones A and Rothen N (2013) Enhanced recognition memory in grapheme-color synaesthesia for different categories of visual stimuli. Front. Psychol. 4:762. doi: 10.3389/fpsyg. 2013.00762*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Ward, Hovard, Jones and Rothen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Grapheme-color synaesthesia is associated with a distinct cognitive style

### *Beat Meier <sup>1</sup> \* and Nicolas Rothen2*

*<sup>1</sup> Department of Psychology, Institute of Psychology and Center for Cognition, Learning and Memory, University of Bern, Bern, Switzerland <sup>2</sup> Department of Psychology and Sackler Centre for Consciousness Science, University of Sussex, Brighton, UK*

### *Edited by:*

*Roi C. Kadosh, University of Oxford, UK*

### *Reviewed by:*

*Roi C. Kadosh, University of Oxford, UK Limor Gertner, Ben-Gurion University of the Negev, Israel*

### *\*Correspondence:*

*Beat Meier, Department of Psychology, Institute of Psychology and Center for Cognition, Learning and Memory, University of Bern, Fabrikstr. 8, 3000 Bern 9, Switzerland e-mail: beat.meier@psy.unibe.ch*

In this study we investigated whether synaesthesia is associated with a particular cognitive style. Cognitive style refers to preferred modes of information processing, such as a verbal style or a visual style. We reasoned that related to the enriched world of experiences created by synaesthesia, its association with enhanced verbal and visual memory, higher imagery and creativity, synaesthetes might show enhanced preference for a verbal as well as for a visual cognitive style compared to non-synaesthetes. In Study 1 we tested a large convenience sample of 1046 participants, who classified themselves as grapheme-color, sound-color, lexical-gustatory, sequence-space, or as non-synaesthetes. To assess cognitive style, we used the revised verbalizer-visualizer questionnaire (VVQ), which involves three independent cognitive style dimensions (verbal style, visual-spatial style, and vivid imagery style). The most important result was that those who reported grapheme-color synaesthesia showed higher ratings on the verbal and vivid imagery style dimensions, but not on the visual-spatial style dimension. In Study 2 we replicated this finding in a laboratory study involving 24 grapheme-color synaesthetes with objectively confirmed synaesthesia and a closely matched control group. Our results indicate that grapheme-color synaesthetes prefer both a verbal and a specific visual cognitive style. We suggest that this enhanced preference, probably together with the greater ease to switch between a verbal and a vivid visual imagery style, may be related to cognitive advantages associated with grapheme color synaesthesia such as enhanced memory performance and creativity.

**Keywords: visualizer, verbalizer, imagery, learning style, memory, creativity**

# **INTRODUCTION**

Synaesthesia involves unusual experiences, such as color experiences in response to letters printed in black (grapheme-color synaesthesia), color experiences in response to music and sound (sound-color synaesthesia), gustatory experiences in response to words and names (lexical gustatory synaesthesia), or the activation of a spatial representation when confronted with an element that belongs to a particular sequence, for example, a day of the week (sequence-space synaesthesia). As synaesthesia—even different types of it—runs in families, it is assumed to have a genetic component (Barnett et al., 2008a; Asher et al., 2009) and, in fact, frequently different types of synaesthesia co-occur within the same individual (Rich et al., 2005; Simner et al., 2006; Novich et al., 2011). Evidence from neuroscience indicates that synaesthesia is associated with structural brain differences such as increased local and global connectivity (Rouw and Scholte, 2007; Hanggi et al., 2011; Rouw et al., 2011). Moreover, for grapheme-color synaesthesia, there is evidence that it is associated with basic differences in perception and excitability of the visual cortex (Barnett et al., 2008b; Terhune et al., 2011). Synaesthesia has cognitive consequences such as enhanced memory performance (Smilek et al., 2002; Rothen et al., 2012; Meier and Rothen, 2013), and greater involvement in creative activities (Rich et al., 2005; Ward et al., 2008; Rothen and Meier, 2010b). Moreover recent evidence suggests that synaesthesia is associated with an atypical personality profile (Banissy et al., 2012, 2013). Here we investigate whether synaesthesia is associated with a distinct cognitive style.

Cognitive style refers to a preference for processing information in a certain manner, thus representing consistencies in cognitive functioning of a person, particularly with respect to acquiring and processing information (Blazhenkova and Kozhevnikov, 2009). An important distinction is between verbalizers and visualizers (Paivio, 1971; Richardson, 1977), with the former assumed to rely more on verbal strategies and the latter more on visual strategies when performing a cognitive task. Originally, this was thought to be a unidimensional construct with verbal and visual preference as complementary poles. However, subsequent research has shown that verbal and visual style dimensions are independent. Moreover, both psychometric and neuroscientific evidence has revealed that the visual factor can be further subdivided into two independent visual style dimensions: a spatial imagery style and an object imagery style, the former corresponding to the dorsal and the latter to the ventral system of visual information processing (Kozhevnikov et al., 2005; Blazhenkova and Kozhevnikov, 2009; Borst et al., 2011). This distinction is also reflected in one of the most common selfreport measures, the revised Verbalizer-Visualizer Questionnaire (VVQ), in which the spatial visualizer style was complemented with a vividness of imagery style (Kirby et al., 1988), reflecting the fact that object visualizers are especially good in generating vivid pictorial images.

Neuroimaging evidence shows that cognitive style affects individual differences in domain-specific brain activations (Kraemer et al., 2009; Hsu et al., 2011). Using functional magnetic resonance imaging (fMRI), Kraemer et al. (2009) compared correlations between brain activations during a word-based and a picture-based task with verbalizer and visualizer scores of the revised VVQ. They found that for the picture-based task, the verbalizer ratings correlated with brain activity in the area of the left supramarginal gyrus. In contrast, in the word-based task, visualizer ratings correlated with brain activity in the right fusiform gyrus. Thus, higher scores on the verbal style dimension were associated with activations in phonological areas when processing pictorial representations and higher scores on the visual style dimension were associated with activations in the visual cortex when processing written descriptions of visual features. These results suggest that a particular cognitive style activates stylespecific brain areas, particularly when the information is not presented in that domain.

In a follow-up study, Hsu et al. (2011) extended these results with a verbal task during which participants were required to retrieve more or less detailed color information. They found more activation in the left fusiform gyrus during retrieval of more detailed color knowledge and this activation correlated with a visual style preference, again suggesting that cognitive style was associated with processing information in the style-specific mode.

It is noteworthy that the brain areas in inferior temporal and inferior parietal regions which are associated with a particular cognitive style, are often found to be more activated in graphemecolor synaesthesia in response to achromatic graphemes (Nunn et al., 2002; Hubbard and Ramachandran, 2005; van Leeuwen et al., 2010). Thus, rather than showing a preference for either a verbal or a visual cognitive style, it is possible that synaesthetes have higher scores on both verbalizer and visualizer style dimensions. Consistent with this notion is the fact that graphemecolor synaesthesia is associated with enhanced verbal *and* visual memory, higher imagery and creativity, and more generally, an enriched world of experiences (Meier and Rothen, 2013). This may go together with enhanced preference for a verbal *and* a visual cognitive style and the goal of this study was to investigate this possibility.

In Study 1 we used a convenience sample of more than thousand participants who filled out a brief survey on our synaesthesia research webpage. In Study 2, we specifically tested graphemecolor synaesthetes with confirmed grapheme-color consistency in a laboratory study.

# **STUDY 1**

## **METHOD**

### *Participants*

The sample consisted of participants who filled out the Synaesthesia-Check on the synaesthesia research webpage of the University of Bern (www*.*synaesthesie*.*unibe*.*ch). The Synaesthesia-Check is a short questionnaire used to establish contact with the general public interested in our research. It involves questions about potential forms of synaesthesia such as grapheme-color, sound color, lexical gustatory, and sequence space, demographic information, and it provides the opportunity to leave contact information for those willing to take part in future studies.

In this study we report data that were collected in the period from October 2010 until April 2013. From a total of 3320 people who completed the Synaesthesia-Check, 1046 also agreed to fill out a questionnaire on cognitive style that was then appended at the end of the Synaesthesia-Check. The supplied mean age of these participants was 30.9 years (*SD* = 13*.*14), 79% of them females, and 88% of them right-handed. Overall, 83% of these participants indicated to have at least one of the four forms of synaesthesia. The exact number of participants for each form and each combination of different forms is presented in **Table 1**.

### *Material and procedure*

In order to assess cognitive style, we used the 30-items revision of the VVQ Verbalizer-Visualizer-Questionnaire (Richardson, 1977; Kirby et al., 1988). The VVQ consists of three scales, verbal style, visual style, and vivid imagery style, each with appropriate psychometric properties (Kirby et al., 1988). A high score on the verbal dimension of the questionnaire is associated with a preference for verbal representations and an enhanced ability to work with verbal materials (e.g. "I enjoy doing work that requires the use of words"). A high score on the visual dimension of the questionnaire is associated with a preference for visuo-spatial representation formats and with the ability to imagine spatial compositions of scenes or real world objects (e.g., "When I read books with maps in them, I refer to the maps a lot"). A high score on the vivid imagery style is associated with a preference to let

**Table 1 | Descriptive statistics for cognitive style measures in Study 1.**


*GC, Grapheme Color; SC, Sound Color; LG, Lexical Gustatory; SS, Sequence Space; CG, Control Group (no synaesthesia).*

the mind wander and the ability to generate vivid mental images, particularly related to dream imagery (e.g., "My dreams are sometimes so vivid I feel as though I actually experience the scene"). Each scale consists of 10 statements, five worded positively and five worded negatively. Participants had to rate each statement on a discrete five-point scale, from strongly agree (5) to strongly disagree (1). Scores for each scale were summed separately. The five positively worded questions received positive scores and the five negatively worded questions received negative scores. Thus, scores from −20 to 20 were possible. Data collection was administered on-line via the internet and lasted approximately 10 min, with text appearing in black on a white background.

### *Design*

Given that different types of synaesthesia often co-occur and our large data-set, we decided to use analyses of variance (ANOVA) with the four types of synaesthesia (grapheme-color, sound-color, lexical-gustatory, and sequence-space) as factors, each with two levels (i.e., present/absent). Hence, it was possible for a given individual to have several types (i.e., multiple synaesthetes) or no synaesthesia at all (i.e., control group). This design allowed to identify types of synaesthesia that influence cognitive style independently and it also allowed to detect interaction effects (e.g., enhancing effects from the presence of multiple types of synaesthesia). As dependent variable, each of the three cognitive style dimensions was analyzed separately. To determine significance an alpha level of 0.05 was used.

### **RESULTS**

**Table 1** presents descriptive information for verbalizer, spatial visualizer, and vivid imagery visualizer styles for each factorial combination of the four types of synaesthesia and the control group, separately.

For the verbalizer style dimension, a 4-factorial ANOVA with the four types of synaesthesia as binary factors revealed a significant effect of grapheme-color synaesthesia, *F(*1*,* <sup>1030</sup>*)* = 7*.*195, *p <* 0*.*01, indicating higher ratings for participants who indicated to have grapheme-color synaesthesia compared to those who did not indicate to have grapheme-color synaesthesia. This result is depicted in **Figure 1**. No other effect reached significance, all *F*s *<* 3*.*1, *p*s *>* 0*.*08.

For the spatial visualizer style dimension, the same kind of analysis revealed no significant effects, neither differences between the four types of synaesthesia, nor any interaction effect, all *F*s *<* 2*.*1, *p*s *>* 0*.*15.

For the vivid imagery visualizer style dimension, the 4-factorial ANOVA showed significant main effects, with *F(*1*,* <sup>1030</sup>*)* = 12*.*56, *p <* 0*.*01, for grapheme-color synaesthesia, *F(*1*,* <sup>1030</sup>*)* = 14*.*36, *p <* 0*.*01, for sound-color synaesthesia, *F(*1*,* <sup>1030</sup>*)* = 7*.*47, *p <* 0*.*01, and for lexical-gustatory synaesthesia, and a marginally significant effect for sequence-space synaesthesia, *F(*1*,* <sup>1030</sup>*)* = 3*.*79, *p* = 0*.*052. Moreover, there was a double interaction between grapheme-color and sound-color synaesthesia, *F(*1*,* <sup>1030</sup>*)* = 7*.*99, *p <* 0*.*01, as well as a triple interaction between grapheme-color synaesthesia, lexical-gustatory synaesthesia and sequence-space synaesthesia, *F(*1*,* <sup>1030</sup>*)* = 6*.*83, *p <* 0*.*01. No other effect reached significance, all *F*s *<* 2*.*1, *p*s *>* 0*.*15.

The double interaction is depicted in **Figure 2**. It indicates that the presence of a color synaesthesia (i.e., either grapheme-color, or sound-color, or both) is critical for the higher vivid imagery visual style ratings. However, it also indicates that the presence of both kinds of color synaesthesia's did not further boost the vivid imagery visual style ratings.

The triple interaction is depicted in **Figure 3**. It indicates that in the absence of grapheme-color synaesthesia, the presence of both, lexical-gustatory synaesthesia and sequence-space synaesthesia also enhanced the vivid imagery visualizer style ratings. However, the presence of only either sequence-space or lexical-gustatory synaesthesia was not sufficient to boost the vivid imagery visualizer style ratings in the absence of grapheme-color synaesthesia.

In order to investigate the independence of the three cognitive style dimensions, we also performed correlational analyses. The correlations between verbal and spatial visualizer style, verbal and vivid imagery visualizer style, and spatial visualizer and vivid imagery visualizer style were *r* = −0*.*08, *r* = 0*.*18, and *r* = 0*.*17 (all *p*s *<* 0*.*01), respectively, representing low effect sizes,

**FIGURE 1 | Mean ratings for participants with grapheme-color synaesthesia (GCS) and without grapheme-color synaesthesia (No GCS) on the verbal style dimension.** Error bars represent standard errors.

**FIGURE 2 | Mean ratings for participants without color synaesthesia (no color) and those with color synaesthesia (i.e., sound-color, grapheme-color or both) on the vivid imagery style dimension.** Error bars represent standard errors.

and replicating the substantial independence of the three style dimensions (Kirby et al., 1988).

# **DISCUSSION**

The results of Study 1 show a clear association between synaesthesia and a distinct pattern of cognitive style. Specifically, grapheme-color synaesthetes scored higher on the verbal style factor. Moreover, several types of synaesthesia go together with a more vivid imagery style. Importantly, the results suggest that those types of synaesthesia which invoke color have the most pronounced effect. In the absence of colored synaesthesia, the presence of multiple types of non-colored synaesthesia such as lexical-gustatory together with sequence space also enhanced a vivid imagery visualizer style. In contrast, the visualizer style dimension that is more concerned with spatial relations was not affected by the presence of any kind of synaesthesia at all, not even sequence-space synaesthesia (cf., Price, 2009, for a similar result for spatial imagery).

Although the pattern of these results is suggestive, we acknowledge that group membership was self-referred and we did not have any objective measurement to confirm the genuineness of synaesthesia. Thus, a replication under more controlled conditions is warranted. In Study 2 we used a laboratory setting and we administered a test of consistency to objectify the presence of synaesthesia (Eagleman et al., 2007).

# **STUDY 2**

# **METHOD**

# *Participants*

For Study 2, a group of 24 grapheme-color synaesthetes and a control group consisting of 48 non-synaesthetes matched for age (Synaesthetes: *M* = 35*.*5 years, *SD* = 14*.*9; Control group: *M* = 35*.*8, *SD* = 14*.*8), gender (Synaesthetes: 22 females, Control group: 44 females) and handedness (Synaesthetes: 22 righthanded, Control group: 44 right-handed) participated. None of them had already participated in Study 1. For each synaesthete,

two control subjects were yoke-matched. Inclusion criterion for the synaesthetes was the presence of grapheme-color synaesthesia. We also asked them about the presence of other forms of synaesthesia. It turned out that five of them also reported to have sequence-space synaesthesia. None reported to have sound-color or lexical gustatory synaesthesia.

# *Material and procedure*

Participants were tested in the laboratory and we administered a test of consistency to verify the grapheme-color synaesthesia, for which grapheme-color associations were assessed by means of a computerized color palette (Meier and Rothen, 2007). RGBvalues were converted into CIELUV-values and Euclidian distances were calculated as consistency scores, with smaller values indicating higher consistency (Rothen et al., 2013). For synaesthetes, mean distance in CIELUV space was 27.5 (*SD* = 13*.*3), and for the control group, it was 103.4 (*SD* = 29*.*1). These consistency scores were significantly different, *t(*70*)* = 12*.*1, *p <* 0*.*001, thus confirming the higher consistency of the synaesthetes.

We used the same VVQ-questionnaire as in Study 1 to assess the three cognitive style dimensions, but this time a paper/pencil version was administered. The VVQ was administered as part of a larger test battery which lasted about two and a half hours. The consistency test was conducted at the beginning and at the end of the test session.

# **RESULTS**

The results of each of the three cognitive style ratings are shown in **Figure 4**, separately for synaesthetes and the non-synaesthete control group. To test for group differences, we used independent *t*-tests for each cognitive style dimension separately. For the verbal style dimension, the grapheme-color synaesthetes scored higher than the control group, *t(*70*)* = 2*.*36, *p <* 0*.*05. For the spatial visualizer style dimension, there was no difference between groups, *t(*70*)* = 0*.*86, *p >* 0*.*05. For the vivid imagery visualizer style dimension, the grapheme-color synaesthetes scored higher than the control group, *t(*70*)* = 3*.*43, *p <* 0*.*05. Thus, these results replicate the main results from Study 1 in a controlled laboratory setting.

**FIGURE 4 | Mean ratings for participants with and without grapheme-color synaesthesia on the verbal style, the visual style, and the vivid imagery cognitive style dimensions in Study 2.** Error bars represent standard errors.

As in Study 1, we used correlational analyses to investigate the independence of the three cognitive style dimensions. The correlations between verbal and spatial visualizer style, verbal and vivid imagery visualizer style, and spatial visualizer and vivid imagery visualizer style were *r* = −0*.*05 (*p* = 0*.*69), *r* = 0*.*26 (*p <* 0*.*05), and *r* = −0*.*16 (*p* = 0*.*19), respectively, representing low effect sizes, and indicating again the substantial independence of the three style dimensions.

# **DISCUSSION**

The results of Study 2 replicate the main results of the first study by demonstrating again enhanced ratings for a verbal cognitive style and a vivid imagery style, but no differences for spatial visual style. A comparison with Study 1 shows that the scores of the verbalizer scale were very similar. However, for the spatial visualizer style and the vivid imagery style the score was somewhat smaller. We have no straight-forward explanation for this discrepancy. One possibility is that the change in the test format may have affected the scoring on the visual scales. Another possibility is that as different groups were tested this may simply reflect random sampling variation. Importantly, however, the general pattern of results was replicated and it occurred within a sample with confirmed grapheme-color synaesthesia as indicated by a test of consistency, and a tightly matched control group under well-controlled laboratory conditions. This indicates that grapheme-color synaesthesia is associated with a distinct cognitive style. In the General Discussion we elaborate on the broader implications of these results.

# **GENERAL DISCUSSION**

This study provides the first systematic investigation into the relationship between cognitive style and synaesthesia. Inspired by the findings related to the cognitive consequences of synaesthesia for memory, creativity, and imagery, and by findings from neuroimaging studies, we hypothesized that in synaesthesia a preference for both verbal and visual cognitive styles may be prevalent. The results of two studies confirm this assumption. Particularly, we found that grapheme-color synaesthesia is associated with a preference for both a verbal style and for a vivid imagery style. Further results from Study 1 suggest that a preference for a more vivid imagery style may be prevalent also for other forms of synaesthesia, in particular those that involve color (i.e., grapheme-color and sound-color), and in the case of the absence of grapheme-color synaesthesia, those that involve multiple other forms such as lexical-gustatory and sequence-space. Thus, in the absence of color synaesthesia, the presence of multiple forms of synaesthesia may also induce a richer world of experience that is then sufficient to enhance the preference for a vivid imagery cognitive style.

The result of enhanced verbalizer style in grapheme-color synaesthesia is consistent with the idea that synaesthetes have a particular affinity for information that is related to their inducer, that is, verbal materials in the case of grapheme-color synaesthesia. This affinity may have consequences in other cognitive domains such as enhanced memory for words and more generally, the verbal domain (Rothen and Meier, 2010a). Moreover, there are many famous writers such as Nabokov, Baudelaire, or Wittgenstein who were also synaesthetes and it is possible that there is a link between synaesthesia and a preference for verbal processing. Recently, Banissy et al. (2013) found that graphemecolor synaesthesia was associated with higher scores in the *openness to experience* personality factor, which is correlated with verbal intelligence (Ziegler et al., 2012). Notably, in graphemecolor synaesthesia the inducers are instances of sequences such as digits, letters, or days of the week, and processing such sequences may involve mainly serial, analytic processing tuning the cognitive system for a verbal cognitive style.

The result of enhanced vivid imagery visualizer style in synaesthesia is also consistent with the idea that synaesthetes have a particular affinity for information that is related to their concurrent, that is, visual and particularly color information in the case of grapheme-color synaesthesia. This affinity may also have consequences in other cognitive domains such as enhanced perception for colors, enhanced memory for colors (Yaro and Ward, 2007; Pritchard et al., 2013; Terhune et al., 2013) and more generally, the visual domain (Rothen and Meier, 2010a).

The result of a more vivid imagery visualizer style in synaesthetes is also consistent with reports of atypical experiences in color synaesthetes, such as more strange perceptual experiences and hallucinations and more fantasizing (Banissy et al., 2012, 2013). Moreover, an enhanced vivid imagery visualizer style can be considered to be directly related to findings from studies that investigated imagery abilities in synaesthetes and non-synaesthetes (Barnett and Newell, 2008; Spiller and Jansari, 2008; Price, 2009). Using an experimental procedure, Spiller and Jansari (2008) found that grapheme-color synaesthetes were faster than a matched control group in an imagery task that involved the generation and inspection of visual images. Using a self-report measure, Barnett and Newell (2008) found that a group of synaesthetes reported experiencing more vivid mental images than controls. Price (2009) also found that people with sequence space synaesthesia reported more vivid mental imagery. In contrast, no such difference was evident for spatial imagery.

The absence of an advantage in spatial imagery in sequencespace synaesthesia is consistent with our findings from the spatial visualizer style, in which we did not find any differences between synaesthetes and non-synaesthetes. There is evidence that a spatial visualizer style is pre-dominant in scientists such as physicists and engineers, while a vivid imagery visualizer style is more pre-dominant in visual artist such as professional painters, photographers and interior designers (Kozhevnikov et al., 2005). Given that synaesthetes have a propensity to engage in creative activities and occupations more than in science, it may not be surprising that we did not find any differences on the spatial visualizer style.

Our results suggest that a preference for both a verbal and a vivid imagery visual style can co-occur within the same group of individuals and that this distinct pattern is associated with grapheme-color synaesthesia. As noted above, graphemes are instances of sequences which involve serial, analytic processing, favoring a verbal cognitive style. However, the concurrents are visual instances, most often colors and visual forms, which involve parallel or holistic processing, thus favoring a visual cognitive style. The combination of a verbal and a vivid imagery visual style in grapheme-color synaesthetes and their propensity to switch easily between these styles may be related to the cognitive benefits associated with grapheme-color synaesthesia. On the other hand, it also might be the result of their better cognitive performance in this domain so they will adapt a cognitive style that fits their ability. Further research is necessary to determine the causality of this relationship.

We have proposed elsewhere that synaesthetic concurrents are represented as additional features in the semantic network, and via spreading activation they allow for supplementary connections between concepts (Rothen et al., 2010; Meier, in press). Here we would like to emphasize that a core difference between synaesthetes and non-synaesthetes may not only be that more connections between semantic representations are existing in the synaesthete brain, but that also the way these representations can be accessed may differ. That is, due to an affinity toward both verbalizer and visualizer processing styles multiple pathways are available to access these representations. In light of these considerations the observed differences between synaesthetes and nonsynaesthetes in memory and creativity can be easily accounted for. There is converging evidence that a synaesthete's brain is in fact hyperconnected, not only locally in the fusiform gyrus and

# **REFERENCES**


in early sensory-perceptual processing in synesthesia: a visual evoked potential study. *Neuroimage* 43, 605–613. doi: 10.1016/j.neuroimage.2008.07.028


the inferior parietal lobe, but also globally (Hanggi et al., 2011). Nevertheless it is not clear whether the hyperconnected brain is the cause of synaesthesia or the consequence. It is possible that due to the lifelong associations between synaesthetic inducers and concurrents together with the propensity to favor multiple cognitive styles the hyperconnected brain in synaesthesia represents a remarkable example of brain plasticity (Bargary and Mitchell, 2008; Cohen Kadosh and Walsh, 2008; Hubbard et al., 2011; Rouw et al., 2011).

To summarize, in the present study, we show that graphemecolor synaesthesia is associated with a distinct cognitive style, namely a preference for both a verbalizer style and a vivid imagery visualizer style. Moreover, our results suggest that, at least a vivid imagery visualizer style may also be more prevalent in other forms of synaesthesia. It is an interesting direction for future research to investigate cognitive styles differences in these types of synaesthesia in more detail.

# **ACKNOWLEDGMENTS**

N. Rothen is supported by the Swiss National Science Foundation (Grant PA00P1\_145370).

*J. Neurosci.* 31, 5816–5828. doi: 10.1523/JNEUROSCI.0964-10.2011


(2012). Openness, fluid intelligence, and crystallized intelligence: toward an integrative model. *J. Res. Pers.* 46, 173–183. doi: 10.1016/j.jrp.2012.01.002

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 May 2013; accepted: 26 August 2013; published online: 19 September 2013.*

*Citation: Meier B and Rothen N (2013) Grapheme-color synaesthesia is associated with a distinct cognitive style. Front. Psychol. 4:632. doi: 10.3389/fpsyg. 2013.00632*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Meier and Rothen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Numerical synesthesia is more than just a symbol-induced phenomenon

# *Limor Gertner\*, Isabel Arend and Avishai Henik*

*Department of Psychology and the Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, Beer-Sheva, Israel \*Correspondence: limorger@post.bgu.ac.il*

### *Edited by:*

*Roi Cohen Kadosh, University of Oxford, UK*

### *Reviewed by:*

*Mark Price, University of Bergen, Norway Jacqueline M. Thompson, University of Oxford, UK*

**Keywords: number-color synesthesia, number-space synesthesia, symbolic numbers, non-symbolic magnitude comparison, numerical cognition**

Synesthesia is a peculiar condition that involves an atypical binding between two seemingly independent sensory modalities or within the same one. All synesthetic bindings are characterized by an inducing stimulus (i.e., inducer) and a subjective sensation triggered concurrently (i.e., concurrent). Synesthetic inducers can be sensory (e.g., sound) or conceptual (e.g., graphemes, time units) while the concurrent sensation is, in the majority of cases, a sensory one (e.g., smell, touch).

In numerical synesthesia, numbers (i.e., inducer) automatically and consistently trigger an ancillary experience of color, texture, spatial location, or personification (i.e., concurrent). For example, for a given synesthete, an audition of the number 5 may trigger a sensation of the color shiny yellow, be mapped on a vertical meridian above four and beneath six in his/her peripersonal space or elicit a cognitive awareness of "a young man, ordinary, and common in his tastes and appearance *...*" (Simner et al., 2011).

Until recently, numerical synesthesia was almost unquestionably viewed as a symbolic-based phenomenon. This was mainly because most synesthetes report their synesthetic experience is elicited solely by symbolic content (i.e., Arabic numbers) but not by non-symbolic ones (i.e., size, quantity) (Cohen Kadosh and Gertner, 2011). Furthermore, some researchers showed that non-symbolic magnitudes (i.e., random clusters of dots) or less familiar symbolic numerals (i.e., Roman numbers) were ineffective in evoking the synesthetic concurrent (Ramachandran and Hubbard, 2001, Experiment 3; Ward and Sagiv, 2007), suggesting that Arabic numbers *per se* (i.e., their form, ordinality, etc.) and not their semantic meaning or numerosity are the critical factors for inducing synesthetic experience (Hubbard et al., 2009).

However, when considering two main findings in the domain of numerical cognition—(a) that symbolic content is intimately associated with non-symbolic dimensions (e.g., size, quantity, brightness) (e.g., Schwarz and Heinze, 1998; Fias et al., 2003; Pinel et al., 2004; Ansari, 2008; Cohen Kadosh et al., 2008a), and (b) that magnitude is assumed to be automatically activated whenever we are presented with numbers (Dehaene et al., 1993; Dehaene and Akhavein, 1995)—one must wonder whether synesthetic experience can also be elicited by different magnitude dimensions.

In this opinion paper we present some recent observations from the literature on numerical synesthesia indicating that magnitude representation may play a role in mediating synesthetic effects found under experimental settings. Based on this evidence we suggest that synesthetic experiences induced by numbers may be produced also by non-symbolic magnitudes, due to the cognitive and neuronal overlap of these two dimensions. In other words, we propose that numerical synesthesia is more than a symbolinduced phenomenon *per se*. Furthermore, we speculate that this suggested association between a non-symbolic inducer and a synesthetic concurrent may manifest at different levels of awareness, resulting in an explicit, reportable experience for some synesthetes but a more non-conscious or implicit representation in others.

Before presenting our arguments, it is important to note that we use the phrase "numerical synesthesia" to include the subtypes of synesthesia that share numerical inducers (i.e., number-color and number-space synesthesia). We acknowledge that in spite of a common inducer (i.e., number), different mechanisms may mediate the various inducer-concurrent associations (Novich et al., 2011), yet we argue that both types discussed here illustrate the suggested involvement of magnitude representation in inducing synesthetic concurrents (i.e., color, spatial location).

# **NUMBER-COLOR SYNESTHESIA**

Berteletti et al. (2010) tested a graphemecolor synesthete (NM) who reported experiencing colors in response to digits, but not in response to non-symbolic numerosities. Surprisingly, when this synesthete was tested on a Stroop-like color naming task, he showed a congruency effect for both Arabic digits and dot patterns. The authors suggested that NM had an explicit association of numbers and colors (i.e., primary synesthetic connection) and an implicit association of numerosity and colors (i.e., secondary synesthetic connection). According to the authors, this implicit association, which they refer to as "*pseudosynesthesia*," is a consequence of an inherent lifetime association between numbers, their magnitudes, and "their" synesthetic colors.

The above study exemplifies an implicit association between *discrete* numerosity and color. However, since non-symbolic discrete numerosities can be counted and labeled by a symbolic number, we cannot affirm that the Stroop-like effect that was found indeed represented a magnitudebased association.

A couple of studies, although it was not their main goal, demonstrated a potential existence of implicit associations between non-symbolic *continuous* magnitudes and colors. Cohen Kadosh and Henik (2006) tested a digit-color synesthete (IS) on an adjusted Stroop-like task with colored lines. The synesthete's task was to decide which of two presented lines was physically longer. Lines were colored either congruently (i.e., a long line presented in a color that was induced by a large digit like 7) or incongruently (i.e., a longer line presented in a color that was induced by a small digit like 2). Digits were not presented at any stage. A congruency effect between line length and color was observed. In a later study, Cohen Kadosh et al. (2007) showed similar effects when they examined the brain activity of the same synesthete while he performed symbolic (i.e., Arabic numbers) and non-symbolic (i.e., triangle height) magnitude comparison in a Stroop-like task similar to the one they used in their previous study (Cohen Kadosh and Henik, 2006). A behavioral congruency effect was observed for physical comparisons for the synesthete but not for controls. Importantly, this congruency effect was found to modulate the eventrelated potential (ERP) wave N170, suggesting a detection of a conflict between size and color1. The above results were interpreted by the authors as an indication for bidirectionality in synesthesia (i.e., colors evoke numbers).

Bidirectionality is an intriguing topic on its own, and the above studies present convincing data supporting it. However, we would like to further interpret these results as suggestive that, for some synesthetes, magnitude might become associated with color (in the absence of symbolic number).

Specifically, we suggest that the congruency effects found between magnitude and color in the studies presented above may be driven by two different conflicts: bidirectionality is responsible for one—that is, the conflict between one physical dimension of the stimuli (size) and the *associated feature* (Arabic number) of the other physical dimension of the stimuli (color). The second conflict, which we believe is more primary, is a conflict between the two physical dimensions of the presented stimuli (size and color). We have several reasons to believe that the latter conflict (i.e., between size and color) happens prior to the former conflict (between size and the *associated* number). First, processing size is faster and easier than processing numbers (Tzelgov et al., 1992; Leibovich and Henik, 2013). Second, while the conflict between the two physical dimensions (i.e., primary conflict) involves only one process—comparing magnitude and color—the conflict between the physical and associated dimension (i.e., secondary conflict) involves two processes activation of the number by the color (i.e., bidirectionality) and comparing the symbolic and non-symbolic magnitude dimensions (i.e., number and size).

To sum up, we suggest that the congruency effects found for the non-symbolic content in these studies may also be a consequence of conflict between the magnitude of a stimulus and the magnitude associated with the stimulus color, which arises via the implicit interaction between the color and magnitude dimensions. This assumed magnitude-color association is less accessible to the beholder, whether it is a researcher who examines the data or a synesthete who introspects his/her experience, since typically it is covered by the synesthetes' explicit symbolic number-color association. However, the data is currently insufficient for determining conclusively whether a direct magnitude-concurrent conflict exists and if it does, whether it happens earlier than the magnitude-number conflict.

# **NUMBER-SPACE SYNESTHESIA**

The above suggestion is further supported by studies in the area of numberspace synesthesia. Recently, Gertner et al. (2012) tested two number-space synesthetes and a group of controls on a simple comparison task with three different types of stimuli: (a) Arabic numbers (symbolic stimuli), (b) patterns of dots (non-symbolic discrete stimuli), and (c) sizes of squares (non-symbolic continuous stimuli). Congruency between magnitude and spatial location was manipulated according to the synesthetic numberspace association. The authors found that synesthetes showed a magnitude–space compatibility effect for dot clusters as well as for Arabic numbers. That is, synesthetes were slower to compare two arrays of dots when these non-symbolic magnitudes were presented incompatibly with their symbolic number-space representation. Such a magnitude–space association was not found for controls. In this study, the authors managed to find a magnitudespace congruency effect only for discrete magnitudes but not for continuous magnitudes (physical size). These findings

**number magnitude (and other non-symbolic content) and their specific concurrents.** The activated entities fire back to the inducer. This simultaneous feedback to the same target forms an autonomous connection between the synesthetic concurrent and the non-symbolic content. However, this connection is not always strong enough to cross the consciousness threshold; thus, in most cases it remains implicit.

<sup>1</sup>Please note that we are referring only to the finding regarding the non-symbolic comparison.

cannot entirely rule out the possibility that the dot patterns activated a symbolic number, which in turn, induced the spatial concurrent. However, in a later study of Gertner et al. (2013), a magnitudespace association was observed also for continuous physical sizes. In that study, two groups of synesthetes (and matched controls) performed a numerical Strooplike task (Henik and Tzelgov, 1982). One synesthetic group visualized numbers in a rightward orientation (left to right) and the other group visualized numbers in an upward orientation (bottom to top). Participants had to decide which number was physically or numerically larger (in different blocks). An interesting finding was observed in the neutral condition of the physical block (e.g., 3 3) in which numerical value was kept constant but the stimuli location on the screen and physical size varied2. Synesthetes of both groups, but not controls, showed a size-space interference effect. For example, for a synesthete with an upward number representation, a smaller 3 at the top and a bigger 3 at the bottom were harder to compare (longer RTs) than vice versa. Similarly, for a synesthete with a rightward representation, a smaller 3 on the right and a bigger 3 on the left was harder to compare than vice versa. The authors suggested that physical size is associated with spatial location according to the directionality that corresponds to the synesthetic symbolic number-form. These findings reflect an association that incorporated the non-symbolic dimension and the spatial concurrent—an association that we assume is inherent in the link between numbers and space in number– space synesthesia.

Taken together, the data presented above suggest that synesthetic association is not only restricted to concrete symbolic inducers but can also be evoked by nonsymbolic, discrete, and continuous information. However, it is not clear to what extent synesthetes are aware of such synesthetic associations (and thus, they may be referred to only as semi-synesthetic associations). The data we have at present are still too premature to address this query. However, it seems that there are individual differences regarding this matter. We have noticed over the past several years that some synesthetes do explicitly report having colors or a spatial arrangement for non-symbolic discrete stimuli (i.e., conscious experience of the association); some synesthetes notice it only during a task, after which they report that the incongruent condition was harder or uncomfortable for them (i.e., possibly a semi-conscious experience); and other synesthetes do not report any synesthetic experience in response to non-symbolic stimuli, however, their results indicate such an association (i.e., non-conscious process). Unfortunately, we do not have reliable statistical data about the proportion of synesthetes who explicitly report having a synesthetic association for non-symbolic stimuli.

How does this magnitude-based semisynesthetic association arise? Magnitude is a property inherent in symbolic numbers. In the case of synesthesia, symbolic numbers have the capability of inducing the experience of color or space. Thus, after endless "trials" of symbolic inducer-concurrent associations, it is reasonable that the magnitudinal content adopts some capability to autonomously induce a similar synesthetic concurrent. Specifically, when synesthetes encounter a numerical inducer, the synesthetic concurrent and number magnitude are automatically triggered (Cohen Kadosh et al., 2008b). In turn, this semantically related content (magnitude, space, color, etc.) "fires" back, co-activating the inducer. This repeated backward activation forms autonomous connections between the concurrent and the non-symbolic content of the inducer [based on the rationale of Hebb's (1949) axiom—neurons that fire together wire together] (see **Figure 1**). Thus, nonsymbolic properties that are functionally and anatomically attached to the symbolic numerical system become inevitably associated with the synesthetic concurrent, creating magnitude-based synesthetic associations.

However, in some cases these connections are not strong enough to cross the threshold for a conscious experience, thus, resulting in an implicit, rather than explicit, association between non-symbolic numerical information and colors or space.

# **SUMMARY**

Numbers cannot be disentangled from their semantic meaning. For numerical synesthetes, numbers also cannot be disentangled from their color, space or personality profile. Thus, with time, connections between numbers' semantic networks and synesthetic concurrents are obligatorily formed. In this paper we presented some introductory evidence for this idea; however, further research is needed in order to systematically test the potential existence of implicit (and explicit), direct or indirect associations between non-symbolic continuous magnitude information like size, length, brightness, etc., and synesthetic concurrents. Such research can further enrich our knowledge about the semantic network of numbers and the shared mechanisms of numbers and magnitudes.

# **ACKNOWLEDGMENTS**

This work was supported by the European Research Council under the European Union's Seventh Framework Programme (FP7/2007-2013)/ERC Grant agreement n◦ 295644.

### **REFERENCES**


<sup>2</sup>There were many findings in this research but we are referring only to the one that is relevant for the current opinion paper.

representation. *Prog. Neurobiol.* 84, 132–147. doi: 10.1016/j.pneurobio.2007.11.001


*Received: 02 June 2013; accepted: 29 October 2013; published online: 14 November 2013.*

*Citation: Gertner L, Arend I and Henik A (2013) Numerical synesthesia is more than just a symbolinduced phenomenon. Front. Psychol. 4:860. doi: 10.3389/fpsyg.2013.00860*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Gertner, Arend and Henik. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# The emotional valence of a conflict: implications from synesthesia

# *Amit Perry1,2 \* and Avishai Henik1,2*

<sup>1</sup> Department of Psychology, Ben-Gurion University of the Negev, Beer-Sheva, Israel

<sup>2</sup> Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, Beer-Sheva, Israel

## *Edited by:*

Roi Cohen Kadosh, University of Oxford, UK

### *Reviewed by:*

Roi Cohen Kadosh, University of Oxford, UK Michael Banissy, Goldsmiths University of London, UK

### *\*Correspondence:*

Amit Perry, Department of Psychology, Ben-Gurion University of the Negev, P.O.Box 653, Beer-Sheva 84105, Israel e-mail: perryamit1@gmail.com

According to some synesthetes' reports, their experience involves an emotional sensation in which a conflict between the photism and presented color of a stimulus may evoke a feeling of discomfort. In order to investigate the impact of this experience on performance, two experiments were carried out on two synesthetes and their matched control groups. Experiments were tailored for each synesthete according to her unique photism. Participants were presented with stimuli (numerals or words) in colors and were asked to name the color of the stimulus and to ignore its meaning. Incongruent colors were associated with negative or positive emotional words or with non-emotional words. Not surprisingly, an incongruent color (e.g., 5 presented in yellow to a synesthete that sees 5 in red) slowed down color naming. Conflict situations (e.g., a numeral in an incongruent color) created a negative emotional experience. Most importantly, coherence between a conflict or non-conflict emotional experience and the emotion elicited by the color of the stimulus for a given synesthete modulated performance. In particular, synesthetes were faster in coherent than in incoherent situations. This research contributes to the understanding of emotional experience in synesthesia, and also suggests that synesthesia can be used as an instrument to investigate emotional processes in the wider population.

**Keywords: synesthesia, conflict, emotional valence, emotional coherency, Stroop-like task**

"fpsyg-04-00978" — 2013/12/24 — 18:03 — page 1 — #1

# **INTRODUCTION**

The word synesthesia means mixing of the senses. Indeed, in synesthesia certain stimuli (e.g., an achromatic written number) may give rise to a perceptual experience in additional modalities (e.g., color or taste) not normally associated with them (Grossenbacher and Lovelace, 2001). Although synesthesia appears to be a perceptual phenomenon, it has been reported that some synesthetes also exhibit strong emotional reactions in response to sensory discord or harmony regarding their synesthetic experiences (Cytowic and Ommaya, 1989; Ramachandran and Hubbard, 2001). Namely, a stimulus colored incongruently with the synesthetic color (i.e., "photism"; e.g., for a synesthete who perceives yellow following a presentation of the digit 5, 5 colored in blue is an incongruent stimulus) would lead to discomfort, while a stimulus colored in a congruent color (e.g., the digit "5" colored in yellow) would lead to a calming mood.

The cognitive conflict between a photism (in the above example yellow) and presented color (in the above example blue) is referred to as a congruity effect. Many use Stroop (Stroop, 1935) and Stroop-like tasks in order to study cognitive conflict. In the commonly used Stroop task, participants are presented with colored color words (e.g., BLUE written in red) and are asked to name the color in which the word is presented (red) while ignoring the word's meaning (blue). Participants respond slower to incongruent stimuli (BLUE in red) than to congruent stimuli (RED in red; MacLeod, 1991). Similarly, in the synesthetic version of this task (i.e.,"synesthetic Stroop task"), synesthetes are presented with colored graphemes and are asked to name the color of the grapheme while ignoring the grapheme's meaning. Synesthetes are slower to respond in the incongruent condition than in the congruent condition (Mills, 1999; Mattingley et al., 2001; Dixon et al., 2004; Cohen Kadosh and Henik, 2006).

Research using Stroop-like tasks (not including emotional Stroop tasks) referred only to cognitive aspects of the conflict (MacLeod, 1991). However, there may be an emotional aspect to such conflicts; namely, an incongruent condition may produce uneasiness, resembling the synesthetic discomfort described previously. Botvinick (2007) suggested that the behavioral outcomes of a conflict (e.g., a bias toward the selection of tasks and strategies that minimize the risk of conflict) might be due to its registration as an aversive or negatively reinforcing event by the anterior cingulate cortex (ACC) rather than by its triggering, via the ACC, of compensatory shifts in control. Because synesthetes respond emotionally to a conflict, we used a synesthetic Stroop task to detect the emotional aspects of conflict. This is in line with the suggestion of using synesthesia to further our understanding of neurocognitive processes in the non-synesthetic population (Cohen Kadosh and Henik, 2007).

Synesthetic emotional experience was first examined in a study conducted by Callejas et al. (2007). They found that the perception of an incorrectly colored word (colored in the color wheel opposite to the photism) affected the emotional judgment of MA, a grapheme-color synesthete, such that the valence of an emotional or neutral word, presented in an incongruent color, was rated lower on a 7-point Likert scale than the valence of a word presented in the congruent color. In a second experiment, MA and controls were asked to indicate as fast as possible, whether a colored word was positive or negative. Here the congruity effect interacted with the semantic valence of the word, such that correspondence between the emotional valences of the word and the congruity condition led to faster reaction times (RTs). However, a difference between congruent and incongruent conditions appeared only for positive words.

In a later work done by Hochel et al. (2009), an emotional synesthete–R–described that a feeling of discomfort in response to incongruity between a photism and a presented color arose only when the presented color was not emotionally coherent with his photism. For example, the number 5 is red–apositively valenced color. It does not bother R (in fact, 5 in red has a calming effect on R), nor would it bother R if 5 were printed in another positively valenced color such as purple. Hochel et al. (2009) examined R's performance during an odd/even decision task. Each trial consisted of a number, colored in white, presented in a colored frame that was either congruent or incongruent with the photism associated with the number. The incongruent color was emotionally coherent or incoherent with the photism's valence.

Hochel et al.'s (2009) work shows that for a specific emotional synesthete, colors possess emotional valence. Taking into account that colors possess an emotional meaning also for the non-synesthete population (Adams and Osgood, 1973;Valdez and Mehrabian, 1994), one can only assume the enhanced emotionality of colors exists for a wider range of synesthetes who experience their world through colors. In this context, we distinguish between three sources of emotionality that arise as a result of the conflict situation described in the previous studies. As mentioned, the first source arises from the congruity between presented and synesthetic color (i.e., the conflict itself). The second arises from the emotional meaning of the colors, while the third arises from the emotional valence of the stimulus. In order to control the emotional valence of colors in our study, each color was selected according to the unique photism of each synesthete and thus had idiosyncratic meaning.

Both Hochel et al.'s (2009) and Callejas et al.'s (2007) studies have shown that the performance of a synesthete is influenced by a different aspect of emotionality that is aroused by the conflict. However, neither study considered all three sources. Specifically, Callejas et al. (2007) related to the emotion aroused by the incoherence between the photism and the printed color as well as to the word's valence. However, their selection of colors ignored the possibility that colors possess an emotional valence, resulting, in our opinion, in a lack of an effect for negative words. Similarly, Hochel et al. (2009) reported that numbers do not have any emotional valence for R. However, several photisms (of numbers) were emotional and this might have had some impact on number emotionality. Hence, we suggest that it is hard to determine the actual valence of numbers for R. Not assigning a valence might have introduced uncontrolled variance, which might have reduced the effect.

The current study was designed in order to: (a) further the understanding regarding the emotional experience in synesthesia, and (b) examine emotional aspects of cognitive conflict. The study was conducted on two lexical synesthetes who perceive color in response to the meaning of written or spoken words. Also, a control group of non-synesthetes was matched to the synesthetic subjects. Two experiments using a synesthetic Stroop task were conducted, a week apart from each other.

# **EXPERIMENT 1**

In the first experiment, in order to establish a conflict as being negatively valenced, we aroused only two sources of emotionality by using emotionally neutral stimuli. Because our participants had number–color synesthesia, in addition to their lexical synesthesia, we employed numbers as stimuli since numbers are more neutral than words and this would allow us to use a synesthetic Stroop procedure. The numbers were colored either congruently or incongruently to the synesthetes' photisms. The incongruent condition (for a given synesthete) was created by using several colors associated with negative or positive emotional words or with non-emotional words (see **Figure 1A**). Namely, we created three incongruent conditions that were similar in conflict between the photism and presented colors but different in their emotional valence (i.e., incongruent positive condition, incongruent negative condition, and incongruent neutral condition).

We expected a two-way interaction between group and congruity. Among synesthetes, we predicted a congruity effect would be found, that is, response to incongruent conditions would be slower than to the congruent condition. More importantly, incongruent conditions would differ from one another – an incongruity effect – reflecting effects of the emotional valence elicited by each of the incongruent conditions. No congruity or incongruity effects were expected for the controls.

# **METHODS**

"fpsyg-04-00978" — 2013/12/24 — 18:03 — page 2 — #2

## **PARTICIPANTS**

Participants were selected if their photisms were elicited not just by physical aspects of a word but also by additional conceptual aspects. Two female synesthetes participated in the experiment – AD (23 years old) and BM (20 years old). Both synesthetes had lexical and number–color synesthesia. According to their report, the color of a word was influenced by the word's meaning and not by the word's letters. Both synesthetes reported that color could be elicited by different aspects of a word, physical as well as conceptual, but that a word's emotional valence had a major influence in determining the exact color of the photism. They both said that the appearance of a stimulus in an incompatible color (i.e., a color different from the photism of a given word) elicited a feeling of unease. Also, they indicated that color had some emotional meaning for them. For example, the photism of a name might change due to their familiarization with a person with that name and their feeling toward him/her. In addition, the photism seemed to appear "in their mind's eye," floating in a black (BM) or white (AD) background. According to their self-report, the synesthetes had no history of psychiatric or neurological disorders. Both synesthetes were recruited through ads published on the Ben-Gurion University of the Negev web site and on the internet, and both received financial reward for their participation (i.e., 30 NIS – equivalent to ∼8.50 USD per hour).

"fpsyg-04-00978" — 2013/12/24 — 18:03 — page 3 — #3

Ten control participants (all females) participated in the experiment, of whom five were matched for each synesthete by age and gender. All of them were students in the Department of Psychology at Ben-Gurion University of the Negev and participated in the experiment as part of a course requirement.

All participants, synesthetes and controls, were Hebrew native speakers, with normal or corrected-to-normal vision and had no diagnosis of attention deficit hyperactivity disorder (ADHD) or dyscalculia.

# **STIMULI**

Data collection and stimuli presentation were controlled by an IBM Intel Pentium III laptop computer. Stimuli were presented on a DELL E198PF 19-inch LCD monitor. A microphone was placed on a table between the participant and the monitor. Participants were tested individually. They sat approximately 24 inches from the computer screen. The experiment was programmed using E-Prime software. Each trial consisted of presentation of a number from 1 to 9. The number was printed in Times New Roman font, size 144, and presented at the center of a computer screen. For each synesthete and matched control group, each number appeared in a color congruent or incongruent to the synesthete's unique photism. As mentioned earlier, the incongruent colors were of photisms of emotional and non-emotional words, as reported by the synesthete during the color-matching phase (see Procedure). For each valence (i.e., positive, negative, and neutral) 10 words were selected from the Hebrew lexical norms (Hochman and Henik, 2007). The groups of words were matched in concreteness and familiarity as well as in word-frequency, based on the word-frequency database for printed Hebrew (The

Hebrew University of Jerusalem Psychology Department, 2005). Two colors were selected for each incongruent condition. All in all, there were six incongruent colors for each synesthete (see **Figure 2**).

# **PROCEDURE**

# *Color matching and reliability test*

A month before the experiment took place, the synesthetes were asked to report their photisms in response to selected numbers and words. Each color was encoded using red-green-blue (RGB) values in Microsoft Paint. A second color-matching session took place two weeks after the experiment. The settings of the three colors were converted into a single integer value according to the RGB color model by using the formula: RGB value = Red + (Green × 256) + (Blue × 256 × 256). Additionally, the synesthetes were asked to grade the valence of words selected from the Hebrew norms, between 1 (most negative) and 10 (most positive), in order to establish a compatibility between the subjective feeling of the synesthetes and the objective norms.

## *Synesthetic Stroop task*

Numbers appeared in color and participants were asked to name the color as quickly and accurately as possible while ignoring the number. There were four conditions – one congruent (e.g., 3 in orange), and three incongruent: incongruent positive (e.g., 3 in the color green associated with the word *joke*), incongruent neutral (e.g., 3 in purple associated with the word *wild*), and incongruent negative (e.g., 3 in bordeaux associated with the word *enemy*; see **Figure 1A**).

Due to the fact that colors used in the experiment uniquely represented each synesthete's photisms, a color naming block composed of 63 trials was carried out at the beginning of the experiment for fine tuning of color clarification. In this block, in each trial a color appeared on screen and the subjects were asked to name the color as quickly as possible. This enabled them to adjust to color names. Following this, a practice block was carried out, which was made up of 42 trials corresponding to the experimental trials (i.e., colored numbers). Finally, a total of 432 trials were run in six blocks of 72 trials each (18 trials for each condition).

Each trial consisted of a fixation cross (500 ms) followed by a blank screen (300 ms) and then by a colored number display that remained in view until a vocal response was obtained. This was followed by a blank screen until the experimenter encoded the

value of the response using a keyboard (**Figure 3**). The background color on the screen was the color in which the photism floated in the mind's eye of each synesthete – a black background for BM and a white background for AD.

### **DESIGN**

The variables manipulated were group (synesthetes vs. controls) and congruity (congruent, incongruent negative, incongruent neutral, incongruent positive). Thus, we had a 2 × 4 factorial design with subjects as a random factor. Group was manipulated between subjects, while congruity was manipulated within subjects. The dependent variables were mean RT in milliseconds and accuracy.

# **RESULTS AND DISCUSSION**

### **RELIABILITY TEST**

"fpsyg-04-00978" — 2013/12/24 — 18:03 — page 4 — #4

We computed the correlations between the RGB values for the two matching sessions for each synesthete. The colors for numbers had a consistency of 0.99 (*p* < 0.0001) for AD and 0.94 (*p* < 0.0001) for BM. Word colors had a consistency of 0.74 (*p* < 0.001) for AD and 0.78 (*p* < 0.001) for BM. This difference between consistencies may be due to the more frequent introspection (of both synesthetes) of number photisms than word photisms, possibly because of the finite number of single-digit numbers compared to the large number of words. Also, due to their physical and conceptual properties, words are more complicated stimuli than numbers and this may lead to a less stable photism.

# **SYNESTHETIC STROOP TASK**

### *Accuracy rate*

Mean accuracy for both groups was high and ranged between 0.94 and 0.98, implying participants had adjusted to color names during the color naming block. This also indicated that differences between groups could not be explained by speed-accuracy tradeoff. The correlation between mean RTs and accuracy rate was 0.26 for synesthetes (*p* = 0.73) and −0.12 for controls (*p* = 0.88).

# *Reaction time*

Incorrect trials, as well as RTs under 150 ms or above 1,500 ms (i.e., 0.7% of correct responses) were excluded from the analysis. Results of the two synesthetes were similar and were grouped.

Mean RTs for correct trials of each of the four conditions (one congruent and three incongruent) were subjected to an analysis of variance (ANOVA) with group and congruity as independent variables. We found an interaction between group and congruity, *F*(3,30) = 4.9, *MSE* = 742, *p* < 0.007 (see **Figure 4**). Further analyses showed an effect of congruity for the synesthetes, *F*(3,8) = 7, *MSE* = 742, *p* < 0.01, which was due to faster RTs to the congruent condition than to the incongruent conditions, *F*(1,10) = 7.5, *MSE* = 1,296, *p* < 0.02, similar to what was found in previous studies (Mills, 1999; Mattingley et al., 2001; Dixon et al., 2004). More importantly, there was an incongruity effect, showing gradual changes from the slowest positive incongruent condition, *F*(1,10) = 11, *MSE* = 354, *p* < 0.007, to the fastest negative incongruent condition, *F*(1,10) = 5, *MSE* = 574, *p* < 0.04.

In the introduction we suggested that conflict situations give rise to a negative emotional reaction such as discomfort. Accordingly, the current results show that the incongruent negative condition produced faster RTs than the incongruent neutral or the incongruent positive conditions. Namely, coherence in emotion, elicited by the relation between the conflict and the color emotional association (i.e., negative incongruent condition), led to faster responding. Likewise, incoherence in emotion (i.e., positive incongruent condition) led to slower responding than in the presence of the conflict alone (i.e., incongruent neutral

condition). In comparison, Hochel et al. (2009) found no difference between positive and negative incongruent conditions (created by a color frame around the stimulus) for the neutral photism condition (i.e., a digit whose color photism was nonemotional). However, the direction of results was similar to the current results, showing faster RT for the negative incongruent condition.

The controls showed a congruity effect, *F*(3,8) = 18, *MSE* = 742, *p* < 0.001, with the incongruent negative condition being fastest, *F*(1,10) = 4, *MSE* = 805, *p* < 0.001. In addition, the congruent condition did not differ from the neutral and positive incongruent conditions, *F*(1,10) = 1.87, *MSE* = 1,059.34, *ns*, and the neutral incongruent condition and the positive incongruent condition did not differ from each other, *F*(1,10) = 1.78, *MSE* = 361.37, *ns*. Because the incongruent conditions did not produce a conflict in the control group, it is possible that the effect (i.e., incongruent negative condition being fastest) was related to the use of vocal responding rather than motor responding. The first type of responding requires *retrieval* of a color name while the second type only requires recognition of a color name according to the options of response specified. Namely, the particular colors used in the negative incongruent condition were more common colors than other colors in the experiment, so it was easier to recall their names from memory. This suggestion is supported by a similar pattern that was generated in the color naming block.<sup>1</sup>

Finally, a main effect for group was found, *F*(1,10) = 14.5, *MSE* = 14,857, *p* < 0.003, showing that synesthetes responded slower than controls did. This effect can be explained by the synesthetes' need to ignore not only the stimulus meaning but also its photism (Callejas et al., 2007).

# **EXPERIMENT 2**

"fpsyg-04-00978" — 2013/12/24 — 18:03 — page 5 — #5

In the second experiment, we employed negatively and positively valenced words as emotional stimuli, adding another source of emotionality (i.e., word meaning) to the color and conflict valences. That way, we could: a) investigate more complex effects of emotional coherence on performance, and b) further investigate a conflict's negative valence.

As in Experiment 1, the words were colored either congruently or incongruently to their photism. Two incongruent conditions, derived from negative or positive words, served as the color inducers (see **Figure 1B**). For each group of words (i.e., negative or positive), we created an incongruent coherent condition in which valences of the word and incongruent color were emotionally coherent (i.e., a negatively valenced word colored in a negatively valenced word's photism or a positively valenced word colored in a positively valenced word's photism), and we also created an incongruent incoherent condition, in which valences of the word and incongruent color were emotionally incoherent (i.e., a negatively valenced word colored in a positively valenced word's photism or a positively valenced word colored in a negatively valenced word's photism).

<sup>1</sup>This explanation is supported by an ANOVA carried out similarly on the color naming block. It showed that among controls, RT for the colors used in the incongruent negative condition was faster to begin with, *F*(1,9) = 60.94, *MSE* = 621.59, *p* < 0.001. In contrast, among synesthetes, no effect of colors was found in the color naming block, *F*(3,3) = 1.09, *MSE* = 2,264, *ns*.

One could hypothesize that among synesthetes, the slowest RT would be found for the incongruent incoherent conditions for positive and negative words. However, coherency between all three sources of emotionality existed only for the negative words in the incongruent coherent condition, while no actual coherent condition was formed for positive words, due to incoherency between word and conflict valence. Thus, we expected an incongruity effect for negative words only.

# **METHODS**

### **PARTICIPANTS**

BM, AD, and the control subjects from Experiment 1 participated in Experiment 2.

## **STIMULUS**

Each trial consisted of a positively or negatively valence word, which constituted an emotional stimulus. The word was printed in Times New Roman font, size 72, and presented at the center of a computer screen. For each synesthete and matched control group, the word appeared in a color congruent or incongruent to the synesthete's unique photism. The incongruent color represented the photism of a positively or a negatively valenced word, as reported by the synesthete during the color matching phase (see Procedure in Experiment 1). In each of the incongruent conditions, four colors were presented. All in all, there were eight incongruent colors, for each synesthete (**Figure 5**).The specific words that were used in the experiment and from which the incongruent colors were selected, were taken from the same list prepared for Experiment 1. 18 words were selected – nine classified as positive and nine classified as negative.

### **PROCEDURE**

### *Synesthetic stroop task*

Words appeared in color and the subjects were asked to name the color as quickly and accurately as possible while ignoring the meaning. There were three conditions – one congruent (e.g., the word *enemy* in bordeaux), and two incongruent: incongruent coherent (e.g., the word *enemy* in gray associated with the word *desperate*), and incongruent incoherent (e.g., the word *enemy* in purple associated with the word *magical*; see **Figure 1B**).

incongruent incoherent condition for positive words. **(B)** Positive

As in Experiment 1, at the beginning of the experiment participants were presented with a color naming block composed of 78 trials, where they had to name a color that appeared on the screen as quickly as possible, in order to adjust to color names. Afterward, the words were separated into different practice and experimental blocks, according to their valence. For each valence (positive/negative), a practice block of 54 trials was created, corresponding to the experimental trials (i.e., colored words), followed by three experimental blocks, each composed of 108 trials (36 trials for each condition). Valence order was balanced between synesthetes and was identical for each synesthete and her controls. Thus, AD received the positive words blocks first, while BM received the negative words blocks first. All in all, there were 648 experimental trials.

The experiment was held under the same conditions as Experiment 1. Also, the experimental trial was identical to that of Experiment 1 apart from words appearing instead of numbers.

## **DESIGN**

The variables manipulated were group (synesthetes vs. controls), valence (negative vs. positive), and congruity (congruent, incongruent coherent, incongruent incoherent). Thus, we had a 2 × 2 × 3 factorial design with subjects as the random factor. Group was manipulated between subjects, while valence and congruity were manipulated within subjects. The dependent variables were mean RT and accuracy.

# **RESULTS AND DISCUSSION ACCURACY RATE**

Mean accuracy for both groups was high and ranged between 0.96 and 1.00. As in Experiment 1, this implied that participants had adjusted to color names during the color naming block. The correlation between mean RTs and accuracy rate was 0.0 for synesthetes (*p* = 1.0) and 0.5 for control (*p* = 0.667).

## **REACTION TIME**

As in Experiment 1, incorrect trials, as well as RTs under 150 ms or above 1,500 ms (0.2% of correct responses), were excluded from the analysis. In addition, results of the two synesthetes were similar and were grouped.

"fpsyg-04-00978" — 2013/12/24 — 18:03 — page 6 — #6

parentheses.

Mean RTs for correct trials for each of the six conditions (one congruent and two incongruent for each valence) were subjected to an ANOVA with group, valence, and congruity as independent variables. We found a marginally significant interaction between group, valence, and congruity, *F*(2,40) = 3, *MSE* = 570, *p* < 0.06 (see **Figure 6**).

Further analyses found a significant simple two-way interaction of valence × congruity for synesthetes, *F*(2,4) = 8.8, *MSE* = 213, *p* < 0.0001. This was due to a simple main effect for congruity for negative words, *F*(2,19) = 4, *MSE* = 213, *p* < 0.03, but not for the positive words, *F* < 1. Furthermore, for the negative words, the incongruent incoherent condition was significantly different from the congruent and incongruent coherent conditions, *F*(1,20) = 8, *MSE* = 714, *p* < 0.001, and no difference was found between the congruent condition and the incongruent coherent condition, *F*(1,20) = 0, *MSE* = 425, *ns*. Namely, RT for the incongruent incoherent condition was slower than the other congruity conditions among synesthetes for negative words only.

These results further support the notion that conflict has a negative valence. Thus, coherence in emotion, elicited by conflict, color and valence of a word led to faster responding. Importantly, Hochel et al.'s (2009) report fits in with our results. They found slower RT for positive photisms than for neutral and negative photisms for the negative incongruent condition only.

A significant simple two-way interaction of valence × congruity was also found among controls, *F*(2,36) = 3.6, *MSE* = 610, *p* < 0.03. However, a further analysis showed a simple main effect for positive words, *F*(2,19) = 6.8, *MSE* = 610, *p* < 0.001, but not for the negative words, *F* < 1. The effect for positive words was due to the difference between the incongruent coherent condition and the congruent and incongruent incoherent conditions, *F*(1,20) = 14, *MSE* = 410, *p* < 0.001. No difference was found between the congruent condition and the incongruent incoherent condition, *F*(1,20) = 0.05, *MSE* = 729, *ns*. Namely, RT for the incongruent coherent condition was faster than for the other congruity conditions among controls for positive words only. These results are surprising since for the control group no conflict existed between the photism and the presented color. One could expect that if a difference was found for the control group, it was due to incoherency between word and color emotional valence, resulting from a similar perception of color emotionality for non-synesthetes. However, in this case, the colors used in the incongruent coherent condition were of the same valence as the colors used in the congruent condition, for positive words. As such, no difference had been expected between these conditions.

Finally, a main effect for group was found, *F*(1,20) = 10, *MSE* = 12,061, *p* < 0.001, showing, as in Experiment 1, that synesthetes responded slower than controls did.

# **GENERAL DISCUSSION**

In summary, both experiments showed an effect of congruity for synesthetes. In Experiment 1, we found a congruity effect showing slower RT for incongruent conditions than for the congruent condition, as well as an incongruity effect showing gradual changes from the slowest positive incongruent condition, to the neutral incongruent condition, until the fastest negative incongruent condition. In Experiment 2, we found the incongruent incoherent condition was the slowest among congruity conditions for negative words only.

Both experiments indicate that emotional coherence influences performance. Namely, faster responses were obtained in the presence of coherence (i.e., incongruent negative condition in Experiment 1; incongruent coherent condition for negative words in Experiment 2) compared to absence of coherence (i.e., incongruent positive condition in Experiment 1; incongruent

"fpsyg-04-00978" — 2013/12/24 — 18:03 — page 7 — #7

incoherent condition for positive words in Experiment 2), and to a pure conflict (i.e., incongruent neutral condition in Experiment 1). This pattern is especially interesting in light of the evidence that responding to negative stimuli is usually slower than responding to positive or neutral stimuli (MacLeod and Rutherford, 1992; Fox et al., 2001).

In addition, our results fit in with the assumption that **conflict situations are associated with negative valence**. Major support comes from the finding that RT for the incongruent coherent condition was similar to that of the incongruent incoherent condition for positive words in Experiment 2. The two incongruent conditions differed in their coherency between word and color valence. However, due to the negative valence of conflict, conflict valence and word valence were incoherent in both incongruent conditions, and actually created two incoherent conditions, compared to the congruent condition. These findings also reinforce Botvinick's (2007) suggestion regarding a conflict's aversive quality as an explanation for behavioral outcomes in his studies. However, further examination using physiological measures such as galvanic skin response (GSR) is needed in order to establish the emotional differences between experimental conditions.

The ability of an incongruent color to elicit a word whose photism it represents, or at least its emotional valence, supports the assumption regarding the bi-directionality of synesthesia (Cohen Kadosh et al., 2005). Another possible explanation could be that colors possess emotional valence due to fundamental differences in dimensions: hue, saturation, and luminance. Similarly, a connection was found between number quantity size and color saturation and luminance (Cohen Kadosh et al., 2007).

Beyond providing a deeper understanding of the emotional experience involved in synesthesia, our research suggests that similar to exploiting synesthesia to shedfurther light on our understanding of cognitive processes (Cohen Kadosh and Henik, 2007), it can contribute to our understanding of emotional aspects of human experience. It is important to note that due to difficulty recruiting suitable participants for this research and the fact that only two synesthetes were found to match the study's criteria, generalization of the current findings to the general population should be done with caution. Importantly, we suggest that in the case of our unique sample of participants, synesthesia enables a glimpse of emotional aspects of cognitive conflict, and how different characteristics of a stimulus, particularly its color, involve an emotional meaning.

# **ACKNOWLEDGMENT**

This research was partially supported by a grant to Avishai Henik from the Israeli Science Foundation (grant 1799/12).

### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 22 May 2013; accepted: 09 December 2013; published online: 26 December 2013.*

*Citation: Perry A and Henik A (2013) The emotional valence of a conflict: implications from synesthesia. Front. Psychol. 4:978. doi: 10.3389/fpsyg.2013.00978*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Perry and Henik. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

"fpsyg-04-00978" — 2013/12/24 — 18:03 — page 8 — #8

# Affect-related synesthesias: a prospective view on their existence, expression and underlying mechanisms

# *Nele Dael , Guillaume Sierro and Christine Mohr\**

*Institute of Psychology, University of Lausanne, Lausanne, Switzerland*

### *Edited by:*

*Michael Banissy, Goldsmiths University of London, UK*

### *Reviewed by:*

*Flor Kusnir, University of Glasgow, UK Elias Tsakanikos, University of Roehampton, UK*

*\*Correspondence: Christine Mohr, Institute of Psychology, University of Lausanne, Quartier Mouline, Bâtiment Géopolis, 1015 Lausanne, Switzerland e-mail: christine.mohr@unil.ch*

The literature on developmental synesthesia has seen numerous sensory combinations, with surprisingly few reports on synesthesias involving affect. On the one hand, emotion, or more broadly affect, might be of minor importance to the synesthetic experience (e.g., Sinke et al., 2012). On the other hand, predictions on how affect could be relevant to the synesthetic experience remain to be formulated, in particular those that are driven by emotion theories. In this theoretical paper, we hypothesize that a priori studies on synesthesia involving affect will observe the following. Firstly, the synesthetic experience is not merely about discrete emotion processing or overall valence (positive, negative) but is determined by or even altered through cognitive appraisal processes. Secondly, the synesthetic experience changes temporarily on a quantitative level according to (i) the affective appraisal of the inducing stimulus or (ii) the current affective state of the individual. These hypotheses are inferred from previous theoretical and empirical accounts on synesthesia (including the few examples involving affect), different emotion theories, crossmodal processing accounts in synesthetes, and nonsynesthetes, and the presumed stability of the synesthetic experience. We hope that the current review will succeed in launching a new series of studies on "affective synesthesias." We particularly hope that such studies will apply the same creativity in experimental paradigms as we have seen and still see when assessing and evaluating "traditional" synesthesias.

**Keywords: synesthesia, emotion, affect, appraisal, stability, state dependency, crossmodal**

# **INTRODUCTION TO THE PHENOMENON OR PHENOMENA**

The etymology of synesthesia is derived from the Greek *syn* meaning "together" and *aesthesis* meaning "sensation"; which means literally "union of the senses" (Cytowic, 2002). In this regard, synesthesia can be understood as a phenomenon wherein a simultaneous dual perception arises from a single sensory input (the inducer). The additional percept (the concurrent) belongs either to the same or to a different sensory modality. Prevalence studies could show that one of the most frequent forms of synesthesia is grapheme-color synesthesia (Day, 2005; Rich et al., 2005; Simner et al., 2006). In this form, the perception of an achromatic grapheme (e.g., the letter "p") is reported to also trigger the subjective perception of a color experience with the respective color perception being idiosyncratic to the individual (e.g., the color "orange"). In some cases, this idiosyncratic color experience is accompanied by texture, shape, and movement qualities (Tyler, 2005). Other forms of synesthesia include reports that seeing or hearing words, numbers, or sounds can activate the simultaneous perception of smells, tastes, and shapes (Cytowic, 2002; Ward and Simner, 2003; Tyler, 2005). Yet, there seem to be countless types of possible synesthetic experiences such as the combinations of time and space (Price and Mentzoni, 2008; Smilek et al., 2010), vision and touch (Banissy and Ward, 2007; Banissy et al., 2009), sound and touch (Beauchamp and Ro, 2008), observing pain in others and feeling this pain oneself (Fitzgibbon et al., 2010), and the attribution of animate-like qualities (e.g., personality, gender) to sequential linguistic units (e.g., letters, numerals) (Simner and Holenstein, 2007).

Reports on synesthesia have existed for centuries (Galton, 1880; Flournoy, 1892; Jewanski et al., 2009). Scientific activities have occurred in waves such as the one around the turn of the last century (Alford, 1918; Mahling, 1926) and a more recent one with the renaissance of the study of cognition (Marks, 1975). Since then, researchers accumulated evidence for the genuineness of the idiosyncratic synesthetic experience (Rich and Mattingley, 2002) using well-established cognitive paradigms (e.g., the Stroop paradigm) and neuroimaging methods. Despite small sample sizes, many of these carefully designed studies were accepted for publication in highly prestigious journals (Dixon et al., 2000; Beeli et al., 2005). Today, the prevalence rate of synesthesia is estimated to be comparable in both sexes and to be evident in around 4% of the population (Ward and Simner, 2005; Simner et al., 2006). This relatively high prevalence rate together with the description of and search for (i) individual differences in the synesthetic experience (Dixon et al., 2004; Skelton et al., 2009; Rouw and Scholte, 2010), (ii) cognitive correlates (Azoulai et al., 2005; Yaro and Ward, 2007; Rothen and Meier, 2009; Banissy et al., 2011; Ward, 2013), and (iii) personality and clinical correlates (Bors, 1979; Forsman, 1993; Ro et al., 2007; Logsdail, 2009; Banissy et al., 2013) contributed to numerous published group studies.

The most studied and thus the most representative type of synesthesia is the developmental one (also known as constitutional, idiopathic or strong synesthesia) (Grossenbacher, 1997; Harrison and Baron-Cohen, 1997; Martino and Marks, 2001). Important to this form is that the experience is present from early childhood, automatic, idiosyncratic, and remains supposedly stable over the course of a lifetime. Apart from the developmental type of synesthesia, the existence of and relationship between different types of synesthesia are discussed (e.g., Rogowska, 2011; Sinke et al., 2012; Ward, 2013), in particular those acquired following particular life events or those affecting the central nervous system more directly. For instance, synesthetic experiences have been reported in association with migraine auras (Podoll and Robinson, 2002; Alstadhaug and Benjaminsen, 2010), post-hypnotic suggestion (Terhune et al., 2010), thalamic lesions (Beauchamp and Ro, 2008), and drug ingestion (Siebert, 1994; Sinke et al., 2012). These latter types, however, lack the features known from the genuine, i.e., developmental, type (automaticity, stability, idiosyncrasy) (Sinke et al., 2012).

# **SYNESTHESIAS INVOLVING EMOTIONAL EXPERIENCES**

In their comprehensive comparison of developmental, acquired and drug-induced synesthesias, Sinke et al. (2012) discuss interesting phenomenological differences. For instance, external sensory input is the common inducer of developmental synesthesias (vision, audition, touch) with body-related sensory input playing no such role. The authors (as well as the current authors) are unaware of a reported case of synesthesia for which proprioceptive or vestibular information was accompanied by a concurrent synesthetic experience. Pain however does seem implicated in synesthetic experiences (Dudycha and Dudycha, 1935; Fitzgibbon et al., 2010). In one case, a person reports that "each type of pain always gives rise to the same photism with its characteristic form, color or brightness" (p. 58) and varies with the extent of pain (Dudycha and Dudycha, 1935). The same synesthete reports that for dull and throbbing pains, the photism "varies in size and seems to correspond exactly with the affected part" (p. 59). In another case of synesthesia for pain, the respective person reports to empathize with another person's pain and to experience the observed pain, or even the imagined pain, as if it was the own (Bradshaw, 2001; Giummarra and Bradshaw, 2009; Fitzgibbon et al., 2010). Thus, although proprioceptive and vestibular sensory experiences do not seem related to synesthetic experiences, bodily experiences may be related—in particular those with an affective component (own pain or observation of other's pain). This latter observation is important when now commenting on affectivity (Sinke et al., 2012).

On page 1426, Sinke et al. (2012) write about the affectivity of synesthetic experiences. The overall conclusion was that "in genuine synesthesia, the impact of affectivity is low." The authors state that in developmental synesthesia the content of the synesthetic experience (i) is commonly not affective or emotional in itself, and (ii) is not influenced by current affective or emotional state. Surely, it has been noted anecdotally and experimentally that affective experiences may occur as a secondary effect. For example, an early report (Alford, 1918) describes a synesthete who gave up music because the respective synesthetic experiences were disturbing. Recent studies report that synesthetes react with feelings of "correctness" or "pleasantness" to experiences in which the synesthetic and actual perceived features of a stimulus match (e.g., a grapheme is presented in its synesthetically matching color) and react with feelings of "incorrectness" or "unpleasantness" in the case of mismatches (e.g., a grapheme is presented in a synesthetically non-matching color) (Ramachandran and Hubbard, 2001a; Cytowic, 2002; Callejas et al., 2007). Such subjective experiences yielded psychophysiological correlates (Hochel et al., 2009).

Additional reports would support the possibility that affect is indeed an important sensory experience to developmental synesthesia, and can be so in three possible ways. The affective component is (1) the trigger or inducer (Cutsforth, 1925; Dudycha and Dudycha, 1935; Fitzgibbon et al., 2010), (2) the concurrent (Ramachandran and Brang, 2008; Ramachandran et al., 2012), or (3) neither the inducer nor the direct concurrent, but a moderator or mediator of the synesthetic coupling (Cytowic, 1989; Emrich et al., 2004; Ward, 2004; Simner and Holenstein, 2007; Hochel et al., 2009; Ramachandran et al., 2012). For instance, reactions to words of affective valence as well as to personally known faces triggered emotionally mediated color synesthesia (Ward, 2004). This author replicated previous reports that darker and less saturated colors (e.g., brown, black) tend to be associated with negative emotions and lighter and more saturated colors (e.g., yellow, green, red) tend to be associated with positive emotions (p. 770). In another single case study, independent authors reported that "Pleasing pictures and faces were typically red to R, while repulsive visuals or unpleasant human faces elicited a pale green color in R's mind's eye" (Hochel et al., 2009; p. 705). In other cases, feeling particular textures triggered specific emotions (Ramachandran and Brang, 2008), and emotional situations triggered the synesthetic experience of color (Cutsforth, 1925). Synesthetes are also known for liking or disliking certain letters based on their apparent "personality"—a synesthetic form coined "ordinal linguistic personification" in which letters have a gender, personality, and emotional connotation (Simner and Holenstein, 2007; Smilek et al., 2007). Moreover, overall affective judgment of colors was influenced by the emotional coherence between the stimulus and photism (Hochel et al., 2009). Finally, a synesthete with a diagnosis of Asperger's syndrome better understood his and other persons' emotions when he could associate them to a particular color (Ramachandran et al., 2012).

These reports would suggest that synesthesias involving affect or even concrete emotions are indeed thinkable, and might, as will be argued in the following, imply facets little considered so far. This argumentation will firstly return to the relevance of physical experiences to emotion, before discussing cognitive appraisal in emotion elicitation, the difference between emotion and affect, crossmodal interactions in non-synesthetes, and dynamic changes of sensory experiences with the perceivers' emotional and affective states. Subsequently, we will briefly mention some studies and reports that question the stability and durability of the synesthetic as well as crossmodal experience. Finally, we will conclude that some synesthetic experiences might change in quality or rather in quantity with ongoing emotional and affective states.

# **THE ROLE OF NEUROPHYSIOLOGICAL EXPERIENCES AND COGNITIVE APPRAISAL IN EMOTION THEORIES**

Emotion researchers frequently disagree on what constitutes an emotion (Scherer, 2005; Mulligan and Scherer, 2012). Historical traditions (James, 1890; Schachter, 1964) and contemporary emotion theories (for example Cacioppo et al., 2000) agree that neurophysiological (e.g., respiratory, cardiac) changes are crucial to emotion processing (for a review, see Kreibig, 2010). Such changes are considered essential to mobilize an organism (e.g., a human) to respond effectively to a given situation. For instance, back in my neighborhood, I am driving around in my car to find a parking space. In front of me, I suddenly see a ball that is rolling onto the street. In this situation, most people are quickly alarmed, experiencing neurophysiological changes that prepare them for action such as to brake hard to avoid the moving object. Our heart rate might increase, our breathing becomes faster and our hands are getting sweaty. The activated peripheral subsystems (cardiovascular, respiratory, electrodermal) respond instantly to situations of potential danger. Yet, the potential danger is not only about the rolling ball, but also about the possibility that a person, perhaps a child, suddenly appears running behind the ball. This possibility, however, goes beyond the perception of the actual situation, it relates to our knowledge of the world and our appraisal of the situation. Indeed, our neurophysiological changes might be particularly strong because hitting the imagined child is much more worrisome than hitting the ball. Thus, the emotional experience is the result of situational elaboration or appraisal, rather than the mere processing of sensory input.

In this regard, it is important to mention that such neurophysiological changes do not sufficiently explain the full emotion process. Firstly, most neurophysiological changes are not specific to a given emotion. Second, the same neurophysiological changes do not necessarily lead to the same subjectively experienced emotion or behavioral response (Cacioppo et al., 2000). While our ball scenario indicated possible danger resulting in a fear or alarm reaction, similar activations of peripheral subsystems (cardiovascular, respiratory, electrodermal) can also accompany the experience of anger (Kreibig, 2010). The latter may occur when one circles around one's neighborhood without finding a free parking space. Needless to say, the fear reaction is subjectively quite unlike the anger reaction despite both being about obstructing events. It has long been understood that the activation of the sympathetic nervous system prepares the organism for both fight and flight responses (Cannon, 1929; Bradley and Lang, 2000). Neurophysiological changes might thus be necessary to have emotional experiences and reactions, but they are largely emotion unspecific. Also, the same emotion can be associated with different patterns of neurophysiological changes (Cacioppo et al., 2000).

Crucial is the way an individual cognitively processes a situation that leads to a behavioral response. When seeing that a ball is rolling onto the street, one individual might speed up to quickly pass the scene of potential danger, while another might brake hard to avoid reaching the scene of potential danger. Also, the lack of a parking space may facilitate an anger response when the driver is in a hurry, and especially so when the driver blames someone else (a badly parked car occupies two parking spaces). Thus, the same event might trigger different emotional experiences and behaviors both between individuals and within individuals at different time points depending on how the individual appraises the event. "Because appraisals intervene between situations and emotions, *different individuals who appraise the same situation in significantly different ways will feel different emotions; and a given individual who appraises the same situation in significantly different ways at different times will feel different emotions*" (Roseman and Smith, 2001; p. 6).

Accordingly, emotions are not "simple" sensory experiences of neurophysiological change, but involve cognitive processes contributing to the elicitation, differentiation, and regulation of emotions (Moors, 2009). Emotions are not directed toward random events or objects. On the contrary, emotional reactions are mainly intentional in that they are directed toward or are about something specific: I am afraid of hitting the child, I am angry at the driver blocking two parking spaces, etc. Emotional intentionality implies again that a given situation or object has gone through some form of prior evaluation, however, minimal. Indeed, in cognitive appraisal accounts (e.g., Scherer, 2001), appraisals range from low level relevance detection (e.g., the rolling ball as a sudden and novel stimulus demanding attention) to higher level goal congruency processing (e.g., lack of parking spaces as an obstacle to relax at home after a demanding working day), coping potential (e.g., evaluating the time needed to stop the car before hitting a potentially appearing child), to the very high level evaluation of the social context with respect to personal and cultural standards (e.g., social evaluation if having hit a playing child) (for reviews, see Roseman and Smith, 2001; Moors, 2009).

Importantly, cognitive appraisal theories would speak against a widely held assumption of modularity in emotion processing such as in discrete emotion theories (Ekman, 1992). Modularity here indicates that an emotion might represent a unique and discrete response pattern that was triggered by a specific affect program, established evolutionarily. This modularity was suggested for prototypical or basic emotions of anger, fear, sadness, disgust, and happiness (Ekman, 1999). For instance, fear would be processed by a discrete neural network in the same way as vision or audition would be processed by discrete visual and auditory neural networks, respectively (Ohman and Mineka, 2001). The above mentioned appraisal rationale would, however, imply that emotional experiences and responses are not simply the result of modular processes that are automatically elicited by the objective properties of a given event or object. Rather, emotional experiences and responses would result from an individual's situational appraisal of a given situation. The quality (type) and quantity (intensity) of the experiences are likely characterized by nonspecific neurophysiological changes in conjunction with specific event appraisals.

Acknowledging the relevance of such cognitive appraisal mechanisms, we equally acknowledge the possibility that these mechanisms lead to systematic differences in the synesthetic experience. In other words, the fact that the same stimulus can lead to qualitatively and quantitatively different affective experiences both within and across individuals may contribute to a potential dynamism of the synesthetic experience (see below). Moreover, the inconsistency with which objective properties of

specific stimuli and events are linked with subjective affective experiences may have led to an underestimation of the prevalence of emotional synesthesias. The emotion literature would point toward consistent couplings of affective experiences to a synesthete's *appraisal* of the stimulus. For instance, Ward (2004) reported that GW experiences synesthetic colors in response to different types of stimuli (names of personal acquaintances, emotional words), their connection shaped by personal life experience and affective (valenced) reactions toward these stimuli. As will be argued below, such connections, shaped over a lifetime, might be accompanied by connections that change with a given situation.

Before moving to the following sections, it seems constructive finalizing this section by noting the basic distinction between emotion and affect. Novices to emotion research tend to meddle with various concepts important to emotion or affect theories. Affective scientists, indeed, make very clear distinctions between affect, emotion, and other affective phenomena. Affect basically refers to mental states characterized by evaluative feelings. Affective states represent a broad and overarching collection of "affective subtypes" such as emotions, moods, attitudes, preferences, interpersonal stances, and affect dispositions (Scherer, 2005). Both psychophysiological reactions and cognitive appraisal influence the affective experience to different degrees. By inference, such reactions and appraisals could be relevant to affectrelated or affect-modulated synesthesias. Before considering temporary aspects of these possibilities (stable vs. dynamic affect relationships), we account for ideas on the origin of synesthesia because they influence our understanding on how idiosyncratic and stable synesthesia may occur.

# **CROSS-WIRING IN SYNESTHESIA AND CROSSMODAL PROCESSING IN NON-SYNESTHETES**

According to aetiological models, synesthetic experiences emerge from co-activation of otherwise independent sensory, closed modular systems (Baron-Cohen and Harrison, 1993). This coactivation might emerge from associative learning (Galton, 1880; Stevenson et al., 1998) disinhibited feedback (Grossenbacher and Lovelace, 2001), enhanced neural connectivity (Rouw and Scholte, 2007; Bargary and Mitchell, 2008), and/or neural crosstalk (Hubbard and Ramachandran, 2001; Ramachandran and Hubbard, 2001b, see also Smilek et al., 2001; Ward, 2013). While none of these models can yet predict in which way these neuronal correlates lead to synesthetic experiences (Deroy and Spence, 2013), it has repeatedly been suggested that such forms of coactivation might also be relevant to crossmodal processing in non-synesthetes (e.g., Simner, 2012 for a recent account, but see Deroy and Spence, 2013). This suggestion is in line with independent notions that synesthesia and crossmodal perception in non-synesthetes might follow a continuum (e.g., Martino and Marks, 2001; Cohen Kadosh and Terhune, 2012; see also Deroy and Spence, 2013 for a critical account on this perspective). Crossmodal mappings common to both synesthetes and nonsynesthetes have indeed been reported such as for pitch-lightness (Ward et al., 2006), vision-touch and spatial-numeric interactions (Sagiv and Ward, 2006).

Following this reasoning, and assuming that affective experiences are relevant to the richness of possible crossmodal experiences (Collier, 1996; Palmer et al., 2013b) (and by inference synesthetic experiences as well), we would expect for nonsynesthetes (i) that stimuli such as letters, words, numbers, and sounds go systematically with experiences such as colors, brightness, and shapes (see e.g., Collier and Hubbard, 2004) but also (ii) that sensory and affective experiences are linked such that they relate to each other directly, or that sensory experiences are changed by an individual's current affective state.

Studies show systematic associations between color and emotion terms in non-synesthetes from different cultural backgrounds (D'Andrade and Egan, 1974; Johnson et al., 1986; Hupka et al., 1997). The non-synesthetes in Ward (2004) associated color brightness and saturation levels with the pleasantness dimension of emotion, in line with previous findings (Valdez and Mehrabian, 1994). Smiling faces appear brighter than non-smiling faces (Song et al., 2012) and redder faces appear angrier than less red faces (Yasuda et al., 2007). Palmer et al. (2013b) reported systematic mappings between stimuli from different modalities in normal populations. The authors observed significant correlations of both emotional ratings of faces and musical excerpts with emotional ratings of colors chosen to go best with the faces or musical excerpts. These correlations were explained by common affective experiences evoked by or associated with these stimuli. In another study, systematic associations were shown between emotion terms and perceptual aspects such as shape, size and direction, color, music, and sound (Collier, 1996). Collier (p. 30) even concluded that the innate basis for such "affective synesthesias" in nonsynesthetes, seemingly present in 2 year olds, "would help explain the universal appeal of the arts." This conclusion is supported by reports that crossmodal processing involving affective qualities and quantities are important to our aesthetic experience (Palmer et al., 2013a) as well as art production and appreciation (Van Meel, 1994; Rogowska, 2011).

Of additional interest are studies on the body conveying affect in art and other abstract forms of expression. For instance, emotion recognition from stylistic dance (e.g., Dittrich et al., 1996) parallels emotion recognition from non-stylized expressive movement (e.g., Atkinson et al., 2007). Aronoff et al. (1992) observed that emotional attributions to both ballet movements and simple geometric forms are associated with similar shape features (angular and diagonal vs. round) (see Lundqvist et al., 1999 on similar observations for faces). Spatiotemporal cues of arm and whole body movement (size, fluency, joint angle) convey expressed or perceived emotion (Pollick et al., 2001; Atkinson et al., 2007; Visch and Tan, 2009; Dael et al., 2013). Visch and Tan (2009) found that dynamic movement pattern in abstract animated objects influenced emotional responses; slow and direct movement generated sadness, for example, whereas high velocity movements evoked fear and surprise. In the words of Larson et al. (2012; p. 410) "emotion can be effectively and implicitly communicated based on primitive geometric forms that are embedded in many common affective cues."

The current section showed that crossmodal correspondences are not specific to synesthesia, but that crossmodal processing is common in the general population. For our current purpose, we here focused on crossmodal processing involving affective experiences. If synesthesia and crossmodal processing are based on a partially overlapping aetiology, more synesthesias should be observable linking affect and sensory (including bodily) experiences. As indicated above, such published observations are relatively rare (Cytowic, 1989; Ward, 2004; Callejas et al., 2007; Ramachandran and Brang, 2008; Hochel et al., 2009; Sinke et al., 2012) probably because understudied. For future studies, we would suggest accounting for "emotionality" (discrete emotions, valence) but to also move beyond such more traditional assessments of emotionality by considering a variety of neurophysiological and spatiotemporal body measures as well as cognitive appraisal driving affective experiences.

# **THE STABILITY AND DURABILITY OF SYNESTHETIC EXPERIENCES QUESTIONED**

We have seen that studies on affect in synesthesia dealt with discrete emotions (so far one report to the authors' knowledge: Ramachandran and Brang, 2008), affective valence (Ward, 2004), affective preferences (Callejas et al., 2007), and pain (Dudycha and Dudycha, 1935). Intuitively, the likely assumption would be that affective cross-links would be stable, durable and idiosyncratic (but see Ward, 2004), as commonly assumed for crossmodal experiences in developmental synesthesia. Yet, as will be shown, synesthetic experiences might disappear, new ones may develop in later life, and some might depend on the context, with possibly synesthetic experiences involving affect.

Changes to synesthetic experiences have been suggested since the beginning of the last century with a potentially higher prevalence of synesthesia in children than in adults (Alford, 1918; Riggs and Karwoski, 1934). Over a 12 months period, new synesthetic experiences were reported from children between 6 and 8 years old (Simner et al., 2009). As will be shown, new synesthetic experiences do also occur in later life. These experiences refer to new synesthetic couplings when encountering new stimuli. For instance, in the case of object personification synesthesia, stable personalities are experienced for familiar but also for novel objects, even after a single encounter (Smilek et al., 2007). When it comes to lexical material, four individuals out of 192 reported synesthetic experiences when exposed to language they do not understand (Rich et al., 2005). In addition, these same authors reported on a synesthete whose mother tongue was English and who had learned Modern Greek as an adult. This synesthete showed comparable synesthetic experiences according to the shape of the letters in the two different alphabets (as compared to the letter's similarity in pronunciation) (see also Witthoft and Winawer, 2006; Simner et al., 2011 for synesthetic experiences in different languages). A recent study showed that it took only a 10-min writing exercise for grapheme-color associations to be transferred to novel graphemes, here Glagolitic letters, characters from an ancient language (Mroczko et al., 2009).

Although synesthetic experiences are generally stable from early childhood, there is some evidence to suggest that new couplings can be formed with new stimuli over even short periods of time, or be mediated by context (encountering new persons, learning new skills). As noted before, the inducer is not the "physical" stimulus for the synesthetic experience itself, but how an object or experience is interpreted (e.g., Simner, 2007). Most of the time, an object (e.g., letter) is likely interpreted in the same way and by inference leads to the same concurrent. Yet, as outlined above, life situations might be appraised differently across time leading to potential changes to the synesthetic experience. If a certain "dynamism" applies to synesthesias involving affect alike, one might expect changes in the synesthetic experience with relatively lasting changes to affective measures (e.g., changed preferences and likings for persons, objects, sounds, activities). However, as will be considered in the next section, changes in the synesthetic experience involving affect might also be short lasting.

# **TEMPORARY CHANGES TO CROSSMODAL PROCESSING WHEN INVOLVING AFFECT**

The literature on crossmodal processing would hint at relatively short-lasting changes to synesthetic experiences involving affect. More specifically, we refer (i) to situations differing in their current affect such as the exposure to pleasant or unpleasant sensory stimulations (e.g., odor, images) and (ii) temporary changes to a person's mood (failing exams, break-up of a romantic relationship, death of a kin). While possible synesthetic and crossmodal links involving affect have been outlined above, we are dealing here with the possibility that crossmodal experiences change in quality or quantity as a function of how the individual affectively perceives the sensory experience at the time of testing or as a function of the affective state the individual is at the time of testing.

Studies assessed altered sensory perceptions after the manipulation of participants' affective states typically by comparing performance after positive/negative mood induction as compared to a neutral state induction using various materials. Siegel and Stefanucci (2011), for instance, reported that participants considered sounds to be significantly louder after a negative mood induction as compared to participants after a neutral mood induction. Noise-induced anger biased the perception of red for an ambiguously colored red-blue computer screen (Fetterman et al., 2012). Affective word evaluations primed brightness perceptions of gray squares congruent with the hypothesized metaphor (negative = dark; positive = light) (Meier et al., 2007)—a result that was replicated after listening to emotionevoking musical excerpts (Lindsen and Bhattacharya, 2012). Chen and Dalton (2005) found that anxious individuals and neurotic individuals detected pleasant and unpleasant odors faster than neutral odors. Moreover, in men, the intensity of the odors was perceived to be stronger when being in an emotional (irrespective of valence) as compared to a neutral state. In another study, olfactory sensitivity was negatively affected after having seen unpleasant pictures, although odors were subjectively rated to be more intense (Pollatos et al., 2007). Thus, affective states seem to modify sensory experiences temporarily.

Additional studies further strengthen the notion that modifications can occur for stimuli of varying complexity (e.g., primary visual analysis, size estimations, processing of social cues) and at early or later processing stages (e.g., Phelps et al., 2006; Bocanegra and Zeelenberg, 2009; Schmitz et al., 2009; Stefanucci et al., 2011; Rossi and Pourtois, 2013). When relatively "simple" target features were concerned, brief presentation of a fearful face as compared to a neutral one resulted in superior orientation sensitivity for images of low spatial frequency and attenuated orientation sensitivity for images of high spatial frequency (Bocanegra and Zeelenberg, 2009). Phelps et al. (2006) observed enhanced contrast sensitivity for Gabor patches (early visual processing) subsequent to fear induction. Also, visual field processing is narrowed when in a negative mood and widened when in a positive mood (Schmitz et al., 2009; see also Gable and Harmon-Jones, 2010). In an ERP study, Rossi and Pourtois (2013) found that effective filtering (perceptual encoding) of peripheral task-irrelevant distractors was abolished when individuals were in a negative mood. Another study showed that height is overestimated when looking at high vs. low arousing pictures and that this effect is enhanced when individuals up-regulated their emotional experience (Stefanucci and Storbeck, 2009). Moreover, when standing on a skateboard on top of a hill, participants who were scared as compared to those who were not judged the hill to be steeper (Stefanucci et al., 2008). Anxious participants who were in the dark judged auditory targets to be closer than non-anxious participants (Gagnon et al., 2013).

When considering "more complex" target features, Anderson et al. (2011) found that participants having experienced an unpleasant affect as compared to a neutral situation had a processing bias for emotional faces in a binocular rivalry paradigm. In lexical decision and word naming experiments, emotional state facilitated responses to words categorically related to the induced emotion (i.e., happy vs. sad words) (Niedenthal and Setterlund, 1997). Ferraro et al. (2003) showed that happy and sad mood inductions (listening to classical music) resulted in faster responding to happy and sad words, respectively. In another series of studies, a decreased holistic processing bias for faces has been reported subsequent to a negative as compared to positive and neutral mood induction (Curby et al., 2011; see also Lynn et al., 2012). Yet, questioning the role of valence, when having undergone an arousal induction procedure, participants yielded a global vs. local processing bias for geographical map information, irrespective of the valence of the arousal situation (Brunyé et al., 2009).

The current section showed that perception changes as a function of individuals' affective state, but also as a function of the affective properties of the material. If these findings apply to synesthesia, we could expect to observe slight changes to the synesthetic experiences in analogy to the observations described here. Changes are potentially subtle and may only become recognizable when of sufficient magnitude. The studies reviewed here would point to quantitative changes rather than qualitative changes: experiences were larger, closer, darker, steeper and faster. For synesthetes, one could thus expect short-lasting quantitative changes such as the synesthetic colors being lighter when in a positive state (Meier et al., 2007; Lindsen and Bhattacharya, 2012), or the months being spatially localized more closely when in an anxious state (Gagnon et al., 2013). These predictions on a mainly quantitative level would also fit with synesthetes' own experience that the synesthetic experience remains the same. If synesthetes would experience major qualitative changes, these would have certainly been reported (but obviously the possibility should not be excluded), while quantitative ones might mainly be too subtle to be overtly recognized as such by the individual (see also Deroy and Spence, 2013).

In this regard, we would like to finish this section with reference to a promising report on a synesthete published in 1935 by Dudycha and Dudycha. Among other synesthetic crossmodal correspondences, their participant reported auditory—shape synesthesias. When hearing the same person speaking in a rasping voice, the accompanying shapes would be larger and sharper. Also, the shapes would be larger with increasing harshness of the same voice. Here the synesthete was aware of these relationships. The authors write the "subject observes the photism which is caused by her own speaking voice, and states that whenever she observes that it is becoming rough or jagged she modulates her voice so as to make the photism smooth and rod-like." (p. 63).

# **CONCLUSIONS**

In the current article, we aimed to provide a rationale for the probable role of affect in developmental synesthesia. We reviewed the literature on synesthesia (in particular relevant to affective processing) on the one hand and introduced some general theoretical concepts and empirical findings from the emotion and crossmodal literature on the other hand. The motivation for this endeavor relates to the relatively minimal attention affect has received in the published literature on synesthesia. Indeed, the synesthetic experience might not be affective to the synesthete who is used to these experiences (Sinke et al., 2012). Yet, when briefly reviewing the literature on crossmodal processing in synesthesia, modular (e.g., on discrete emotions) and non-modular (e.g., cognitive appraisal) emotion theories, as well as crossmodal processing (including affect) in the general population, two major predictions emerged. Firstly, since cognitive appraisal determines the form and intensity of the affective experience, it should also to some extent determine potential idiosyncrasies of or changes to the synesthetic experience. In these cases, the synesthetic experience involving affect would be relatively longlasting if the affective appraisal is likely to remain stable (e.g., very unpleasant odors) or might change depending on changes in affective appraisal (e.g., from liking to disliking of particular persons). Secondly, synesthesias involving affect might be subject to temporary changes of relatively short duration according to the affective evaluation of a stimulus (e.g., a less pleasant odor being judged more intense) or the current affective state of the individual (e.g., an anxious person perceives the hill as more steep). Here, changes to the synesthetic experience are proposed to be mainly quantitative rather than qualitative.

We suggest that such future studies might also benefit from the consideration of appraisal accounts of emotion and affect (see for example Fontaine et al., 2007) as well as psychological theories suggesting associative mechanisms by which prior affective (valenced) experiences with objects or concepts can explain current preferences (Palmer and Schloss, 2010).

# **AUTHOR CONTRIBUTIONS**

Development of basic article idea and structure Nele Dael, Christine Mohr. Literature search, analysis, and writing Nele Dael, Guillaume Sierro, Christine Mohr

# **ACKNOWLEDGMENTS**

We wish to thank Akzonobel, Imperial Chemical Industries (ICI) Limited, in particular Dr. David Elliott, Leader Color R&I team, Slough, UK, for supporting our empirical work on color and emotion links. Without these

# **REFERENCES**


*Individ. Dif.* 54, 828–831. doi: 10.1016/j.paid.2012.12.018


recent activities, we would have had fewer opportunities merging different research expertise, which also fostered ideas presented in the current manuscript. We are grateful to Christina Györkös for English proofreading of the manuscript.

emotions elicited by synesthetic grapheme-color perceptions. *Brain Res.* 1127, 99–107. doi: 10.1016/j.brainres.2006.10.013


*J. Cross. Cult. Psychol.* 28, 156–171. doi: 10.1177/0022022197282002

James, W. (1890). *The Principles of Psychology*. New York, NY: Dover. doi: 10.1037/11059-000

Jewanski, J., Day, S. A., and Ward, J. (2009). A colorful albino: the first documented case of synaesthesia, by Georg Tobias Ludwig Sachs in 1812. *J. Hist. Neurosci.* 18, 293–303. doi: 10.1080/09647040802431946


judgments. *Emotion* 7, 366–376. doi: 10.1037/1528-3542.7.2.366


synaesthesia in childhood populations. *Brain* 132, 57–64. doi: 10.1093/brain/awn292


influences perception. *Soc. Pers. Psychol. Compass* 5, 296–308. doi: 10.1111/j.1751-9004.2011.00352.x


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 06 July 2013; accepted: 27 September 2013; published online: 18 October 2013.*

*Citation: Dael N, Sierro G and Mohr C (2013) Affect-related synesthesias: a prospective view on their existence, expression and underlying mechanisms. Front. Psychol. 4:754. doi: 10.3389/fpsyg. 2013.00754*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Dael, Sierro and Mohr. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Do you see what I hear? Vantage point preference and visual dominance in a time-space synaesthete

### *Michelle Jarick1 \*, Mark T. Stewart 2, Daniel Smilek3 and Michael J. Dixon3*

*<sup>1</sup> Neurocognition of Attention and Perception Lab, Department of Psychology, MacEwan University, Edmonton, AB, Canada*

*<sup>2</sup> Department of Psychology, Willamette University, Salem, OR, USA*

*<sup>3</sup> Department of Psychology, University of Waterloo, Waterloo, ON, Canada*

### *Edited by:*

*Clare Jonas, University of East London, UK*

### *Reviewed by:*

*Harold H. Greene, University of Detroit Mercy, USA David Brang, Northwestern University, USA*

### *\*Correspondence:*

*Michelle Jarick, Department of Psychology, MacEwan University, 10700 - 104 Avenue, Edmonton, AB T5J 4S2, Canada e-mail: jarickm@macewan.ca*

Time-space synaesthetes "see" time units organized in a spatial form. While the structure might be invariant for most synaesthetes, the perspective by which some view their calendar is somewhat flexible. One well-studied synaesthete L adopts different viewpoints for months seen vs. heard. Interestingly, L claims to prefer her auditory perspective, even though the month names are represented visually upside down. To verify this, we used a spatial-cueing task that included audiovisual month cues. These cues were either congruent with L's preferred "auditory" viewpoint (auditory-only and auditory + month inverted) or incongruent (upright visual-only and auditory + month upright). Our prediction was that L would show enhanced cueing effects (larger response time difference between valid and invalid targets) following the audiovisual congruent cues since both elicit the "preferred" auditory perspective. Also, when faced with conflicting cues, we predicted L would choose the preferred auditory perspective over the visual perspective. As we expected, L did show enhanced cueing effects following the audiovisual congruent cues that corresponded with her preferred auditory perspective, but that the visual perspective dominated when L was faced with both viewpoints simultaneously. The results are discussed with relation to the reification hypothesis of sequence space synaesthesia (Eagleman, 2009).

**Keywords: synaesthesia, spatial-cueing, spatial perception, attention, mental vantage points, orienting, reification**

# **INTRODUCTION**

Synaesthesia is a fascinating phenomenon whereby ordinary sensory information perceived in one modality elicits a second extraordinary sensory experience in the same or different modality. The types of associations most studied include experiencing colors for letters and numbers (e.g., Dixon et al., 2000), specific tastes for words (e.g., Simner and Ward, 2006), and colors evoked by different musical tones (e.g., Cytowic and Eagleman, 2009). Here we examined an individual (L) who *sees time in space*, meaning that years, months, days, and hours elicit highly specific spatial locations that surround her. This type of association is encased under the larger umbrella term of sequence-space synaesthesia (SSS; Eagleman, 2009).

An intriguing theory of SSS is that the spatial forms are the objectification of overlearned sequences (the *reification hypothesis*; Eagleman, 2009). In other words, the spatial forms that synaesthetes report may be closer to the experience of real "objects" than mere figments of visualization/imagination. In fact, synaesthetic reports of sequences having fixed, objectcentered coordinates speaks to that very possibility (Smilek et al., 2007; Eagleman, 2009). As L once described, "When I hear the month January it sits to my right, but then if I see the word January it is all of a sudden now to my left. Even within my spatial maps, I can choose to take on a variety of perspectives by zooming in and out voluntarily." Being able to move about in one's spatial representation suggests that the spatial forms are not necessarily tied to the synaesthete's body (ego-centric), but rather encompass a coordinate system similar to that of objects (object-centered; Eagleman, 2009). If the ability to change perspectives is a consistent quality of SSS, then it must be incorporated in current theories. To date this characteristic has been described as a mere curiosity (Eagleman, 2009).

Numerous subjective reports from sequence-space synaesthetes have suggested that their fixed spatial representations can be viewpoint variant (Galton, 1880; Seron et al., 1992; Cytowic and Eagleman, 2009). One of the challenges to objectively verifying these subjective claims has been in designing a reliable experimental paradigm to measure the changes in viewpoint. The majority of month-space synaesthetes that we have interviewed report altering their vantage point as the months progress over time or take on the viewpoint of the month cued. If cued with the month April, for example, the vantage point could change for some synaesthetes and appear as if they were all of a sudden standing in front of April. These constant and fleeting changes in viewpoint make it very difficult, not only to document consistency of their month-space, but to capture a reliable change in perspective. For instance synaesthete H reported, "As I moved through the year, I am very aware of my place in the oval at the current time, and the direction I am moving in" (Mann et al., 2009).

Synaesthete L, on the other hand, has two perspectives from which she views her month-space, triggered involuntarily by whether she *hears* or *sees* the month in question (Jarick et al., 2009a, 2011a). In other words, L's perspectives are not dependent on the current month, cued month, or current season. We can reliably trigger changes in L's vantage points by modifying the sensory presentation of the cued stimulus. Empirical confirmation of L's ability to change perspectives has already been established in a previous study (Jarick et al., 2009a), where we have demonstrated opposite response patterns to months and hours depending on the modality of the inducing cue used in a spatial cueing paradigm. For instance, if the word January was presented as a central cue, L responded faster to detect targets on the left because that it was congruent with where January was spatially located (according to her visual spatial map). However, if L was then presented with a voice saying January out loud, she was faster to detect a target on the (opposite) right side because it was now congruent with where January was spatially located (according to her auditory spatial map). This three-way interaction between cue modality (auditory vs. visual), cue type (early vs. late), and target location (left vs. right) was not found for the non-synaesthetes tested. Thus, L's subjective report of being able to change perspectives depending on the modality of the cue was objectively confirmed. In hindsight, these results provided one leg of support for the reification hypothesis in the sense that the month-space is not necessarily tied to the synaesthete (egocentric), but can exists with respect to object-centered coordinates (taking on object-like properties).

As we discovered a year or so later, the uniqueness of L's synaesthesia was not limited to her ability to view her time-space from opposite vantage points. L also claimed to prefer to view her space from her auditory viewpoint. As shown in **Figure 1**, viewing written words from this vantage point causes the words to appear upside-down—a viewpoint for which she would seldom see in real life. By preference, we mean that L views her time-space from the auditory perspective most often and feels most comfortable doing so. Just as one might have a default point of view when we imagine a car (a 3D object that conforms to objectbased coordinates), L reports to have a "default" viewpoint of her month-space. As if the month *names* and month spaces encompass objects-based coordinates themselves. According to the reification hypothesis, this would be the case if synaesthetes were treating the months as "objects" and not just a location in space.

In order to test the idea that spatial synaesthetes could have a preferred or "default" mental vantage point for viewing spatialforms, we focused on the synaesthete L's representation of her months. L has participated in previous studies over the years aimed at verifying her time-spaces (months and hours; Jarick et al., 2009a, 2011a), as well as her number-space (Jarick et al., 2009b, 2011b). As such, the consistency of L's spatial-maps have all been well-established and we have empirically demonstrated that L views her month-space from different perspectives depending on whether the month is *seen* vs. *heard* (Jarick et al., 2009a). In fact, L takes the exact opposite vantage point, making designing an experiment fairly simple. For instance, when L sees a month name written, she views her space organized such that January, February, and March are on her left side and May, June, and July on her right (the rest of the months flow behind her, as seen in **Figure 1**). When she hears the name of a month spoken, however, she takes the opposite viewpoint and January, February, and March are now on her right side and May, June, and July on her left (the other months extend outward into space). Consistent

with the objectification of overlearned sequences (Eagleman, 2009), L reports that she views her month-space from the auditory perspective it is as if she were looking at her months (and month names) literally upside down. What is even more striking about this unusual perspective, is that L claims to *prefer* it that way.

# **EXPERIMENT 1**

To first verify L's claim that her auditory perspective was akin to viewing her months upside down, we tested whether we could elicit her "auditory" vantage point with the *visual* month names alone (i.e., month names written upside down). We used a spatialcueing task (Posner, 1981) containing two cueing conditions: month names upright or month names inverted. Our predictions were straightforward: we expected that the upright month cues would trigger L's visual perspective and as such, early month cues (*Jan., Feb., March*) should orient her attention to the left side of space (faster to detect left targets), while later month cues (*May, June, July*) should orient her attention to the right (faster to detect right targets). Our key prediction to verify L's claim of viewing months upside down, however, was that the *inverted* month names should trigger the reverse "auditory" perspective. Thus, inverted early months (*Jan., Feb., March*) should orient her attention to the right side of space (faster to detect right targets), while inverted later months (*May, June, July*) should orient her attention to the left (faster to detect left targets), essentially reversing the cueing pattern seen for the upright month cues. If our predictions are confirmed, we should find the same threeway interaction that we previously found (Jarick et al., 2009a,b) between cue condition (upright vs. inverted), cue type (early vs. late), and target location (left vs. right) for L that is absent in non-synaesthetic controls.

# **METHODS**

# *Participants*

We tested a 23-year-old time-space synaesthete L and five nonsynaesthetic controls from Willamette University, all of which received for a honorarium for their time. All participants had normal or corrected-to-normal vision and hearing and were righthanded. Participants gave informed consent before participating and the board of research ethics at Willamette University and the University of Waterloo approved the experimental procedures.

# *Materials*

All stimuli were presented on a white background. Visual month cues were black text (Geneva font, 72 pt). The centrally located cues consisted of 6 month names (height 0.6◦ visual angle and maximally 6.5◦ in length)—January, February, March (*early months*), May June, and July (*late months*). Targets were black squares (each side subtending 0.6◦ of visual angle) and placed at an eccentricity of 10.5◦ from the center of fixation. Stimuli presentation and data collection were controlled by SuperLab 4.0 experimental software.

# *Procedure*

Participants were seated ∼57 cm in front of a computer screen and asked to perform a spatial cueing task. A typical trial involved a fixation cross for 600 ms, a central cue (month name) appearing either upright or inverted for 600 ms, followed immediately by a target square to either the left or right of the month cue. The target remained on the screen until the participant responded or 3500 ms elapsed. The task for L and a group of five nonsynaesthetic controls was to detect this target as quickly as possible by pressing a central button on a response pad with their dominant (right) hand. Once a response was given, the next trial began. On about 10% of the trials no target would appear ("catch trials"), for which participants were advised to withhold their response until the next trial. These trials were inserted to make certain participants were staying on task. There were four blocks of 132 trials (60 valid, 60 invalid, 12 catch), for a total of 528 trials. Inverted and upright month cues were randomly intermixed within the blocks. Participants had self-paced breaks between blocks.

It is important to note that although the month names served as spatial cues for L, she was aware that they were not predictive of the target location—targets appeared to both the right and left 50% of the time (i.e., half valid, half invalid). Thus, there was no incentive to even pay attention to the month names while detecting the targets presented.

# **RESULTS AND DISCUSSION**

The synaesthete L and non-synaesthetic controls performed perfectly on catch trials (100% accurate). The response times for L and the controls were submitted to a recursive outlier procedure, for which observations greater or less than 2.5 standard deviations were discarded. As a result, only 0.65% of data was removed from L, and an average of 1% from controls.

An analysis of variance (ANOVA) with L's response times showed a significant cue condition (upright vs. inverted) × cue type (early vs. late) × target location (left vs. right) interaction, *F(*1*,* <sup>35</sup>*)* = 1178*.*68, *p <* 0*.*0001, as depicted in **Figure 2**. That is, when the cue was an upright month name, it elicited L's visual vantage point. As such, she detected targets on the left side faster following early months and detected targets on the right side faster following late months. However, the opposite response pattern resulted when the cues were inverted month names, as they elicited L's "auditory" viewpoint. Thus, following inverted cues, L detected targets on the right faster following early months and faster to detected targets on the left following late months. This three-way interaction suggests that when cued by an early month (for example), L responded significantly faster to targets on the left when the cue was *upright*, but detected targets on the right when the cue was *inverted*. As predicted, these results demonstrate a clear reversal in viewpoint triggered by the two types of visual cues, with the upright cues eliciting the visual vantage point and the inverted cues eliciting the "auditory" vantage point. As expected, not one of the five non-synaesthetic controls showed this three-way interaction (all *p*'s *>* 0.303). Thus, our findings show that upright and inverted month names cued opposing viewpoints for L, but not for the non-synaesthetic controls. In essence, our data objectively verified L's subjective claim that her "auditory" mental vantage point causes her to view her months upside down. This finding suggests that L's vantage point is not solely dictated by the modality of the inducing stimulus

(i.e., a month spoken aloud) like previously believed (Jarick et al., 2009a,b), but rather can be induced by visual stimuli alone as long as the orientation of the stimulus is consistent with L's vantage point of her month-space.

# **EXPERIMENT 2**

Next we tested L's second claim that she *prefers* to view her months from the auditory perspective (i.e., visually upside down). To reiterate, by preference we mean a "default" point of view for which L might show a bias in her behavior toward. To test this, we attempted to create a situation where her visual and auditory viewpoints would be in conflict in order to observe which one she is more biased toward using. If her auditory perspective is indeed her "default" viewpoint, then that perspective should be the viewpoint she takes. Using the same spatial-cueing task, we included four types of cue conditions: auditory-only month names, visual-only month names (upright), audiovisual congruent month names (visual inverted + auditory), and audiovisual *incongruent* month names (visual upright + auditory). Our predictions were as follows: audiovisual congruent cues should not only trigger L's "auditory" perspective, but should facilitate detection of valid targets (due to the multisensory enhancement of having bimodal cues triggering the same perspective). Therefore, the audiovisual congruent cues should produce the largest cueing effects (i.e., difference in response time between valid and invalid targets) compared to auditory-only cues. Our key prediction, however, was regarding the audiovisual incongruent condition. We hypothesized that when the month cues trigger both perspectives simultaneously (i.e., auditory cue triggers "auditory" perspective and visual upright cue triggers visual perspective), the preferred "auditory" perspective should be the one elicited. That is, if L prefers her auditory viewpoint, then she should show a bias in her response pattern consistent with the "auditory" perspective (i.e., similar to the auditory-only condition).

# **METHODS**

# *Participants*

The time-space synaesthete L participated for a honorarium. We did not include non-synaesthetic control participants, as this examination was purely a test of L's ability.

# *Materials and procedure*

Stimuli were the same as Experiment 1, except the two audiovisual (AV) conditions were added. The AV congruent condition elicited the same vantage point for L and consisted of the visual month name inverted paired simultaneously with the month spoken aloud from the computer loudspeakers (both cues triggered the "auditory" perspective). Both visual and auditory stimuli were similar in the degree of saliency to the participant, in that the month name appeared on the computer screen at a comfortable reading size (height of 0.6◦ visual angle and maximally 6.5◦ in length) and the voice was presented at a comfortable hearing level (∼65 dB). The AV incongruent condition should elicit conflicting vantage points for L and consisted of the visual month name upright paired simultaneously with the month spoken aloud from the computer loudspeakers (visual cue triggered visual perspective while auditory cue triggered "auditory" perspective). Stimuli were presented in two blocks, each containing 30 valid and 30 invalid of each type of cue (visual-only, auditoryonly, AV congruent, AV incongruent) plus 12 catch trials, all randomly intermixed. Participants L had a 5-min break between blocks. There were 528 trials in total and the experimental session lasted about 20 min.

# **RESULTS AND DISCUSSION**

L performed perfectly on catch trials (100% accurate). Only 0.01% of L's data was removed by the outlier procedure (±2.5 standard deviations).

To evaluate which viewpoint each cue condition would elicit (i.e., whether L's response pattern would be biased toward her "auditory" or visual perspective), we conducted a 4 × 2 × 2 repeated measured ANOVA with the factors cue condition (auditory-only, visual-only, AV incongruent, and AV congruent), cue type (early and late), and target location (left and right). The ANOVA revealed a significant main effect of cue condition, *F(*2*.*28*,*72*)* = 30*.*97, *p <* 0*.*001, and a significant two-way interaction between cue type and target, *F(*1*,* <sup>24</sup>*)* = 6*.*27, *p <* 0*.*05. The key finding was the significant three-way interaction between cue condition, cue type, and target location, *F(*3*,* <sup>72</sup>*)* = 46*.*36, *p <* 0*.*001, which suggested that the different cue conditions were eliciting different vantage points for L. The results can be seen in **Figure 3**.

To investigate this three-way interaction further, we conducted a 2 (cue condition) × 2 (cue type) × 2 (target location) ANOVA for each unimodal (auditory-only vs. visual-only) and bimodal (AV congruent vs. AV incongruent) conditions separately. The unimodal conditions have been tested previously (Jarick et al.,

2009a), for which a significant three-way interaction was found that depicted the vantage point reversal between visual and auditory cues. In other words, the auditory cues elicited the "auditory" viewpoint and the visual cues elicited the opposing visual viewpoint. Based on our previous findings, we predict the same three-way interaction for the unimodal conditions here. Not surprisingly, our prediction was confirmed and the ANOVA revealed a significant three-way interaction between cue condition, cue type, and target location, *F(*1*,* <sup>27</sup>*)* = 34*.*93, *p <* 0*.*001, again replicating the opposing viewpoints triggered by the visual-only and auditory-only cues. The ANOVA also showed significant main effects of cue condition, *F(*1*,* <sup>27</sup>*)* = 19*.*93, *p <* 0*.*001, and cue type, *F(*1*,* <sup>27</sup>*)* = 4*.*71, *p <* 0*.*05, as well as a significant two-way interaction between cue condition and target, *F(*1*,* <sup>27</sup>*)* = 6*.*24, *p <* 0*.*05. No other comparisons were significant. Therefore, consistent with previous findings (Jarick et al., 2009a,b), L responded in a manner consistent with her visual perspective when cued by the visualonly upright months, as well as the "auditory" perspective when cued by the auditory-only months.

The more interesting finding is whether L showed a bias toward her preferred "auditory" viewpoint. This was tested in the AV incongruent condition, where the cues triggered conflicting vantage points. Thus, our predictions for the bimodal conditions really depend on which vantage points L is biased toward in the AV incongruent cue condition. We predicted that the AV congruent cues (auditory + month inverted) would undoubtedly trigger L's "auditory" vantage point, since we have already shown that each does individually from the unimodal analysis. The AV incongruent cues (auditory + month upright), on the other hand, were expected to trigger both visual and "auditory" perspectives, thereby putting L in conflict. Due to L's declared preference for her auditory perspective, we predicted that the "auditory" viewpoint would be more strongly elicited. If this is the case, we should not find a significant three-way interaction due to both cue conditions showing similar cueing patterns, but rather a two-way interaction between cue type and target location. Contrary to our predictions, the ANOVA did reveal a significant three-way interaction between cue condition, cue type, and target location, *F(*1*,* <sup>26</sup>*)* = 155*.*27, *p <* 0*.*001, which indicated that the AV incongruent cues elicited L's visual vantage point. Visual inspection of **Figure 3** illustrates the perspective reversal (i.e., three-way interaction), such that the AV incongruent cues (i.e., when the cues were in conflict) elicited the same mental vantage point as the visual-only cues, while the AV congruent cues (when the cues were complementary) elicited the same mental vantage point as the auditory-only cues.

At first glance, this latter finding seems to falsify L's claim of preferring the "auditory" viewpoint, since when pitted against one another, the visual viewpoint was the one that biased L's attention. However, there are other possibilities to consider. Perhaps when the cues triggered opposite vantage points, L needed to reconcile the conflict with a strategy. Intuitively speaking, the simplest strategy in this case would be to actively ignore one of the cues and in this situation (i.e., being visual detection task) the auditory cue is easiest to suppress. In other words, it would be more difficult to suppress visual information and simultaneously perform a visual task. It would therefore be easiest for L to suppress the auditory information and focus on the visual, which would result in response times consistent with her visual perspective. Another possibility is that even though L has a claimed preference for the "auditory" perspective this does not mean that the auditory perspective will be dominant. It is wellknown that vision is the dominant sense, and it would not be too far fetched to believe that the visual cues would be dominant in determining L's vantage points.

There is, however, another avenue to evaluate L's claim of her preferred "auditory" vantage point—to analyze the *magnitude* of the cueing effects (i.e., response time difference between valid and invalid targets) within the cue conditions. To get a measure of magnitude, we calculated response times to detect valid targets (target in cued location) and invalid targets (target in uncued location) for each of the cue conditions. We performed a repeated measures two-way ANOVA with the factors cue condition (visualonly, auditory-only, AV congruent, and AV incongruent) and validity (valid vs. invalidly cued targets). Notably, the "valid" trials in the AV incongruent condition were with reference to the vantage point L took that being the visual perspective. We predicted that due to the multisensory nature of the AV conditions, both would show a multisensory benefit with larger cueing effects compared to the visual-only or auditory-only conditions. In terms of L's preference for her auditory viewpoint, the AV congruent cues

times to detect valid targets (target appeared at cued location) from response times to detect invalid targets (target appears in uncued location). Error bars represent 95% confidence intervals.

should show a significantly stronger cueing effect compared to the other cue conditions. The ANOVA showed a significant main effect of cue condition, *F(*3*,* <sup>177</sup>*)* = 38*.*97, *p <* 0*.*001, and a main effect of validity, *F(*1*,* <sup>59</sup>*)* = 104*.*28, *p <* 0*.*001. Cue condition and validity was also shown to interact, *F(*3*,* <sup>177</sup>*)* = 2*.*58, *p* = 0*.*05. To interpret this interaction we calculated cueing magnitude scores (invalid response times minus valid response times). These cueing magnitude scores are shown in **Figure 4**. *Post-hoc* (LSD) comparisons indicated that cueing magnitudes were greatest in the AV congruent condition than the visual-only (*p <* 0*.*05), auditoryonly (*p <* 0*.*05), and AV incongruent condition (*p <* 0*.*05). The visual-only, auditory-only and AV incongruent conditions did not differ at all (all *F*'s *<* 1 and *p*'s *>* 0.05). Although the audiovisual conditions both showed faster overall response times compared to the unimodal conditions, only the AV congruent cues differed in the magnitude of their cueing effects. Specifically, following the AV congruent cues (both elicited auditory vantage point), L showed the greatest cost when the target was invalid, but also the greatest benefit when the target was valid. This finding is consistent with L's claim of preferring the viewpoint elicited by the AV congruent cues (i.e., "auditory viewpoint").

# **CONCLUSIONS**

Together our data show that L's performance in the spatialcueing task was clearly influenced by her extraordinary monthspace associations in which the vantage point from where she views her month-space has a profound influence on her spatial attention. In the first experiment we were able to capture L's auditory mental vantage point MVP using *visual* stimuli alone and in the second experiment show some objective support for her claim that she prefers to view her space from the "auditory" perspective. Thus, far, L is the only synaesthete who has shown these unique traits. Other accounts of time-space synaesthesia have reported being able to "zoom in and out" of their spatial forms or report feeling like they are walking along with the months as they pass through the year (Galton, 1880; Seron et al., 1992; Cytowic and Eagleman, 2009), but none who report reversing their perspective with reference to an inducing stimulus.

The finding that L can reverse her perspective on "time" is not new (Jarick et al., 2009a,b), however our finding that she vividly experienced the month names upside down when viewing her month-space from the "auditory" perspective is very informative. It perhaps suggests that her month-space does not always adhere to egocentric coordinates and could exhibit object-like properties. The fact that we could induce both of her vantage points using visual stimuli alone speaks to how visually detailed L's month-space is for her. Just as we can be influenced to view objects from different perspectives, L too can view her spatial calendar from different mental vantage points. Her report of having a preferred viewpoint is also consistent to how we might imagine objects from a canonical perspective. Keeping in mind of course that our mental viewpoints are initially the product of viewing real objects in the external world, while L's viewpoints are completely internally generated. This evidence can provide some behavioral support for Eagleman's *reification hypothesis*. Of course this is a first step and only one synaesthete; more data is needed to generalize these findings to other synaesthetes.

However, if it is the case that these implicit sequences trigger an experience of objecthood for synaesthetes, theories regarding the development and neural architecture of synaesthesia need to take that characteristic into account. For instance, previous studies (Hubbard et al., 2005; Tang et al., 2008) have primarily focused on the parietal lobes to uncover the neural correlates of SSS, being that the seeing as though the parietal areas process visuo-spatial information including well-learned sequences. However, as Eagleman (2009) suggests, perhaps we should also be looking at activity in the temporal lobes where properties of objects are represented. In fact, Pariyadath et al. (2008) have recently shown brain activation in temporal lobe areas during the processing of overlearned sequences in nonsynaesthetes. Temporal lobe activation for SSS has yet to be confirmed, however it would provide some clues as to why many sequence-space synaesthetes also see colors for the months of the year and days of the week (Sagiv et al., 2006; Smilek et al., 2007).

What is further unique about L is that she possesses a strong preferred viewpoint that is unconventional to what most people would prefer, what most people would be "taught." She prefers to view her space upsidedown, where the average person would intuitively feel more comfortable viewing months written upright. In terms of the developmental debate of synaesthesia, some argue that it is simply the product of learning. However, if that were the case, it is unclear what might have motivated L to learn a mental calendar that would be upsidedown.

# **ACKNOWLEDGMENTS**

This research was supported by a Natural Sciences and Engineering Research Council of Canada research grant to Michael J. Dixon and postgraduate scholarship to Michelle Jarick We sincerely thank E. M. for her commitment to this project.

# **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 10 June 2013; accepted: 12 September 2013; published online: 16 October 2013.*

*Citation: Jarick M, Stewart MT, Smilek D and Dixon MJ (2013) Do you see what I hear? Vantage point preference and visual dominance in a time-space synaesthete. Front. Psychol. 4:695. doi: 10.3389/fpsyg.2013.00695*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Jarick, Stewart, Smilek and Dixon. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Color and texture associations in voice-induced synesthesia

# *Anja Moos 1,2\*, David Simmons 2, Julia Simner <sup>3</sup> and Rachel Smith1*

*<sup>1</sup> School of Critical Studies, Laboratory of Phonetics, College of Arts, University of Glasgow, Glasgow, UK*

*<sup>2</sup> College of Science and Engineering, School of Psychology, University of Glasgow, Glasgow, UK*

*<sup>3</sup> Department of Psychology, School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, UK*

### *Edited by:*

*Michael Banissy, Goldsmiths University of London, UK*

### *Reviewed by:*

*Sarah Creel, University of California at San Diego, USA Carolyn McGettigan, Royal Holloway University of London, UK*

### *\*Correspondence:*

*Anja Moos, School of Critical Studies, Laboratory of Phonetics, College of Arts, University of Glasgow, UK e-mail: anja.moos@gmail.com*

Voice-induced synesthesia, a form of synesthesia in which synesthetic perceptions are induced by the sounds of people's voices, appears to be relatively rare and has not been systematically studied. In this study we investigated the synesthetic color and visual texture perceptions experienced in response to different types of "voice quality" (e.g., nasal, whisper, falsetto). Experiences of three different groups—selfreported voice synesthetes, phoneticians, and controls—were compared using both qualitative and quantitative analysis in a study conducted online. Whilst, in the qualitative analysis, synesthetes used more color and texture terms to describe voices than either phoneticians or controls, only weak differences, and many similarities, between groups were found in the quantitative analysis. Notable consistent results between groups were the matching of higher speech fundamental frequencies with lighter and redder colors, the matching of "whispery" voices with smoke-like textures, and the matching of "harsh" and "creaky" voices with textures resembling dry cracked soil. These data are discussed in the light of current thinking about definitions and categorizations of synesthesia, especially in cases where individuals apparently have a range of different synesthetic inducers.

**Keywords: voice-induced synesthesia, color, texture, cross-modal correspondence, speech acoustics**

# **INTRODUCTION**

Voice-induced synesthesia is a relatively rare type of synesthesia. According to a database compiled by Simner and Ward using an extensive questionnaire (https://www*.*survey*.*bris*.*ac*.*uk/ sussex/syn), less than 10% of the synesthetes filling out the form have voice-induced synesthesia. In this variant, people experience synesthetic perceptions induced by the sound of people's voices. Aside from one recent case study (Fernay et al., 2012), there has been no systematic research into this form of synesthesia, and no group study has been reported. According to personal reports by synesthetes from our participant pool, the most common synesthetic perceptions (so-called "concurrents") that accompany the sound of voices are colors, textures, shapes and movements/spatial arrangements. Informal reports from approximately 15 voice synesthetes, which have been gathered from both personal communication and via an international synesthesia email forum (http://www*.*daysyn*.*com/Synesthesia-List*.*html) illustrate the condition's multiple facets and complexity. For example, some voice synesthetes "see" the voice better when the person is singing. For some synesthetes colors vary little between voices but for others, colors depend strongly on the individual speaking. Concurrents may also be influenced by familiarity with the voice or the medium it is transmitted through, such as direct personal communication vs. radio. Some voice synesthetes identify the pitch to be a strong influence whereas others cannot define any criteria of the voice that change their concurrents. In Fernay et al. (2012), the synesthete's perceptions included color, size, and location of the associations. The authors found that a higher pitch, or fundamental frequency (f0), resulted in lighter color associations and a higher position in vertical space. Male voices induced larger shapes than female voices.

Coming to grips with voice-induced synesthesia requires a critical analysis of the concept of "a voice." The voice of a speaker can be distinguished from the linguistic information that the voice carries when talking (i.e., vowels, consonants, words, and intonation patterns), even though these types of information are always intertwined in the acoustic speech signal. The voices of individual speakers differ for a number of reasons, including the anatomy and physiology of the speech organs, and aspects of learned behavior reflecting group affiliation (Esling, 1978; Stuart-Smith, 1999) as well as individual idiosyncrasy and habit. Variation among voices is complex yet principled, and the approach taken in this paper is to find out what light can be cast on voice synesthesia by a systematic phonetic analysis of voices.

The phonetic concept of *voice quality* (Abercrombie, 1967; Laver, 1980) is a specialist term describing characteristics "which are present more or less all the time a person is talking: [voice quality] is a quasi-permanent quality running through all the sound that issues from his mouth" (Abercrombie, 1967, 91). The most widely-adopted framework for the analysis of voice quality is the auditory componential analysis pioneered by Laver (1980) and further developed by Nolan (1983) and Beck (2005) which analyses the voice in terms of long-term settings of the various speech organs, most importantly the larynx, the vocal folds, the soft palate, tongue, lips and jaw. *Laryngeal* settings relate to the position of the larynx in the neck, and to the mode of vibration of the vocal folds: a number of modes are distinguishable, including not only full periodic (modal) vibration of the folds, but falsetto voice (with taut vocal folds), creaky voice (with slow, irregular vibration of the folds), breathy voice (with high airflow), whispery voice (with audible friction produced by incomplete closure of the vocal folds), and harsh voice (resulting from constriction of the ventricular folds). *Supralaryngeal* settings include the location of the center of gravity of the tongue body, the degree of raising or lowering of the soft palate (which affects the degree of air escape through the nose, giving rise to the contrast between a "stuffedup," denasal voice, and a nasal twang), and the positions of the lips (spread, protruded) and jaw (raised, lowered, protruded).

In addition to voice quality, speakers are characterized by the pitch of their voices, or more technically by f0 (the fundamental frequency, determined by the rate of vocal fold vibration). The limits on the range of f0 that a speaker can produce are determined anatomically and physiologically (larger vocal folds produce lower frequencies) but every speaker is able to produce extensive variation in f0, the so-called pitch range. The size of the supralaryngeal vocal tract also affects its natural resonances, known as *formants*, with larger vocal tracts having lower resonant frequencies. Within the range determined for each speaker by their anatomy and their supralaryngeal settings, formant frequencies vary constantly during talking as the configuration of the vocal organs is changed to produce a sequence of vowels and consonants.

So how might these voice qualities be brought to bear when synesthetes experience colored voices? Due to the lack of literature on voice-induced synesthesia, related studies on synesthesias induced by speech sounds or the timbre of instruments might serve as a guideline for our research questions. In music-color synesthesia, Ward et al. (2006) found that higher notes triggered the experience of lighter colors, and lower notes triggered darker colors. Additionally, and in closer relation to voice quality, they found that timbre affected lightness choice: piano and string notes triggered more "colorful" experiences (i.e., higher chroma colors) than pure tones. Both these findings suggest that voice pitch or quality may influence the colors experienced by voice-triggered synesthetes. In linguistic synesthesias (e.g., triggered by spoken words), the acoustic and articulatory characteristics of vowels have also been shown to systematically influence color and luminance associations (Jakobson, 1962; Marks, 1974, 1975; Moos, 2013). Moos (2013) showed that acoustic measures (formant measures) could be used to analyse and explain inducer-concurrent relations.

In addition to color, voice-triggered synesthetes often report texture perceptions, for example a voice might be "smooth but granulated" or "[with a] soft center and very slight fuzziness around the outside." Despite our knowledge of perceptions such as these, no systematic investigation of visual texture perceptions in synesthesia has yet been conducted—which is perhaps not surprising considering that it is not easy to quantify texture or to relate this quantification to perceptual categories (Petrou et al., 2007; Clarke et al., 2011). Eagleman and Goodale (2009) state: "Quantitatively testing these prevalences [of texture concurrents] will be a challenge: it is straightforward to develop a user-friendly color chooser [...], but not so with the multidimensional varieties of texture" (Eagleman and Goodale, 2009, 291). In the current study we take on this challenge by analysing not only color associations, but also texture.

In this first group study on voice-induced synesthesia, we set out to answer three questions. First, we assess how voice-induced synesthesia expresses itself in individuals and across groups. Specifically, we ask about precise relationships between acoustic characteristics of the voice and its synesthetic concurrents, focusing on both color and texture. Second, we test the consistency of color and texture associations over time. The consistency with which synesthetic concurrents are described by synesthetes in different test sessions over time is often taken as the hallmark of synesthesia (e.g., Rich et al., 2005), although levels of consistency have since been shown to vary according to the particular type of synesthesia under investigation (e.g., Simner et al., 2011), and the particular methodology used (e.g., Simner and Ludwig, 2012). Finally, we ask how synesthetic colors for voice might differ from the normal cross-modal associations made by the general population. This will allow us to investigate common aspects of cross-modal perception as for example discussed in Spence (2011).

In our study, we tested voice synesthetes, professional phoneticians, and control participants, conducting the experiment online to facilitate participation for people with this rare type of synesthesia. Phoneticians were included to examine the potential influence of this profession on cross-modal associations with voices. Participants heard auditory samples which were controlled recordings of (non-participant) phoneticians producing different voice qualities (see section Voice Stimuli), and which were both perceptually and acoustically clearly distinct from each other. Participants cross-modally matched these to items from a closed set of colors and textures. Additionally, free verbal descriptions were elicited to gain a richer picture of participants' associations with the voices. After 2–8 months, a retest was conducted with a subset of the stimuli to test for consistency in participants' associations. The study was approved by the ethics board of the University of Glasgow, and participants provided informed consent before testing.

# **METHODS AND MATERIALS PARTICIPANTS**

There were three groups of participants: synesthetes, phoneticians, and controls. All participants were native speakers of English and had no severe sight or hearing difficulties. Participants were paid with Amazon vouchers.

We tested 14 voice-induced synesthetes (mean age = 34, age range = 18–70, *SD* = 19; 11 female), recruited from the Sussex-Edinburgh Database of Synesthete Participants, or via announcements on an online synesthesia forum (http://www*.* daysyn*.*com/Synesthesia-List*.*html). Synesthete participants were initially identified by self-report and in nine cases additionally by a synesthesia questionnaire designed by Simner and Ward (https://www*.*survey*.*bris*.*ac*.*uk/sussex/syn). Genuineness is also usually confirmed with consistency tests (consistently perceiving the same concurrents for the same inducers over time); this matter will be returned to in the discussion section. Thirteen synesthetes additionally self-reported color and/or texture perceptions induced by stimuli other than the sound of voices (e.g., digits, music). On average, they had 10 different inducers, ranging from 4 to 18. Five synesthetes were students; the rest came from a variety of professional backgrounds.

We also tested 10 phoneticians (mean age = 40, age range = 24–68, *SD* = 15*.*9; 7 female), recruited through an announcement on a phoneticians' email list and by individually contacting colleagues outside our universities by email. Three of them were PhD students, the rest were professionals. Phoneticians were identified as being non-synesthetes using a short questionnaire describing the phenomenon of synesthesia. In this, they were shown a list of 20 possible inducers (e.g., sounds, letters, words) and asked whether any triggered spontaneous colors, textures, or other sensations. They were classified as non-synesthetes when none of the inducers were selected. Finally, we additionally tested 28 control participants (mean age = 23, age range = 18–30, *SD* = 3*.*5; 17 female), recruited through the participant pool at the School of Psychology, University of Glasgow. Twentyfour participants were students and the rest were professionals. The same procedure as for the phoneticians identified them as non-synesthetes.

### **MATERIALS**

Our materials comprised a set of voice stimuli, and a response display showing a set of colors and a set of textures. (The response display also contained a set of semantic differentials which were included for another study presented elsewhere; Moos, 2013; Moos et al., in press). These are described in turn below.

### *Voice stimuli*

Materials were two short spoken passages taken from the story "The rainbow passage" (Fairbanks, 1960): "These take the shape of a long round arch, with its path high above, and its two ends apparently beyond the horizon" and "People look, but noone ever finds it." To avoid the influence of color terms on participants' perceptions, these sentences did not contain any color information. Our materials were recorded by two male phoneticians, who were able to deliberately vary their voice quality settings. We thereby avoided using many different speakers among whom voice quality would vary in less constrained ways. Ten voice qualities were chosen based on the criterion that they were perceptually maximally different. These were as follows:



due to incomplete closure of the vocal folds *CREAK* low pitched irregular voicing with slow, irregular vibration of the vocal folds.

To facilitate online access, recordings were converted from 11 kHz wave files into mp3 format with a bit rate of 192 kbps. The intensity was equalized for all sound files to 70 dBSPL using Praat (Boersma and Weenink, 2012) to avoid differences in volume. With two speakers, two sentences and ten voice qualities, there were 40 stimuli in total. One sound sample per voice quality can be found online in the supplementary material.

In preparation for our quantitative analysis, the voice recordings were also acoustically analyzed using Praat (Boersma and Weenink, 2012) or WaveSurfer (Sjölander and Beskow, 2005). To reduce the amount of data for treatment in our main experiment, we fed a set of 14 possible acoustic features into a factor analysis. Features with strongly correlating scores were reduced into one group, which we named according to whichever feature had the strongest regression coefficient within it. The resultant four features are defined below, and their quantitative values are given in **Table 1** (which shows their values across each of the ten different voice qualities, averaged from the two speakers and across the two sentences).


### **Table 1 | Acoustic values for the different voice qualities, averaging across speakers, and sentences.**


*Pitch-related values could not be entered for WHISPER because there is no voicing in WHISPER. Formant-related measures cannot be given for FALSETTO because the pitch is so high that formant frequencies cannot be accurately resolved.*

("app*a*rently" for sentence 1, "*e*ver*"* for sentence 2). Spectral tilt is the extent to which energy in the signal falls off as frequency increases: energy at higher frequencies is less damped when spectral tilt is shallow and more damped when spectral tilt is steep. It relates to various physiological characteristics of vocal fold vibration, including the proportion of the vibratory cycle during which the folds are open, and the abruptness or gradualness of vocal fold closure. BREATHY voice, e.g., is associated with a steep spectral tilt, CREAK and HARSH among others with a shallow tilt. Spectral tilt was quantified here using measures of the corrected first harmonic minus the corrected amplitude of F3 (H1∗-A3∗) (see Hanson, 1997 for further details).

*Pitch range* Variability of f0 in a speaker, calculated by subtracting the minimum from the maximum pitch of a voice recording and converting to semitones. This describes the differences between (for example) a "singsongy" vs. monotonous voice (i.e., large vs. small pitch range).

∗Formants are spectral peaks of intensity at different frequencies (usually measured in Hz) in the frequency spectrum of the sound. They are created by the resonances of the vocal tract (Clark et al., 2007). A vowel sound contains several formants. The lowest two formants mainly characterize the vowel quality, while all formants additionally give information about speaker characteristics.

### *Response display*

*Colors.* A forced choice response display was presented with 16 different colors, comprising the 11 focal colors of English (Kay et al., 2009): white, black, blue, green, yellow, red, gray, brown, orange, pink, and purple, plus an additional five colors also varying in luminance: pale pink, dark green, light green, cyan and dark blue. This limited set was preferred over an unlimited color picker to reduce task demands and shorten the time required of our participants. Participants also had the opportunity to describe fine-grained details of their color associations in a verbal response (see below).

Our color stimuli were created by entering red, green, blue (RGB) values into our computer display and these colors were subsequently quantified for our main analysis using a Minolta CS-100 chromameter and converted into CIELUV color space (Westland and Ripamonti, 2004). The chromameter measured the luminance (L) and chromaticity (x, y) values on ten different computer screens in different lighting conditions to get an estimate of the variation of settings that participants would use. The average of these ten measures was then used to convert the numbers into co-ordinates within the CIELUV color space, using the formula published in Westland and Ripamonti (2004, p. 50f). This color space is suitable for self-luminous colors such as those displayed on computer screens, and achieves perceptual uniformity (i.e., a given change in color value produces the same visual significance regardless of where in color space that change occurs). Within this color space, colors are represented by L∗, u∗, and v∗ co-ordinates, representing luminance, red-green and yellow-blue respectively. When converting our colors, our "reference white" was taken from the background gray, to place our palette in the correct color context. This occasionally resulted in L∗ values above the usual upper limit (100) when white is used as reference. Both RGB and L∗, u∗, v∗ values are listed in **Table 2**. A positive u∗ value stands for red tint and a negative one for green tint; a positive v∗ value stands for yellow tint and a negative one for blue tint. A high L∗ value stands for light and a low one for dark.

*Textures.* Our response display also presented visual representations of 16 textures (**Figure 1**). The selection of textures was dictated by those mentioned most often in synesthetes' descriptions of their textural concurrents, communicated in forum posts in a synesthesia community (http://www*.*daysyn*.* com/Synesthesia-List*.*html) and through personal communication. The textures most often named were: rough, liquid/fluid, smooth, shiny, hard, dry, soft, bumpy, sharp, bubbly, milky, transparent, metallic, and textiles like velvet, linen, flannel, corduroy, plaid, and felt. For logistical reasons this list of texture descriptions was reduced to 16 for use in the experiment, with each texture designed to be close to the descriptive words used by the synesthetes, but distinct from the other textures. The textures in **Figure 1**, from left to right and top to bottom, are referred to as: 1. rough, 2. smoke, 3. bumpy, 4. water, 5. rough-ish, 6. jeans, 7. milk, 8. sharp, 9. net, 10. dry, 11. drops, 12. fleece, 13. stripes, 14. foil, 15. velvet, and 16. bubbly. Textures were uniform with respect to their simulated viewing angle, and presented as gray-scale images to avoid a confounding influence of color. Pictures were taken from the database created by Halley and colleagues (Clarke et al., 2011; Halley, 2011), Brodatz (1966) and from homepages without copyright limitations.

**Table 2 | RGB values and CIELUV coordinates of the 16 colors used for creating the color patches in the online survey.**


*L\*, u\*, v\* values calculated from the average L, x, y measures of ten randomly selected computer screens.*

Our limited set was selected to allow for an assessable display, and to make the data manageable for analysis. To prepare for our analysis, the textures were quantified using human ratings gathered from 32 native English speaking participants (9 female, mean age = 26, *SD* = 8 years) who did not take part in the main experiment. Human ratings were selected since these match the perceptual space of textures better than computer algorithms (Clarke et al., 2011). Participants rated each texture along eight semantic scales following textural classifications in Rao and Lohse (1993, 1996) and Tamura et al. (1978), presented on horizontal sliders with descriptive words at opposing ends, as follows:


Participants' ratings were fed into a factor analysis to again reduce the amount of data for treatment in our main experiment. Semantic differentials whose scores were strongly correlated with a common latent variable or underlying dimension were reduced into one group, i.e., a factor. Each factor was named according to the semantic differential that had the strongest regression coefficient with it: repetitiveness, roughness, complexity, and linevs. blob-likeness. Rating results of these four semantic differentials for each individual texture image can be viewed in the supplementary material. Factor loadings, sums of squares and variance data are given in **Table 3**.

In summary, our materials comprised 40 voice samples (two speakers × 10 voice qualities × two sentences), as well as a response palette showing 16 colors (each quantified by its L∗, u∗, v∗ values) and 16 textures (each quantified along eight semantic scales).

# **PROCEDURE**

The experiment was conducted online using the software LimeSurvey (www*.*limesurvey*.*org). Participants were encouraged to use the best possible audio equipment at their disposal, usually headphones or external speakers.

Participants heard each audio file one at a time (and could replay each as often as needed before advancing) and were asked the following question: "What are your first impressions of and associations with this voice? Please describe the voice in your own words." Participants entered their replies in a text box. Below the box, there were eight sliders with the semantic differentials (the results of which are presented elsewhere; Moos, 2013; Moos et al., in press). This was followed by the color display on the same screen with the question "Which color matches the voice best?" and the texture display asking participants to choose the best match as well. At the bottom of the screen, there was space for optional comments: "On a scale from 0 to 9 (where 0 is nothing and 9 very intense), how intense are your color and texture experiences? Is there anything more you want to add?" Every time a stimulus was accessed on the homepage, the semantic differentials were displayed in random order, as were the color and the texture display.

Our materials were presented in a block design in which block 1 was the 20 recordings of sentence 1 (10 voice qualities × 2 speakers), followed by a screen that collected demographic data (where participants had the option to save results and return at a later point), followed by block 2, which was the 20 recordings of sentence 2 (again, 10 voice qualities × 2 speakers). Within blocks, all trials were presented in a random order. The study ended with a voice comparison task (for a study reported elsewhere) and a short synesthesia questionnaire collecting data about participants' types of synesthesia. The experiment lasted for about 1.5–2 h. After 2–8 months, a retest was conducted with a subset of the stimuli to test for consistency in participants' associations (in all but three cases, after 5–8 months). The subset comprised each voice quality once, with five voice qualities produced by speaker 1 and five by speaker 2. For five stimuli, sentence 1 was used, for the other five, sentence 2. This resulted in 10 stimuli. Twelve synesthetes, 10 phoneticians, and 20 controls took part in the retest.

# **RESULTS**

We first consider the cross-modal associations of our participants as they were given in the initial testing session. Here we analyzed participants' responses from both their qualitative/verbal descriptions, and their responses given via our color/texture response-display. Subsequent to this, we analyzed their consistency over time by comparing responses in the first vs. second testing sessions. We present these analyses in turn.

### **Table 3 | Factor analysis with promax rotation of texture rating data.**


*Top: rotated factor loadings of the semantic differentials for four factors. Bottom: sums of squares (SS) loadings of the above factor loadings and the proportional and cumulative variance of the data that they account for. Bold typeface indicates a strong regression coefficient of the semantic differential with a factor. The semantic differential with the strongest regression coefficient, chosen as representative of a factor, is indicated in italics.*

# **QUALITATIVE RESULTS OF VERBAL DESCRIPTIONS**

Participants' verbal descriptions of associations to our voice stimuli were coded based on the systematic methodology of the Grounded Theory (Strauss and Corbin, 1998), and each reply was inspected at least twice during coding. In total, 25 codes were created which were subsequently grouped into six different categories as listed below:

1. Associations

*Person* associations with real or fictitious people *Anno* time period of recording *Pre* comparing the speaker to one previously heard

2. Description of person


3. Feelings in listener

*Feelings* emotions or feelings evoked in the listener

4. Phonetics



### 5. Synesthetic perceptions


### 6. Unclassified

*Misc* terms not falling within the categories above.

We tested whether the use of verbal descriptions differed between the three subject groups using a MANOVA, with Gabriel's test for *post-hoc* tests because group sizes were different. The dependent variable was defined as the number of times a category was assigned to the verbal descriptions per participant. Testing the difference between the six categories produced four significant main effects, of: Phonetics [*F(*2*)* = 14*.*90, *p <* 0*.*001], Description of person [*F(*2*)* = 4*.*08, *p* = 0*.*023], Feelings in listener [*F(*2*)* = 4*.*69, *p* = 0*.*014], and Synesthetic perceptions [*F(*2*)* = 22*.*8, *p <* 0*.*001]. Planned comparisons explain these differences as follows. As would be expected, phoneticians used phonetic descriptions on average 35.2% more than synesthetes [*t(*1*)* = 5*.*16, *p <* 0*.*001]. Synesthetes used synesthetic descriptions on average 38.7% more than phoneticians [*t(*1*)* = 5*.*15, *p <* 0*.*001] and 38.2% more than controls [*t(*1*)* = 6*.*42, *p <* 0*.*001]. Perhaps reflecting their lower use of synesthetic descriptions, controls instead used descriptions of the speaker on average 13.5% more than synesthetes [*t(*1*)* = 2*.*53, *p* = 0*.*039]. They also used phonetic descriptions 22.9% more than synesthetes on average [*t(*1*)* = 4*.*24, *p <* 0*.*001]. Finally, synesthetes also described their feelings on average 4.1% more often than phoneticians [*t(*1*)* = 2*.*87, *p* = 0*.*017], and this perhaps reflects the affective quality of synesthesia (Callejas et al., 2007).

The data also show that synesthetes used terms from the "synesthetic perceptions" category in 42.7% of their responses, although the range was rather wide (2–94%). **Figure 2** breaks down these responses to show the types of concurrent modalities expressed. This finding confirms previous research that most concurrents are colors (Day, 2005; Novich et al., 2011). It also supports our decision to use color and texture displays as these are the most frequent concurrents. However, care must be taken since synesthetes' verbal descriptions may not indicate their range of concurrents, *per se*, but rather, the different degrees to which they might express them in this task.

# **RESPONSE DISPLAY**

## *Color associations*

One way to consider analysing color responses is by color category (e.g., FALSETTO voice might generate more "pink" responses, and CREAK voice might generate more "gray" responses). This approach to analysing voice quality is presented in Moos et al. (in press). Here we instead consider how quantified measures of different voice qualities map onto the quantified measures of color (and below, texture).

The influence of the acoustic measures of the voice qualities on the CIELUV coordinates of the associated colors was tested using linear mixed effects modeling carried out in R (Baayen, 2008). L∗, u∗ and v∗, representing luminance, red-green and yellowblue, respectively, were used as dependent variables. Our four acoustic features, namely f0, LTF2, spectral tilt and pitch range, were used as predictors, as was participant group, and the interactions of group with each acoustic feature. Participants were included as random effects. **Table 4** shows our significant results, as non-significant predictors were not retained in the model.

In summary, our results were:

• A higher f0 (as in FALSETTO), a higher LTF2 (as in WHISPER) and a larger pitch range (as in RAISED LARYNX) led to lighter color choices across groups, whereas a lower f0 (as in CREAK), a lower LTF2 (as in LOWERED LARYNX) and a smaller pitch range (as in HARSH) resulted in darker color associations.


Color associations with FALSETTO, for example, illustrate the dominance of red-related associations with high f0, as a majority of participants chose red, pink, pale pink, purple and orange, resulting in high u∗ values. The lower luminance values for color associations with CREAK, for example, result from black, brown, dark blue and gray selections by a large part of the participants.

**Table 4** also reveals a group difference between controls and phoneticians on the luminance scale: phoneticians associated the voices with significantly darker colors than controls. LTF2 interacts with group in the following way: phoneticians use the luminance scale more extensively in response to changes in LTF2 than controls do; there is no significant difference between synesthetes and others.

### *Texture associations*

First, a visual impression of associations between voice quality and texture is given in **Table 5**. It lists the four voice qualities that evoked the highest agreement in terms of texture associations

**Table 4 | Results of linear mixed effects modeling testing acoustic influences on participants' color associations.**


**Table 5 | The four voice quality—texture associations with highest agreement between and across groups.**


both within and across groups, i.e., those associations shared by most participants. With 16 textures offered in the response display, chance level of associations between the different textures and a voice quality is 6.25%. The strongest agreement was found for the association of the smoke-like texture with WHISPER. This texture image may have evoked thoughts of high-frequency noise travelling through the dark; some participants described associations with darkness for WHISPER because the darkness of the night is a common environment for whispering. The associations of dry cracked soil with HARSH and CREAK seems intuitive as these voice qualities give an auditory impression of a dry throat.

Linear mixed effects modeling was used to assess the influence of voice acoustics on texture associations and to reveal potential group differences in those associations. The four semantic qualities "rough—smooth," "high—low complexity," "repetitive non-repetitive" and "line-like vs. blob-like" were used as dependent variables, representing the texture choices of participants. The four acoustic features were again used as predictors. Detailed significant statistical results, with brief qualitative explanations in the last column, are shown in **Table 6**. Again, non-significant predictors were not retained in the model.

It was found that both higher pitch and higher LTF2 were associated with textures that were "smoother," "more complex," and "less repetitive." A steeper spectral tilt as in BREATHY resulted in "smoother," "less complex," and "blob-like" texture choices, whereas a larger pitch range as in RAISED LARYNX triggered choices of textures with "more repetitive" patterns. Key results are summarized in **Table 7**. A small but significant group difference in the usage of textures was found for the complexity scale: synesthetes chose "more complex" textures than phoneticians and controls. This could be due to the fact that synesthetic concurrents are on average more complex in their structure than associations of non-synesthetes which are only consciously present when triggered by visual input.

# **RETEST**

In the retest, synesthetes chose exactly the same color again 25% of the time, phoneticians 21% of the time and controls 15% of the time. Synesthetes chose exactly the same texture again 15% of the time, phoneticians 19% and controls 13% of the time (see Moos et al., in press; Moos, 2013 for more details). Synesthetes' consistency scores ranged from 10 to 60%. Although there is a tendency for synesthetes to outperform controls in consistency, the scores do not reach levels of consistency found, for example, in grapheme-color synesthesia, which are usually above 70% (Asher et al., 2006; Simner et al., 2006). Two reasons may explain this result. First, the complexity of voice as an inducer (combination of voice quality, intonation, content of words etc.) plus the additional types of synesthesia of the participants suggest that results cannot be compared to a relatively clear-cut and wellresearched type such as grapheme-color synesthesia. Second, the visual response displays may not have resembled the exact synesthetic reactions. To circumvent this issue, we also tested whether participants chose colors and textures *similar* to their choices in the initial test.

**Table 7 | Summary of significant influences of acoustic characteristics of the voices on the associated textures.**


**Table 6 | Results of linear mixed effects modeling testing acoustic influences on participants' texture associations.**


For this, a MANOVA was conducted. Tukey's method is reported for the *post-hoc* test because this is powerful while having good control over the Type I error, and results differed little from those of Gabriel's method which takes into account the different group sizes. L∗, u∗ and v∗ were used as the measures for color. Testing consistency across voice qualities, all group differences for color associations were non-significant [*F(*2*)* = 2*.*16, *p* = 0*.*117 for L∗; *F(*2*)* = 2*.*07, *p* = 0*.*127 for u∗; *F(*2*)* = 2*.*56, *p* = 0*.*079 for v∗]. However, v∗ approached significance between synesthetes and controls (*t* = 12*.*37, *p* = 0*.*063), indicating that synesthetes are marginally more consistent in the yellow-blue dimension of their color associations.

Interestingly, results for texture associations across voice qualities show partly higher consistency for phoneticians than synesthetes: phoneticians were more consistent in choosing "rough-smooth" (*t* = 6*.*40, *p* = 0*.*04) and "fine-coarse" textures (*t* = 4*.*50, *p* = 0*.*053). It could be argued that the (ir)regularity of vocal fold vibration has a strong textural parallel. Harsh and creaky voice qualities are examples of irregular voicing, with harsh often being called rough in lay terms. Also, visual textural patterns emerge from the phonetician's tool for speech analysis, the spectrogram, which looks smoother for a modal voice and coarser for a harsh or creaky voice. Phoneticians may therefore be most consistent in these measures.

# **DISCUSSION**

### **MAIN FINDINGS**

We conducted an exploratory study on voice-induced synesthesia using both qualitative and quantitative analyses. The qualitative approach—coding and analysing the verbal descriptions—gave insights into the different ways participants perceive and express their perceptions of different voice qualities. This was a necessary first step toward understanding this under-researched type of synesthesia. As conjectured, synesthetes regularly used their synesthetic perceptions to describe voices, mostly color and texture terms; phoneticians used more technical terms; controls focussed on describing personal characteristics of the speaker. Nevertheless, individual differences in the use of synesthetic descriptions and in the consistency range of color and texture associations were found within the group of synesthetes. The use of synesthetic terms in the verbal descriptions ranged from 2 to 94% across individuals, while the range of test-retest consistency in choosing identical colors and textures for the same voice qualities varied between 10 and 60%.

Parameterization of the color and texture choices made it possible to quantify how participants' responses were influenced by acoustic attributes of the different voice qualities. Higher fundamental frequencies were associated with lighter colors across groups. This is in line with the findings of Ward et al. (2006) on musical pitch, whereas de Thornley Head (2006) found group differences: pitch changes did not affect the lightness of his synesthetes' color choices, but did influence control participants. A recent study on pitch-luminance mapping found that even chimpanzees prefer to match white to high pitched sounds and black to low pitched sounds (Ludwig et al., 2011). This finding suggests a common underlying mechanism of sensory processing in primates, which seems to be hard-wired rather than acquired through culture or language. Neurons in the auditory cortex are organized tonotopically according to the frequency of sound to which they respond best in both humans and other primates (Lauter et al., 1985; Talavage et al., 2004; Bendor and Wang, 2005), similar to the arrangement of sound processing in the cochlea. This frequency map in the auditory cortex may possibly relate systematically to a luminance map in V4.

Other acoustic effects on responses are less easy to explain in terms of frequency-luminance matching and may be mediated by connotative influences. A higher f0 also led to redder color choices in our participants, e.g., for falsetto, where red, pink, pale pink, purple, and orange associations dominated the responses. Although not statistically demonstrated by de Thornley Head (2006), the graphs on p. 170 suggest that his participants also associated redder colors with higher pitched tones. A larger pitch range (as in raised larynx) resulted in lighter and yellower color choices across groups, while a steeper spectral tilt (as in breathy) triggered more blue associations across groups. Some voice qualities are judged to have different degrees of pleasantness. A breathy or modal voice, for example, is usually perceived as pleasant to listen to (Reich and Lerman, 1978). In our study a high f0, high vowel formants, a larger pitch range and a steeper spectral tilt resulted in associations with textures that were "fluid," "smooth" and have, according to Reich and Lerman (1978) and Lucassen et al. (2011), a pleasant connotation, whereas a shallow spectral tilt (as in harsh) resulted in "rough" and "line-like" texture choices and those with unpleasant connotations.

Lucassen et al. (2011) published a study on the affective connotations of texture and color. The authors list semantic differentials for color and texture ratings, namely warm-cool (for colors only), feminine-masculine, hard-soft and light-heavy. They found that more complex textures are more masculine, hard, and heavy. Light weight is associated with light colors and heavy with dark colors. Softness is associated with less saturated colors than hardness. Femininity is rated as more pink and masculinity more blue, green and dark. Warmth is perceived as more red and brown, whereas coolness is more blue and green. Although a detailed comparison between Lucassen et al.'s experiments and ours is not possible, parallels can be found for the femininemasculine scale and ratings for f0 by our participants: high f0 (falsetto, reaching frequencies typically associated with a female voice) was associated with redder colors than low f0, matching Lucassen et al.'s findings exactly. Furthermore, one synesthete had temperature concurrents with the voices which she expressed in her verbal descriptions. It is noticeable that her descriptions of warm temperature are often accompanied by red color choices and cold temperature by green or sometimes blue color choices. The interesting likely relationship between affective responses and color/texture associations is potentially worthy of further investigation, particularly in the light of visual aesthetics (Palmer et al., 2013).

### **METHODOLOGICAL ISSUES**

The use in this study of visually-presented textures as opposed to tactile presentation (Simner and Ludwig, 2012) appears to have been successful. Simulated viewing angle was kept constant and textures presented in grayscale to avoid confounds with hue. A potential confound of the texture images *per se* and their luminance was ruled out *post-hoc*: Texture images that were light overall were not associated with the same voice qualities which induced light colors. The picture of the dry, cracked soil, for example, has a relatively high luminance overall; but voice qualities which were associated with this picture were generally given dark color associations.

Occasionally, some synesthetes complained about the limited set of colors. However, using a more complicated color response display was deemed cumbersome, as it would have made an already lengthy experiment even longer. The synesthetes who found the set of colors to be limited used the opportunity to detail their color perceptions in the verbal descriptions: a majority of them used customized color terms, such as "tonic green" or "dark wine color" or named more than one color to describe a voice (see Moos et al., in press, for more details). The set of textures was also not comprehensive. It is unclear how to present textural displays more optimally to synesthetes, and there is little research on this topic. One option would be a browsing environment similar to that presented by Halley (2011) and Clarke et al. (2011), although this would be very time consuming. There were fewer comments by synesthetes about the limited set of textures than about that of colors. Potentially, their synesthetic reactions were less clear or more indifferent for textures than for colors; or they found it easier to match their concurrents to the given set because their textures were more similar to those on display.

Consistency in people's associations was measured for textures and colors through a retest. Synesthetes were marginally more consistent in some of their color associations than controls, while phoneticians showed regular patterns in some measures that underline their expert knowledge of assessing voice qualities, e.g., the extensive use of the smooth-rough scale. However, the main and unexpected—pattern to emerge from tests of consistency of synesthetic perceptions was the lack of significant differences between groups. This surprising lack of strong consistency in synesthetes' choices leaves four main interpretations.

First, it is possible that some of our group were not true voice synesthetes. Of course we tried to rule out this possibility by conducting synesthesia tests with questionnaires. Moreover, the richness of the verbal descriptions provided by the synesthetes supports the idea that they were genuine: Simner et al. (2005) have shown that verified synesthetes use significantly more descriptive color terms when describing concurrents. Nonetheless, there cannot be absolute certainty with an under-researched type of synesthesia such as voice-induced synesthesia. Second, the stimuli might have evoked voice synesthesia less strongly, and therefore less consistently, than would have been optimal. Perhaps varying voice qualities within two speakers gave in fact less perceived variation than might have been obtained using many different speakers including female voices (cf. Fernay et al., 2012), even though acoustic variation was present within the stimulus set.

Third, it might be the case that voice-induced synesthesia is not as easily defined as other types of synesthesia; ergo, the definition of synesthesia needs to be revised, especially regarding consistency as a main criterion, as has recently been suggested by Simner (2012a,b). Fourth, and relatedly, perceptions might be influenced by other types of synesthesia to a certain degree, in ways which could not be separated out in the analysis. A synesthete might attend to the voice in the first run of the test and have the corresponding synesthetic reactions. In the second run, she might attend to the words being said which induce different synesthetic reactions. The influence of other synesthesias is one of the largest difficulties faced in this experiment.

All but one of the 14 synesthetes reported having at least four other types of synesthesia. Of these, 13 had music as an additional inducer, 11 had letters, and 9 had words. These will undoubtedly be the types of synesthesia interfering most strongly with the sound of a voice. The influence of words on synesthetic perceptions could have been avoided by using nonsense syllables. However, some word synesthetes also experience concurrents with nonsense words. Moreover, if the interference of words as an inducer had been overwhelmingly strong, participants would have reported the same associations for all stimuli containing the same sentence. This scenario was not found. The complexity of voice as an inducer and its tendency to coexist with other types are, in our opinion, the most likely cause of the low consistency scores found in this experiment, and underlines the necessity both for further research on this type of synesthesia, and for redefinition of the role of consistency in the definition of synesthesia, as discussed in the next section.

# **THEORETICAL CONTEXT AND IMPLICATIONS**

Results of the verbal descriptions showed a clear distinction of the use of synesthetic terms between synesthetes and non-synesthetes: the former use more synesthetic terms than the latter. It cannot be excluded that this result may be biased since self-reported synesthetes may be more explicit in naming synesthetic associations because of their knowledge that they were selected for this experiment *as* voice-induced synesthetes.

In light of the findings of weak consistency within synesthetes and shared results across participant groups, the question arises whether voice-induced synesthesia is in fact a clearly defined and distinct variant of synesthesia at all. Furthermore, it is also possible that voice-color synesthesia may be nothing more than an epiphenomenon of *music*-color synesthesia. Indeed, many of our synesthetes had both types of synesthesia co-existing together suggesting they may be one and the same phenomenon. However, two facts argue against this: first, the co-existence of both forms is not in itself a reason to dismiss voice-color synesthesia, and this is because different variants of synesthesia do tend to coexist together within the same individual (i.e., even if they are separate forms; see e.g., Simner et al., 2006; Novich et al., 2011). More importantly, we found at least one case (LP) of a synesthete reporting voice-color synesthesia *without* the music-color variant. It is possible therefore that these forms do exist as separate conditions.

Traditionally, low consistency rates define somebody as a nonsynesthete (Asher et al., 2006; Ward and Mattingley, 2006). With new approaches, however, this rigid definition faces revision. There is a risk of circularity in defining synesthesia by its consistency over time: If non-consistent synesthetes are not defined as synesthetes, consistency becomes a defining criterion (Cohen Kadosh and Terhune, 2012; Eagleman, 2012; Simner, 2012a,b). Recent studies have shown that consistency cannot be used as a proof of genuineness for all variants of synesthesia, nor in fact for all synesthetes (Simner et al., 2011; Simner and Ludwig, 2012). With the suggestion of introducing a synesthesia spectrum (Eagleman, 2012) it becomes apparent that one can be more or less synesthetic. Based on this suggestion there are two speculations on how to define our synesthete participants. First considering consistency as a defining criterion—they could be "moderately synesthetic," Second, we could further speculate that a spectrum exists not only within types of synesthesia but also in terms of the influence of neighboring types. Music-induced synesthetes for example may show an increased likelihood of additionally having voice-induced synesthesia to a certain degree, but may be more or less consistent in their associations of this additional type.

It can be concluded that most results of our study show similarities across participant groups, feeding into the discussion that being synesthetic lies on a continuum (Eagleman, 2012; Simner, 2012b). This suggests common underlying mechanisms

# **REFERENCES**


grapheme-color perceptions. *Brain Res.* 1127, 99–107. doi: 10.1016/j.brainres.2006.10.013


in associations, which synesthetes access at a conscious and nonsynesthetes at a subconscious level. The indications in our results of individual differences in descriptive use of synesthetic terms as well as in consistency suggest that more emphasis should be put on these differences within synesthetes, and they should be taken into account when "classifying" synesthetes. In fact, a categorization of synesthetes and non-synesthetes might not be achievable in the same way for voice synesthesia as for other variants of the condition; the entanglement of multiple types of synesthesia within one individual must be taken into account in future research seeking to develop a fuller understanding of the role of voice as an inducer.

# **ACKNOWLEDGMENTS**

This work was funded by a Lord Kelvin/Adam Smith scholarship granted to the first author by the University of Glasgow.

# **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/Cognitive\_Science/10.3389/fpsyg. 2013.00568/abstract

groups. *J. Int. Phon. Assoc.* 8, 18–23. doi: 10.1017/S0025100300001699


Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (*Pan troglodytes*) and humans. *Proc. Natl. Acad. Sci. U.S.A.* 108, 20661–20665. doi: 10.1073/pnas. 1112605108


*J. Speech Lang. Law* 12, 143–173. doi: 10.1558/sll.2005.12.2.143


experience in lexical-colour associations. *Cognition* 98, 53–84. doi: 10.1016/j.cognition.2004.11.003


to us all. *Cortex* 42, 264–280. doi: 10.1016/S0010-9452(08)70352-6

Westland, S., and Ripamonti, C. (2004). *Computational Colour Science using Matlab*. Chichester: Wiley. doi: 10.1002/0470020326

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 05 June 2013; accepted: 09 August 2013; published online: 02 September 2013.*

*Citation: Moos A, Simmons D, Simner J and Smith R (2013) Color and texture associations in voice-induced synesthesia. Front. Psychol. 4:568. doi: 10.3389/fpsyg. 2013.00568*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Moos, Simmons, Simner and Smith. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Color associations for days and letters across different languages

### *Romke Rouw1 \*, Laura Case2, Radhika Gosavi <sup>2</sup> and Vilayanur Ramachandran2*

*<sup>1</sup> Department Brain and Cognition, University of Amsterdam, Amsterdam, Netherlands*

*<sup>2</sup> Department of Psychology, Center for Brain and Cognition, University of California, San Diego, La Jolla, CA, USA*

### *Edited by:*

*Michael Banissy, Goldsmiths University of London, UK*

### *Reviewed by:*

*Fiona N. Newell, Trinity College Dublin, Ireland Kathleen Akins, Simon Fraser University, Canada Steffie Tomson, University of California Los Angeles, USA*

### *\*Correspondence:*

*Romke Rouw, Department Brain and Cognition, University of Amsterdam, Weesperplein 4, 1018 XA Amsterdam, Netherlands e-mail: r.rouw@uva.nl*

While colors are commonplace in everyday metaphors, relatively little is known about implicit color associations to linguistic or semantic concepts in a general population. In this study, we test color associations for ordered linguistic concepts (letters and days). The culture and language specificity of these effects was examined in a large group (457) of Dutch-speaking participants, 92 English-speaking participants, and 49 Hindi-speaking participants. Non-random distributions of color choices were revealed; consistencies were found across the three language groups in color preferences for both days and letters. Interestingly, while the Hindi-speaking participants were presented with letter stimuli matched on phonology, their pattern of letter-to-color preferences still showed similarities with Dutch- and English-speaking participants. Furthermore, we found that that the color preferences corresponded between participants indicating to have conscious color experiences with letters or days (putative synesthetes) and participants who do not (non-synesthetes). We also explored possible mechanisms underlying the color preferences. There were a few specific associations, including red for "A," red for "Monday," and white for "Sunday." We also explored more general mechanisms, such as overall color preferences as shown by Simner et al. (2005). While certainly not all variation can be explained or predicted, the results show that regularities are present in color-to-letter or color-to-day preferences in both putative synesthetes and non-synesthetes across languages. Both letter-to-color and day-to-color preferences were influenced by multiple factors. The findings support a notion of abstract concepts (such as days and letters) that are not represented in isolation, but are connected to perceptual representational systems. Interestingly, at least some of these connections to color representations are shared across different language/cultural groups.

**Keywords: cross-modal, synesthesia, color, metaphor, letters, language, days, association**

# **INTRODUCTION**

According to the British Psychologist Chris Arnell, the third Monday in January is the day of the year at which people feel the most depressed. He coined this notion with the term "Blue Monday" (Stone et al., 1985; see also Chow et al., 2005). Interestingly, in Dutch there is an expression of "blauwe Maandag" (blue Monday), indicating a short and meaningless period of time. Color perception is an elementary property of our visual system, based on the receptors' sensitivity to different wavelengths of light. In daily life, however, these simple physiological processes have a wide array of effects, ranging from metaphorical use of color terms in poetry or songs, using certain colors to signal certain connotation (Meier et al., 2004; Meier and Robinson, 2005; Moller et al., 2009; Fetterman et al., 2012), or choosing a particular color for the interior of a house based on the atmosphere that color provides the house. It seems that our ability to "simply" discriminate different wavelengths of light is in our cognitive and affective system interconnected with many different concepts, feelings, associations, and memories (Palmer and Schloss, 2010; Taylor et al., 2013). However, little is currently known about which regularities do or do not exist, and what are the underlying mechanisms explaining such associations. One condition that reflects and augments our understanding of our ability for these types of cross-domain associations is synesthesia.

Synesthesia is a fascinating condition wherein one particular sensation evokes another, seemingly unrelated, sensation. Common types of synesthesia include colors evoked by letters or numbers, or by other ordinal sequences such as the days of the week (Baron-Cohen et al., 1987; Mattingley et al., 2001; Ramachandran and Hubbard, 2001; Beeli et al., 2005; Simner, 2007). Synesthesia can occur in a wide variety of sensory modalities (Novich et al., 2011), exemplified by synesthesias like tasteword synesthesia (Ward and Simner, 2003; Simner and Haywood, 2009; Jones et al., 2011; Richer et al., 2011) or movement-sound synesthesia (Saenz and Koch, 2008). These experiences of synesthetic sensations have a truly perceptual nature and can activate the corresponding sensory cortex (Aleman et al., 2001; Smilek et al., 2001; Nunn et al., 2002; Palmeri et al., 2002; Barnett et al., 2008b). However, other types of synesthesia are more conceptual in nature. For example, for grapheme-color synesthetes, it is not uncommon to associate a particular personality with each letter (Simner and Holenstein, 2007; Smilek et al., 2007a; Amin et al., 2011). Synesthesia is well established as a genuine and "real" condition (Baron-Cohen et al., 1987; Cytowic, 1995; Paulesu et al., 1995; Ramachandran and Hubbard, 2001; Asher et al., 2006; Eagleman et al., 2007; Barnett et al., 2008a).

One of the currently most debated issues is the degree to which this condition is "special" or unique (Cohen Kadosh and Terhune, 2012; Eagleman, 2012; Simner, 2012a,b). In recent years, a body of synesthesia research has gleaned a set of properties that set synesthetes apart from non-synesthetes. In short, synesthetes differ from non-synesthetes in their functional and structural brain properties (for a review see Rouw et al., 2011), as well as in their cognitive profile (for a review see Rothen et al., 2012). Furthermore, synesthetic experiences themselves are traditionally distinguished from normal associations by their specificity, consistency, and automaticity (in the sense that evoking concurrents does not take effort), and by the conscious and perceptual nature of the synesthetic experience (Baron-Cohen et al., 1987; Simner et al., 2005; Simner, 2012a). In contrast, commonalities between synesthetes and non-synesthetes have been found in shared trends in inducer-concurrent associations (e.g., Ward et al., 2006). For example, lighter stimuli fit better with higher pitches, and darker stimuli fit better with lower pitches (e.g., Ortmann, 1933; Karwoski et al., 1942; Marks, 1975, 1978; Hubbard, 1996).

This raises the important question why non-synesthete participants show synesthesia-like mappings across sensations. Note that the question whether the *nature* of an associative experience is shared, is different from the question whether the *specific association* is shared (Rouw et al., 2011). In particular, critical to having synesthesia (Simner, 2012a,b; Deroy and Spence, 2013) is the conscious (explicit), specific and oftentimes perceptual nature of the synesthetic concurrent. Despite absence of these explicit experiences, inter- or intramodal associations across sensations have been obtained in non-synesthetes (Marks, 1987, 1989; Vroomen and deGelder, 2000; Spence, 2011). Thus, perhaps the seemingly subjective and irregular color associations of synesthetes, are somehow related to general regularities in the typical population (see Rich et al., 2005; Simner et al., 2005).

Unfortunately, non-synesthetic cross-sensation correspondences are often measured as associations between scales (e.g., "intensity" mechanisms such as increasing loudness of sounds to increasing luminance (Stevens and Marks, 1965; Marks and Stevens, 1966), or "magnitude" processing, such as the nonsynesthetic correspondence between numerical quantities and physical size (Moyer and Landauer, 1967; Henik and Tzelgov, 1982; Foltz et al., 1984; Walsh, 2003; Cohen Kadosh et al., 2007). While these types of correspondences are intriguing in their own right, they do not allow for a direct comparison with synesthetic associations, which are marked by the specificity of the associations, e.g., between a particular pitch and a particular color. In fact, the most common synesthetic concurrents are specific colors.

In this study, we will examine linguistic-color associations in a "general" population (of both non-synesthetes and synesthetes), focusing on letters and days as inducers, and colors as concurrents. These associations were chosen as they constitute the most common synesthetic associations. Main hypothesis of the current study is that patterns of color preferences can be obtained in such a general population as well. Furthermore, the study explores potential mechanisms underlying such letter-color and day-color preferences. First, perhaps the non-synesthetic associations depend on a particular orthographic or phonological property of the letter/word. We therefore compare non-synesthetic letter-color and day-color preferences across different language groups. Second, we asked participants about conscious color associations (to be able to exclude putative synesthetes from our analyses with non-synesthetes). A conscious sensory experience is a key characteristic of synesthesia, which might or might not be related to the patterns of color associations. We contrast these participants without conscious color associations, with participants who do experience colors with (certain) days, letters, and/or numbers. We hypothesize that there are similarities in the patterns of color preferences between the two groups. Third, a set of factors are explored that might help explain general color preferences, such as within-language cross-associations and ordinality effects. Below, we summarize what is currently known about factors underlying patterns of letter-to-color and day-to-color preferences. The current knowledge stems mostly from synesthesia research, but a few studies have compared the synesthetic patterns with general (non-synesthetic) cross-modal correspondences.

# **COLOR PREFERENCES TO LINGUISTIC ELEMENTS**

While synesthetic linguistic-color associations may seem "arbitrary," studies show that they are not actually completely random. Cross-participant patterns of color preferences are consistently found in synesthetes (Marks, 1975; Rich et al., 2005; Simner et al., 2005; Barnett et al., 2008a). Barnett et al. (2009) found concordance between their study on synesthetic color preferences and those of Rich et al. (2005), and Simner et al. (2005). Studies with synesthetes have found that phonological and orthographical properties, as well as the meaning/conceptual properties of the inducers affect the concurrents (colors) in synesthesia (Barnett et al., 2009; Asano and Yokosawa, 2011, 2012; Brang et al., 2011). Simner et al. (2005) furthermore showed that these biases in color associations are shared between synesthetes and non-synesthetes. For example, the letter A tends to be red and F tends to be green. Letter-color preferences were also shared across language classes: non-synesthetic German participants had significant letter-color correspondences, which showed similarities with the English pattern of color preferences.

A more complex picture appears when the mechanisms underlying the linguistic-color preferences are examined. The regularities can both be based in a "first-order" relationship (a category of color relates to a category of letter/day inducer) and "secondorder" relationships (relative differences in inducer relates to relative differences in the concurrent)1 . As we will explain below, most researchers have studied second-order relationships, although a few findings on first-order of regularities are also reported. We examine both types of relationships, which we view as complementary rather than in conflict in providing explanations for obtained color preferences.

<sup>1</sup>We thank an anonymous reviewer for suggesting this distinction

For the sake of clarity, we divide the current literature in two types of data sets on linguistic-color associations in nonsynesthetes. The first set relates to number-color associations. As discussed above, quantitative differences between numbers have been related to quantitative changes in color properties. Cohen Kadosh and Henik (2006a) found in non-synesthetes an interference of irrelevant color luminance variations on numerical comparison and vice versa. This effect was replicated in Cohen Kadosh et al. (2008). The same authors used the same paradigm to study synesthete MM in 2006 (Cohen Kadosh and Henik, 2006b). The outcome of this testing partly contradicted their earlier findings; while the synesthete showed the congruity effect between luminance and numerical size, the controls now did not show this effect. Cohen Kadosh et al. (2007) studied nineteen digitcolor synesthetes and found that magnitude of the inducing digit was related to luminance (but not to the hue or saturation) of the synesthetic color experience. The non-synesthetic magnitudeluminance effect might be age-dependent; Smith and Sera (1992) showed that two-year-old children, but not adults and older children, associate brightness with small objects and darkness with large objects. In summary, there is a tendency to relate brighter color with smaller numerical value and vice versa. This effect has so far been more clearly and consistently found in synesthetes than in non-synesthetes.

The second set relates to letter-to-color associations in nonsynesthetes, which was examined in an elegant study by Simner et al. (2005). In this study, both synesthetes and non-synesthetes showed significant linguistic-color preferences. There was little to no relationship between color preference and alphabetical/presentation order of the letters. Instead, characteristics of the graphemes themselves influenced color preferences; the stimulus letter tends to elicit a color name beginning with the same letter (e.g., b -*>* blue). This effect was replicated in synesthetes by Rich et al. (2005). These "first-order" relations suggest shared characteristics between synesthetes and non-synesthetes in generating letter-color associations. There are also, however, differences between synesthetic and non-synesthetic preference patterns. First, synesthetes produced a greater depth of color descriptions, longer descriptions, and more color terms (Simner et al., 2005). Second, Simner et al. related properties of the color names to the preferences to those colors. These were three different types of properties (see Supplementary Material); the typicality or ease with which color names are generated according to the Battig and Montague orderings (Battig and Montague, 1969), the frequency (in English) of these color names, and how early these color names are learned in life according to Berlin and Kay's typology (Berlin and Kay, 1969, but see Pitchford and Mullen, 2005). Synesthetes tend to pair high frequency graphemes with both high frequency color terms and with the earliest (i.e., earlier learned in life) color distinctions. In contrast, non-synesthetes showed no effect of grapheme frequency, and preference patterns were not influenced by color name frequency, and the Berlin and Kay (when color names are learned). Non-synesthetes did, however, show an effect of order of material and typicality/ease of generation. Letters presented early in testing are paired with more "typical" (easy to generate, as defined by Battig and Montague) colors. The "ease of generation" ranking did not correlate with color choices of synesthetes.

The role of letter frequency in synesthetic color preferences is somewhat debated. Beeli et al. (2007) found, in German-speaking synesthetes, a positive correlation between letter frequency and saturation (highest linguistic frequency is least saturated). Digit frequency was found to be associated with luminance (with lowerfrequency digits generating darker colors). Smilek et al. (2007b) found a relationship between grapheme frequency and luminance, and consistent with Beeli et al., this was stronger for digits than for letters. In a reply to the study by Beeli et al., Simner and Ward (2008) compared this data with their previous findings, by converting the synaesthetic physical color choices from Beeli et al. into the 11 color terms from Berlin and Kay; i.e., black, white, red, yellow, green, blue, brown, orange, purple, pink, and gray. Simner and Ward found that higher-frequency graphemes tended to be paired with higher-frequency color terms, and proposed that certain aspects of the HSL color space (upon which Beeli et al. based their conclusions) may be predicted from color naming. These effects of letter frequency on color preferences were however much weaker (Smilek et al., 2007b) or not present at all (Simner et al., 2005) in non-synesthetes. Watson et al. (2012) found that different letter properties had independent mappings restricted to different dimensions of synesthetic color. Shape was related to hue and letter frequency to luminance. Similarly, Brang et al. (2011) found that more similarly shaped graphemes were related to more similar synesthetic colors. This effect was strongest in individuals who experience their synesthetic color in the outside world (projector synesthetes, Dixon et al., 2004).

Another factor affecting the linguistic-color preference is the phonetic characteristic of the linguistic unit. Rich et al. (2005) found phonetic associations influencing color choice (e.g., the letter i evokes the color white, and the letter j, /dz/, evokes the color orange). A role for phonetic properties of vowels was also found in a meta-analyses performed by Marks (1975). In all these studies, the vowel "a" predominately aroused the colors red and blue, e and i tended to be yellow and white, o tended to be red and black, u was usually blue, brown, or black, and ou (in French) was brown. The author notes that there was no systematic linguistic relation between colors and vowels in their study, (e.g., this would have explained if ou was red ("rouge") and the e was green ("vert"). Instead, the authors show that vowel-color synesthesia reflects regularities between the sound of the vowel and the hues and brightness of the colors (e.g., brighter colors with higher pitched vowels).

In summary, both synesthetes and non-synesthetes might show non-random patterns of linguistic-color preferences. Interestingly, there is evidence for some highly specific relationships across languages (e.g., the letter A tends to be red and the F tends to be green). In this study, we will probe such specific relationships, and also extend this exploration to days of the week. Furthermore, possible mechanisms underlying color-preference tendencies are explored. These can be language-specific factors; we will test the hypothesis that non-synesthetes use specific relations based on linguistic properties (such as the "r" is red or "b" is blue). Factors may also be cross-language; we will explore if phonological similarities/similar concepts across languages lead to similar color preferences. Some effects are not, or not consistently, obtained in non-synesthetes, such as the frequency effects and the tendency to relate brighter colors with smaller numerical value. Other factors have not yet been studied in non-synesthetes (in particular the effect of ordinality, see below). One factor that did however clearly relate to color preferences in non-synesthetes was "ease of generation" of the color names. This is one of the three factors studied by Simner, and in following of their findings we will test the hypothesis that for non-synesthete this particular factor helps explain preferences of colors assigned to letters or days.

One important factor that has not yet been studied in nonsynesthetes, is the sequence effects reported by Rich et al. (2005). In line with the reasoning that color associations reflect the age at which they are acquired, synesthetic colors for days of the week (learned earlier) were less likely to be predicted by the initial letter of the day than were those induced by months of the year (learned later, when the child has already learned to read and write). Similarly, as the conceptual relationship between digits and numbers is learned before the spelling of these number words, the concept of ordering was reflected in the color choice (same choice for "one" and "1"), rather than the spelling of the word ("one" and "O"). Sequence effects are of particular interest because of the strong correlation between cultures that might not share specific letter forms, allowing the relative influence of each effect to be explored in the current participant groups.

# **CURRENT STUDY**

In a series of experiments, participants are asked to assign particular colors to particular letters, numbers, and days of the week. We expect that non-random patterns of letter-to-color and dayto-color preferences are obtained. Furthermore, we hypothesize that similarities in these color preferences are obtained across languages. Third, we expect that the patterns of color preferences of the participants with no conscious color experience show similarities to the patterns of preferences of the "putative synesthetes," who indicate conscious color experiences with the days/letters. The study also contains exploratory analyses, examining possible mechanisms underlying obtained color preferences. The setup of the study is as follows. We first present day-to-color preferences. We test each of the three hypotheses: patterns of preferences, similarities in these patterns despite diversity in language, and similarities in these patterns despite diversity in (the presence or absence of) conscious color experiences. Next, mechanisms underlying the day-to-color preferences are explored, in particular the role of overall color preferences (e.g., a tendency to choose "blue," independent of which stimulus is presented). In the second section, this same sequence of analyses is repeated for letter-to-color preferences: testing the three hypotheses, followed by exploring the role of overall color preferences. The third and final results section explores the role of overall color preferences, across language groups. In particular, what is the role of the three factors examined by Simner et al. (2005): ease of color generation, entry in language (age of acquisition of the color term), and color name frequency.

# **MATERIALS AND METHODS PARTICIPANTS**

# *Dutch*

First-year Psychology students at the University of Amsterdam (all Dutch-speaking and living in the Netherlands) received course credit for participating in the current experiments, which were both part of a two-day testing session for all first-year Psychology students. Included were 429 participants, (125 male, 299 female and 4 missing information), mean age = 21 (*SD* = 6). Experiments were approved by the Ethics Committee of the University of Amsterdam, and all participants read and signed an informed consent form before starting the experiment.

### *English*

As a match comparison 92 English-speaking participants (living in the USA) were included in the current study (41 male; mean age = 21 ± 2). English-speaking participants were students at the University of California, San Diego (UCSD) and the experiment was approved by the UCSD Human Research Protections Program. Participants completed the survey online on a lab computer after completing an unrelated behavioral experiment in the Department of Psychology and were compensated with credit for experiment participation in psychology courses they were enrolled in.

In contrast to the Dutch-speaking participants (all are Dutch natives), the UCSD participants come from diverse backgrounds. The majority (39) indicated English as their native language, 13 indicated Chinese/Mandarin/Cantonese, 6 Korean, 8 Spanish, 8 Vietnamese, and 18 indicated a variety of other languages. The majority of participants (67) were born in the USA, while 25 participants were not (e.g., 4 in China, 5 in South Korea, and 5 in Vietnam). 21 Participants indicated that both parents were born in the USA, but the majority of participants indicated that either one or both parents were not born in the USA, [e.g., both parents in Vietnam (10), Mexico (6), Iran (3), or China (11)].

# *Hindi*

Forty nine Hindi speaking participants were recruited by flyer and word-of-mouth from the general UCSD community and participated in the survey (mean age = 35, *SD* = 7*.*34). These participants included 23 females and 26 males. Most of the participants (45) were born and raised in India. The others (4) were born in the United States, but were still fluent in reading, writing and understanding Hindi. 22 of the participants currently reside in the United States, while 26 reside in India, and 1 in Malaysia. Participants who reside in the United States moved from India on average about nine years ago and thus have strong cultural roots in India. Participants were given either an online survey or an identical paper version.

### *Non-USA/non-English*

The previous three participant groups were tested in order to increase diversity in language/cultural background. As an anonymous reviewer pointed out, none of the three groups are however purely monolingual/monocultural. In particular, all participants are familiar with the English language. This means that obtained effects might be driven by shared English language/ cultural influences that language and culture. Indeed, it is not easy to truly avoid this factor: in these modern times there are fewer and fewer mono-cultural participants (i.e., no influence of other culture/language through movies, television, music or internet). Unfortunately, testing such mono-linguistic/mono-cultural participants would have implied a type of recruiting that was not feasible in the context of the current project. Instead, we examined whether the effects seemed driven by the relative influence of language/culture. This was done by comparing previous results with results pertained in subgroups of participants with a strong cultural influence other than USA/English language.

In this "Non-USA/non-English" participant group, we selected a subgroup of 23 participants within the "English" participant group who were not born in the USA, neither were their parents, and who indicated a language other than English as their native tongue (see also description "English participants" above). Almost all (*N* = 20) of these participants indicated the same native country for themselves and their parents, with a major language of their native country as their native tongue. The native languages in this group were Chinese (*N* = 5), Korean (*N* = 4), Vietnamese (4), Spanish (*N* = 2), and Burmese, Farsi, German, Indonesian, Japanese, Khmer, Swedish, Tagalog. Similarly, in the "Hindi" participant group, we selected a subgroup of 15 participants who were born in India, as were their parents, and who indicated that their mother tongue was not English. All of these native languages are spoken in India, (Hindi (*N* = 5), Bengali (*N* = 2), Tamil (*N* = 2), Gujarati (*N* = 2), and Kannada, Kateli, Punjabi, Urdu). All of these 15 participants were living in India. These two subgroups can be correlated with the Dutch participant group (as these participants are all native Dutch, with Dutch as native tongue). We examine if correlations across languages are still present with these subgroups. Furthermore, the two Hindi/English subgroups are taken together (38 participants total), which is a sufficient number of participants to examine if similar color-to-day and letter-to-day regularities persist in this subset of participants, as compared with the previous findings in Hindi/English participant groups.

## **PROCEDURES**

This study examines regularities obtained in color preferences, in a "normal" participant group. It originally started with the question whether in a large group of first-year Dutch Psychology students, non-random patterns of color preferences would appear. Interestingly, the students often indicated that they felt that their assigning of a color with a day, letter or number seemed "random." Results showed however that these 'random' answers did indeed reveal regularities. As the non-random patterns were obtained, we then examined if these patterns were similar across participant groups with a diverse cultural and linguistic background. We also asked participants if they had conscious color associations with the stimuli (days and letters). Unfortunately, logistical constraints prevented more elaborate testing of the color associations of these participants. Still, the question whether participants experience colors is by at least some researchers taken as the defining feature that separates synesthetes from nonsynesthetes (Deroy and Spence, 2013), and other researchers have challenged the standard test of using consistency to determine synesthesia (Simner, 2012a). No doubt, this is an interesting and important factor by itself in understanding patterns of colorpreferences. We therefore focus on the role of this factor on the color preferences. As more elaborate testing would likely exclude at least some of these participants as "synesthetes," we separate those participants with conscious color experiences from those who do not, by referring to these groups as "putative synesthetes" vs. non-synesthetes.

# *Dutch*

First, participants read a short description of synesthesia, followed by probe questions designed to reveal any possible synesthesia; participants were asked whether to them, days of the week, certain letters, and/or certain numbers have a certain color. If participants responded "yes" to any of the probe questions, they were asked to describe the color they associated with each item (e.g., day of the week), as precisely as possible. Those who responded "no" (non-synesthetes) were instructed to still assign a color: although to you days/letters/numbers do not have a color, we would still like to know which color you chose with a certain day/number/letter. Each participant was tested twice (test-retest). In the retest, the participants indicating no colors with days, letters or numbers received additional instruction to remember which color they had provided last time, and to provide as accurately as possible the same color description as before. Participants indicating they did experience color were simply asked to provide their color association again (no instruction to remember or match their previous description). These same questions were repeated for several categories: 7 days of the week, all 26 letters of the Latin alphabet, and numbers (1–14 as well as the numbers 20, 50, 100, 250, 4000, and 20.000). In the retest, presented at least two weeks later, the same categories were repeated but with the items rearranged in a random order. The whole test was presented in Dutch.

# *English*

As in the Dutch survey, participants read a short description of synesthesia on the computer-screen, followed by questions about synesthesia and then questions about their color associations. The main stimuli where all days of the week, and letters printed in lower case or upper case [A, S, U, K, T, W, n, b, s, l, F, H, I (capital i), D, i, E]. While the UCSD students saw 16 stimuli in total, two letters were presented twice, in upper case and in lower case (s and i). As participants assigned highly similar colors to capital and small font, and because we would not compare them in subsequent analyses with the other language groups, the two lower case versions of the letters i and s were not included in further analyses, leaving 14 letters in the analyses. For the follow-up analyses, participants saw 10 words and 10 non-words (scrambled versions of each of the real words), and 11 numbers. Putative synesthetes were asked to provide the color of each item (e.g., day of the week), as precisely as possible and then to rate how strongly the color was perceived on a scale of 1–100. Non-synesthetes were asked to provide a color that they associated with the item. The colors of letters and days were analyzed and can be compared with those from the Dutch sample. (The color associations with words raised new questions that are now explored in a separate study).

# *Hindi*

All participants were presented with the same questionnaire as the English-speaking participants. 30 Surveys were administered on paper and 19 online. Participants were asked about their colorassociations for days of the week (Monday to Sunday), numbers (1, 2, 3, 4, 5, 6, 7, 8, 9), letters (phonetic translations of the letters in the English test) and words (again, the word-colors are not further analyzed). For each of the 14 letters in the English test, a phonetic translation was provided (see **Table 1**). To more clearly define the exact sound on which the letter should be translated, we provided an English word defining pronunciation of each letter. These were: wear, some, zoo, cat, tin, wobble, know, best, lamp, fun, hush, these, dumb, and ache. This translation was then send to an independent Hindi-speaking evaluator for back translation. If this resulted in a different letter than the original English letter, the letter was again translated. This process was repeated until no discrepancies between translation and back translation remained. The whole survey was presented in Hindi.

In Hindi, each of the seven days of the week is apportioned to one or more Hindu gods or goddesses, and several days have folklore or ritual fasting associated with them. The seven days are named after the "celestial bodies" of the solar system: Raviãra: Sunday, day of Sun; Somavãra: Monday (day of Moon), Mañgalvã: Tuesday (day of Mars), Budhavãra: Wednesday (day of Mercury), Guruvãra: Thursday (day of Jupiter), Sukravãra: Friday (day of Venus), and Sanivãra: Saturday (day of Saturn).

# *Coding*

Each color-item association made by each participant was coded by two research assistants according to the coding schema of Simner et al. (2005) and Rich et al. (2005). These coding schemes includes eleven basic colors: red, yellow, green, blue, purple, pink, orange, brown, black, gray, white (Berlin and Kay, 1969). In accordance with these earlier studies descriptions of other colors, were recoded according to a fixed pattern. For example, "apricot" was coded as light orange. Responses that could not be classified as a color (e.g., clear or transparent), or responses that included several colors (e.g., green and blue, or rainbow) were excluded. Turquoise was coded into green.

# **RESULTS**

# **PREVALENCE OF EXPLICIT COLOR ASSOCIATIONS** *Dutch*

*Questionnaire.* In the Dutch questionnaire a test—retest was administered. We identified participants who said "yes" to the questions on explicit (conscious) color associations both times. Of the 457 participants, 47 indicated in both the test and retest that they perceived colors with days (10%), 7 indicated colors with letters (2%), and 12 indicated colors with numbers (3%).

# *English*

*Questionnaire.* Of the 92 participants, 15 (16%) indicated that they saw colors with days of the week, 10 indicated colors with letters (11%), 9 indicated colors with numbers (10%).

# *Hindi*

*Questionnaire.* Of the 49 participants, 15 (31%) indicated that they saw colors with days of the week, 3 indicated colors with letters (6%), 2 indicated colors with numbers (4%).

# **COLOR PREFERENCES IN DUTCH, ENGLISH AND HINDI SUBGROUPS: DAYS**

In this section we present the patterns of color preferences for days of the week. In each of the three language groups, non-random color preferences were present for each of the days of the week. In the next sections, we first correlate results across languages, and then examine if these results still hold in a subgroup of participants selected on their non-English, non-USA background. Next, we examine results separately for participants indicating colorday associations ("putative synesthetes") and those not indicating color-day associations (non-synesthetes). The last subsection of this paragraph examines the effect of overall color selection biases on color-item preference patterns (across the three language groups).

# *Cross-language*

First, we examined cross-language consistencies in these day-color preferences. The number of participants choosing a particular day-color combination (*N* stands for number of colorto-day categories used by participants in that test) were correlated between the three participant groups. The distribution of these variables was not normally distributed (Kolmogorov-Smirnov of Dutch and Hindi color choices *p <* 0*.*1 and of English color choices *p <* 0*.*05), thus non-parametric correlations are calculated in this results section ("Days"). The Spearman's correlations showed consistencies in the order of day-color preferences across these languages: Hindi—English [*rs*(64) = 0*.*51, *p <* 0*.*001]; Hindi—Dutch [*rs*(66) = 0*.*54, *p <* 0*.*001], and English—Dutch [*rs*(73) = 0*.*84, *p <* 0*.*001]. These correlations suggest that participants did not randomly chose colors for weekdays. While some days do not show a clear first-choice color, there are a few days with high similarity of most important color choices across languages. **Table 2** shows similarities in the primary and secondary color preferences across languages. The strongest cross-language effects are obtained with Monday and Sunday: in each language group the strongest preference for Monday is red or blue, and for Sunday it is white or yellow. See Supplementary material for an overview of color choices per color category for these days, for each of the three language groups.

**Table 1 | Hindi letters used in these experiments, and their matching phonetic sounds.**



**Table 2 | Consistency between languages for day-color preferences.**

*The highest and second-highest color preferences are presented for each day of the week for the Dutch, English, and Hindi speaking participants. Next to each day-color, the percentage of participants choosing that particular combination (of the total participants assigning valid colors choices for that day). If two colors have an equal number of choices, both are presented in one square.*

# **NON-ENGLISH NATIVE LANGUAGE SUBGROUPS**

We selected a subgroup within the English-speaking participants who were not born in the USA, whose parents also were not born in the USA, and who indicated that their mother tongue was not English (see "Participants"). Similarly, in the Hindi group we selected a subgroup whose native language was not English, were currently living in India, were born in India, as were there parents. These two groups allowed to examine if the obtained effects were due to an underlying shared language/cultural influence (in these participant groups, USA/English). We examined the degree to which effects are diminished in these subgroup with a relatively weak influence of USA culture/English language. The color-word preferences of this group of participants were correlated with the color-word preferences in the Dutch subgroup, who also do not have a USA/English background. Both subgroups still showed significant correlations with the Dutch group; English [*rs*(77) = 0*.*46, *p <* 0*.*001] and Hindi subgroup [*rs*(77) = 0*.*36, *p <* 0*.*001]. The correlation between the two subgroups is however only marginally significant [*rs*(77) = 0*.*19, *p <* 0*.*1]. Next, we examined the degree to which in these participant groups the main effects (as reported in the previous section) were still present. We collapsed the two subgroups (Hindi and English participants with relatively little English/USA background), as else there would be too few counts or participants in these item-byitem comparisons. This subgroup has 38 participants (note that sometimes invalid answers were given, so not all color choices to a particular day add up to 38). For the color-to-days, the main effects were a red and a blue color preference with Monday, and white and a yellow color preference with Sunday. The first and second choice for Monday are still blue (*N* = 8, 26%) and red (*N* = 7, 23%). Similarly, the first choice for Sunday is still white (*N* = 6, 20%). The second choice is now red (*N* = 5, 17%) as much as yellow (*N* = 5, 17%). Thus, the main effects in day-tocolor preferences are largely preserved in these participants with diverse cultural and linguistic backgrounds. Overall, even though none of the participants included in this analysis had English as their native language or USA as their cultural background, they still shared regularities in their color-to-day preferences. However, the effects were stronger in the combined/more inclusive groups, and a possible influence of USA culture or English language (e.g., through movies, music and internet) cannot be excluded. The results are still clearly present when nobody in the participant group has an English/USA background, raising the question of why across participants with different language/cultural backgrounds, certain days tends to evoke a particular preferred color association.

# **EXPLICIT vs. IMPLICIT COLOR PREFERENCES**

Next, cross-language correlations were examined separately for participants indicating color-day associations ("putative synesthetes") and those not indicating color-day associations (non-synesthetes). This analysis revealed similar patterns of dayto-color preferences between languages, with the consistencies somewhat stronger for non-synesthetes (participants indicating that to them days of the week do not have color). The correlation between Hindi and Dutch speaking participants was significant for non-synesthetes [*rs*(67) = 0*.*54, *p <* 0*.*001], but not for putative synesthete participants [*rs*(67) = 0*.*31, *p <* 0*.*011]. The correlation between Hindi and English speaking participants was a bit higher in non-synesthetes [*rs*(63) = 0*.*52, *p <* 0*.*001], than in putative synesthetes [*rs*(41) = 0*.*46, *p <* 0*.*003]. This same pattern was obtained for non-synesthete English and Dutch speaking participants [*rs*(73) = 0*.*84, *p <* 0*.*001] vs. putative synesthete participants [*rs*(49) = 0*.*63, *p <* 0*.*001]. We offer two cautionary notes regarding these analyses. First, no thorough screening of putative synesthetes took place, so the "synesthete" group could in fact be a mix of synesthetes and non-synesthetes. Second, there are relatively few participants in each synesthetic group.

As can be expected based on these correlations, within each language group the putative synesthetes and non-synesthetes were very similar in their day-to-color preferences. Significant correlations for color-to-day preferences were obtained between putative synesthetes and non-synesthetes within the Hindi [*rs*(71) = 0*.*58, *p <* 0*.*001], English [*rs*(45) = 0*.*63, *p <* 0*.*001], and Dutch [*rs*(90) = 0*.*86, *p <* 0*.*001] speaking participants.

There are similarities across participants in day-to-color preferences. These similarities are found both across languages. Furthermore, they are not dependent on the trait of synesthesia, if anything the cross-language effects were stronger in nonsynesthetes than in participants indicating explicit day-to-color associations (putative synesthetes). What factors underlie these strong correlations? To examine this question, we first consider the role of overall color preferences. For example, participants might exhibit a general bias to choose red more often than gray; such overall preferences could skew the day-to-color preference patterns.

# **OVERALL COLOR PREFERENCES**

This section presents exploratory analyses examining possible underlying mechanisms to the day-to-color preferences. First, to examine the effect of overall color selection biases on color-item preference patterns, we calculated overall color selection preferences in the three participant groups. Next, we related these preferences to the three factors studied by Simner et al. (2005), as explained in the Introduction: 1. order of entry of color into language (e.g., "white" is learned earlier in life than "yellow"); 2. color name frequency (in English language); 3. ease/order of color generation (in spontaneous generation of color words, some are produced earlier and more often than others). The sequence in which colors are ordered according to color entry and color name frequency did not correlate with overall color preferences in English, Hindi or Dutch speaking participants. In contrast, the color frequency did correlate with ease of color generation in all three participant groups [*rs*(11) ranged between 0.84 and 0.96, *p <* 0*.*001]. Thus, the most frequently produced colors were those ranked highest for typicality/ease-of-generation (Battig and Montague, 1969).

This raises a question of whether the cross-language correlations are driven by this overall bias. The analyses performed on day-to-color preferences per participant group were repeated while controlling for the factor "ease of color generation." This analyses showed that the correlations were still significant: English and Hindi [*rs*(61) = 0*.*32, *p <* 0*.*05], English and Dutch [*rs*(61) = 0*.*76, *p <* 0*.*001], Dutch and Hindi [*rs*(61) = 0*.*40, *p* = 0*.*001].

The overall number of choices for a color was related to ease to generate these colors. What, then, is the influence of these general color preference on the patterns of day-to-color associations we obtained? Are the correlations amplified by general color biases within the participant group? We next examined day-to-color preferences while controlling for overall color biases. First, the overall color selection frequencies (across days) were calculated for each color in each language group. Then, for each language group, and for each day, a chi-test was computed to determine whether the number of selections for a given color was significantly different from the expected value (based on the weekly percent selection of that color in that language group). Following the previous results, we examined whether the red and a blue color preference for Monday still holds, and the white and yellow color preference for Sunday. **Table 3** shows only the chi values that are at or below 0.05 (and for which the color is chosen more often than on average across all weekdays). Note that this is an exploratory study only, thus the variety of results in the whole table is mostly meant to create hypotheses for follow-up studies.

Some clear effects are obtained in this analysis. First, red is assigned to Monday in all language groups at a significantly higher rate than to the other days of the week. Interestingly, red is also chosen at a significantly higher rate only for Monday in all language groups. The other effect shared by all language groups is assigning the color white to the Sunday. The association between yellow and Sunday was not confirmed by this analyses and thus may in fact be reflecting an overall color bias. Finally, blue is now only found related to Monday. This effect is present in Dutch and English but not in Hindi speaking participants. Possibly it reflects the notion of "Blue Monday," which exists in Dutch ("blauwe maandag") as a saying, and as an urban myth (the most depressing day of the year) in English. As far as we know such expression does not exist in Hindi. The rest of these (exploratory) tests showed that the only other day related to white is the Saturday (in the Dutch speaking group). Furthermore, primary colors black, white and gray are only given to Monday or week-end days. Different from the previous analyses that did not take overall color bias into account, we now also find that days in the middle of the week (Tuesday, Wednesday, Thursday and also Friday) are matched to secondary and/or more complex colors, namely purple, pink, orange, yellow, green, and brown.

### **SUMMARY: DAY-TO-COLOR PREFERENCES**

In this section we present the patterns of color preferences for days of the week. In each of the three language groups, non-random color preferences were present. We obtained cross-language correlations on the patterns of color preferences. Next, we explored possible underlying mechanisms. We first established that these results still hold in a subgroup of participants selected on their non-English, non-USA background. The cross-language correlations were still present, and specific main effects were also still present: a preference for red or blue color with "Monday" and white color preference to "Sunday." Next, we examined


**Table 3 | Colors for each of the weekdays that were significantly different from expected values (in a chi-test), in the Dutch (D) English (E) and Hindi (H) speaking participant groups.**

*Expected valued is based on the percentage selection of each color in each language group.*

results separately for participants indicating color-day associations ("putative synesthetes") and those not indicating color-day associations (non-synesthetes). Synesthesia does not appear to be the underlying reason for the cross-language similarities; instead the cross-language effects were stronger in non-synesthetes than in putative synesthetes. The last subsection of this paragraph presents exploratory analyses on the effect of overall color selection biases on color-item preference patterns (across the three language groups). It showed that while "ease of color generation" affects color choices, taking out this factor still results in significant cross-language correlations. Furthermore, taking out overall color preferences still shows particular day-to-color main effects (e.g., red Monday and white Sunday).

# **COLOR PREFERENCES IN DUTCH, ENGLISH AND HINDI SUBGROUPS: LETTERS**

In this second subsection, we show patterns of color preferences for letters. Analyses are with the 14 letter stimuli presented to all language groups. These letters were chosen from the latin alphabet to include both vowels and consonants, letters from both the beginning and end of alphabet, and letters with both curved and straight-shapes. For the hindi questionnaire, letters were translated based on their phonological properties (see methods section). We first examine letter-to-color preferences across the three language groups. We then examine whether cross-language similarities in letter-color preferences still hold in a subgroup of participants selected on their non-English, non-USA background. Next, results are examined separately for participants indicating color-day associations ("putative synesthetes") and those not indicating color-day associations (non-synesthetes). The last subsection of this paragraph examines the effect of overall color selection biases on color-item preference patterns (across the three language groups).

# *Dutch, English and Hindi subgroups*

First, a cross-group analysis was performed on color preferences with letters. English and Hindi letters were matched on phonology (but of course differed in orthography), see **Table 1**. The distribution of these variables (reflecting the number of particular colors assigned to particular letters, in each of the three language groups) was not normally distributed (Kolmogorov-Smirnov test of Dutch, Hindi and English color choices were all significant, *p <* 0*.*005), thus non-parametric correlations are calculated in this results section ("Letters"). English-to Hindi color associations correlated significantly [*rs*(136) = 0*.*43, *p <* 0*.*001]. English and Dutch letters had exactly the same shape, and similar (but not exactly the same) phonology. Again significant correlations [*rs*(167) = 0*.*68, *p <* 0*.*001] were obtained. Dutch and Hindi had different orthography and similar (but not exactly matched) phonology, but still color preferences correlated [*rs*(139) = 0*.*59, *p <* 0*.*001].

As can be viewed in **Table 4**, the (first and second) strongest color-to-letter preferences have different cross-language consistency for the different letters. A few specific effects, however, do arise. First, there is a strong tendency to choose red for the letter/sound "A." This effect is present in all language groups. Second, there is a tendency to choose blue for the letter/sound B. See Supplementary material for the percentage color choices for the letter A and letter B in the three language groups. Perhaps this is because of linguistic priming; both the English and Dutch color words ("Blue" and "Blauw") start with the letter B. (In Hindi the word blue starts however with "*N*"). Other preference effects are weaker, and not expected: for example, it is not clear to us why the letter T has a green or blue color.

The "Red A" replicates earlier findings (Rich et al., 2005; Simner et al., 2005; Barnett et al., 2009). In this study, the effect is replicated in different participant groups with different linguistic backgrounds; Dutch, English and Hindi. The preference for the color red for the Hindi letter pronounced as "A" indicates that it is not the shape, nor the particular (latin) letter identity. One possible explanation is that the red color is somehow connected with the position of the letter in the alphabet: the first letter in each of the three alphabets gets a "signal" color. While not all calendars are typeset in the same way, for many people (in USA, Holland, India), Monday is the start of a new workweek. Similarly, the red color of the Monday could be to mark it is the first day of the (work)week. To explore this explanation, the preferred color for the number 1 was determined for each of the three language groups. In the Hindi subgroup, of all numbercolor combinations, the highest preference was obtained for red "1" (14 participants indicated a red "1," 33% from the total of 42 provided colors to the number 1). In the English group, red and blue were most often chosen (each 17 participants, 18.5% of the choices). The Dutch participant group behaved differently, with white (22%) and yellow (21%) as most common color choices. The preference of white with the number 1 is in line with previous findings with English and German speaking synesthetes (Beeli et al., 2007; Barnett et al., 2008a) and English speaking synesthetes as well as non-synesthetes (Rich et al., 2005). Thus, while there is some support for the idea of the red color of "A" or "Monday" as signaling the first (ordinal) item in a sequence, the evidence for associating the number "1" with red is rather mixed.

### *Non-English native language subgroups*

We looked at the subgroup of participants in both the Hindispeaking and English-speaking groups who were not born in USA and did not have parents born in the USA, and whose native language was not English (for details see the "participants" section). If an overall USA/English language influence is driving the


*vowel of the color name).* shared color-preferences, the effects should be diminished in the current subgroups. The letter-color preferences of these two subgroups correlated [*rs*(153) = 0*.*84, *p <* 0*.*001]. Furthermore, the color preferences of both groups correlated with the Dutch group; English participants [*rs*(154) = 0*.*46, *p <* 0*.*001] and Hindi participants [*rs*(153) = 0*.*56, *p <* 0*.*001]. The Hindi subgroup also correlated with the color preferences of the Dutch participant group [*rs*(153) = 0*.*56, *p <* 0*.*001]. Thus, while none of the participants included in this analysis had English as their native language or USA as their cultural background, they still clearly showed shared regularities in their color-to-letter preferences. Furthermore, these correlations were of comparable size to the ones obtained in the previous (overall) analyses. Next, we examined whether the main color-to-letter regularities reported above still persist in this combined subgroup (English and Hindi who are non-native English/USA). These participants showed a clear and strong letter-to-color preference for the letter A (*N* = 18, 51%). Next, there was a tendency to choose black with the letter K (*N* = 10, 28%), for blue with the letter "b" (*N* = 9, 28%), and green for T (*N* = 9, 24%). This is in line with the findings in the native Dutch group (see **Table 4**), where the strongest effects were a red A and a blue B.

# *Explicit vs. implicit color preferences*

Cross-language consistencies were examined in putative synesthetes (participants indicating explicit color associations with the letters/days) vs. non-synesthete participants. While orthography was different, the Hindi- Dutch correlation in nonsynesthetes were significant [*rs*(139) = 0*.*26, *p* = 0*.*002]. It was however not significant between putative synesthetes [*rs*(139) = 0*.*09, *p* = 0*.*27]. Similarly, the Hindi - English [*rs*(134) = 0*.*44, *p <* 0*.*001] correlations were significant in non-synesthetes, but again not significant for putative synesthetes [*rs*(71) = −0*.*05, *p* = 0*.*70]. The English and Dutch alphabet has the same orthography (but sometimes different pronunciation) and showed significant correlation in non-synesthetes [*rs*(79) = 0*.*31, *p* = 0*.*006] and in putative synesthetes [*rs*(79) = 0*.*29, *p* = 0*.*009].

While between languages the correlations were higher for nonsynesthetes than for putative synesthetes, there is an overall effect of between-participant consistency in letter-to-color choices. In each individual language group the putative synesthetes and non-synesthetes were very similar in their day-to-color preferences: English [*rs*(84) = 0*.*42, *p <* 0*.*001]; Dutch [*rs*(182) = 0*.*56, *p <* 0*.*001] and Hindi [*rs*(140) = 0*.*23, *p* = 0*.*007].

# *Overall color preferences*

Next, the overall color preferences (collapsed across letters) were calculated per language group. Color preferences of the English speaking group correlated highly with the Hindi group [*rs*(13) = 0*.*91, *p <* 0*.*001] and with the Dutch group [*rs*(14) = 0*.*95, *p <* 0*.*001, *N* = 14]. The Dutch overall preferences correlated highly with the Hindi preferences [*rs*(13) = 0*.*91 *p <* 0*.*001].

To examine the nature of the overall color bias, these color preferences were correlated with Color Frequency, Color Ease, and Color Entry. Frequency of Color Name did not correlate with color preference, for the English participants (the factor was extracted originally from English language, see Simner et al., 2005) the correlation was [*rs*(11) = 0*.*05, *p* = 0*.*89]. Color entry specifically correlated in Hindi subgroup only, [*rs*(10) = 0*.*65, *p* = 0*.*04] (this is significant both for putative synesthetes and non-synesthetes). Ease of Color Generation correlated with preferences in all groups, English [*rs*(11) = 0*.*85, *p* = 0*.*001], Dutch [*rs*(11) = 0*.*90, *p <* 0*.*001], and Hindi [*rs*(11) = 0*.*89, *p* = 0*.*003].

This raises a question of whether the cross-language correlations are driven by this overall bias. The analyses performed on color-letter preferences per participant group were repeated for letters, while controlling for Ease of Color Generation. This analyses showed that the correlations were still significant: English and Hindi [*rs*(129) = 0*.*331, *p <* 0*.*001], English and Dutch [*rs*(129) = 0*.*60, *p <* 0*.*001], Dutch and Hindi [*rs*(129) = 0*.*28, *p* = 0*.*001].

### *Chi-test*

As previously performed in the analyses of colored days, we examined the pattern of color preferences with letters while taking overall color bias for letters into account. For each language group, and for each letter, a chi-test was computed to determine whether the number of selections for a given color was significantly different from the expected value (based on the percent selection of that color in that language group). We examined whether the main effects in the previous analyses still hold in this control analysis; these were red/A, blue/B, and a green or blue T. This analysis only showed that the "red A" and "blue B" effects are still present in Dutch and English, but not Hindi, subgroups. The blue or green association with T is not present. **Table 5** shows only the chi values that are at or below 0.05 (and for which the color is chosen more often than on average across all letters). As with Table 3, please note that these results are exploratory only. These patterns will need to be confirmed by further research on larger samples of individuals in these language groups.

### **SUMMARY: LETTER-TO-COLOR PREFERENCES**

In this section we present the patterns of color preferences for letters. We found cross-language similarities in color choices, even with the Hindi language group were letters were matched based on phonology. While most letters do not show a clear and strong "first choice" color across languages, a few specific preferences do appear. Particularly strong is the cross-language preference for red with the letter A, but also a preference of blue with B and of green or blue with T. Next, we explored possible underlying mechanisms for non-random color preferences. We first established that both the cross-language correlations and the obtained specific "first-choice" effects still hold in a subgroup of participants selected on their non-English, non-USA background. Next, we examined results separately for participants indicating letter-to-day associations ("putative synesthetes") and those not indicating these associations (nonsynesthetes). Synesthesia does not appear to be the underlying reason for the cross-language similarities. Again, while there are similarities in color choices between putative synesthetes and non-synesthetes, the cross-language regularities were stronger in non-synesthetes than in putative synesthetes. The last subsection of this paragraph presents exploratory analyses on the effect of overall color selection biases on color-item preference patterns (across the three language groups). It showed that while


**Table 5 | Colors for each of the letters that were significantly different from expected values (in a chi-test), in the Dutch (D) English (E) and Hindi (H) speaking participant groups.**

*Expected valued is based on the percentage selection of each color in each language group. Only the chi values that are at or below 0.05 are presented.*

"ease of color generation" affects color choices, taking out this factor still results in significant cross-language correlations. An exploratory analyses showed that some specific effects, such as green or blue with T, might be explained by overall color preferences. There are however still specific "red A" and "blue B" effects.

# **OVERALL COLOR PREFERENCES (ACROSS LANGUAGE SUBGROUPS)**

We next explored the role of overall color preferences in the assignment of colors to days and letters. The data are collapsed across the three language groups. In this section we examine the role of three factors possibly influencing color biases, following the analyses of Simner et al. (2005): ease of color generation, color entry, color name frequency'. As in Simner et al., the color name frequency was measured with English color words (Appendix I). Based on the previous results, we expect that the factor "ease of color generation" is particularly important for non-synesthetes, as compared with the putative synesthetes (who indicate to experience colors with days or letters).

# *Days*

We first looked at the color responses provided with days of the week, across language subgroups. The data were collapsed to letter-color preferences over the three language groups. In these collapsed data, we calculated the percentage of number-color choices separately for putative synesthetes and non-synesthetes. These variables indicate, per day, the ratio of how often a certain color is chosen (from all color choices for that day). These two variables, percentage day/color choice for putative synesthetes and non-synesthetes, were not normally distributed [Kolmogorov-Smirnov; *Ks*(85) = 0*.*16 *p <* 0*.*001 and *Ks*(85) = 0*.*14, *p <* 0*.*001], respectively), thus a non-parametric correlation was calculated. The correlation showed a high correspondence between the synesthetic and non-synesthetic color choices, [*rs*(85) = 0*.*87, *p <* 0*.*001].

Next, we examined whether the color choices were related to ease of color generation, and found this significant for both non-synesthete [*rs*(77) = 0*.*73, *p <* 0*.*001] and putative synesthete [*rs*(77) = 0*.*68, *p <* 0*.*001] participants. Entry in language (age of acquisition of the color term) showed a significant, but weaker, correlation with both and non-synesthete [*rs*(77) = 0*.*28, *p* = 0*.*01] and putative synesthete [*rs*(77) = 0*.*33, *p* = 0*.*003] participants. There was no significant relationship with color name frequency. As this latter factor might be a more language-specific effect, we separately examined the English participant group, but again found no correlation between color name frequency and color preference in putative synesthetes and non-synesthetes (*p >* 0*.*1).

A possible alternative explanation for the cross-participant similarities is that ease of color generation underlies the correlations. Therefore, the correlations were repeated with ease of color generation partialled out; still a strong correlation was maintained between "putative synesthete" and non-synesthete color choices [*rs*(74) = 0*.*80, *p <* 0*.*001].

Thus, there are regularities in day-to-color preferences, and these are shared between the participants who indicate that to them, days have color, and those who do not experience color with days. For both participant groups, overall color preferences to days are related to ease of color generation and entry into language.

# *Letters*

We then explored the overall color biases with letters, across language subgroups. Kolmogorov-Smirnov tests also indicated that neither 'putative synesthetic' [*Ks*(182) = 0*.*20, *p <* 0*.*001] nor "non-synesthetic" letter-to-color percentages [*Ks*(182) = 0*.*12, *p <* 0*.*001] were normally distributed, therefore a nonparametric correlation was used. This showed similar overall color-to-letter preference for putative synesthetes and nonsynesthetes [*rs*(182) = 0*.*69, *p <* 0*.*001].

Non-parametric correlation analyses showed a strong effect of "ease of color generation" on the color choices of nonsynesthetes [*rs*(154) = 0*.*77, *p <* 0*.*001] and a significant, but weaker, effect with the putative synesthetes [*rs*(165) = 0*.*33, *p <* 0*.*001]. Furthermore, "color entry" correlated with nonsynesthete color choices [*rs*(140) = 0*.*28, *p* = 0*.*001], but not those of the putative synesthetes. There was no effect of "color name frequency" on the choices of color categories of non-synesthetes or putative synesthetes. As color name frequency was measured with English color words, a separate analyses on the "English" participant group was performed. This however again showed no correlation between color name frequency and color preferences, either in the non-synesthetes [*rs*(160) = 0*.*05, *p* = 0*.*51] or in the putative synesthetes [*rs*(85) = 0*.*15, *p* = 0*.*16].

If ease of color generation influences both synesthete and nonsynesthete color choices, does this explain the obtained correlation between the two participant groups? A partial correlation was run, showing that synesthetic and non-synesthetic color choices still correlated [*rs*(151) = 0*.*671, *p <* 0*.*001] when controlling for the factor "ease of color generation."

### *Summary: overall color preferences*

There was a clear effect of "ease of color generation" on the color choices, somewhat stronger with the non-synesthetes but also significant for putative synesthetes. There is a weaker effect of "entry in language" (age of acquisition of the color term) on both participant groups, and no effect of "color name frequency". Note however that in previous results this effect was related to specific properties of the color that were not measured in the current study, as the naming procedure in assigning colors only defines categories of the color hue. The cross-participant correlations (putative synesthetes and non-synesthetes) show that there are regularities in the data. These regularities are still present if the overall color preference based on "ease of color generation" is partialled out.

# **DISCUSSION**

In the current study we find statistically significant, non-random patterns of day-to-color preferences and letter-to-color preferences in non-synesthetes. Moreover, there are similarities in these patterns across three language groups: Dutch, English and Hindi language. The third hypothesis was also confirmed: there are similarities in the patterns of color preferences for non-synesthetes as for the putative synesthetes in this study. While clearly not all variation is explained by cross-language associations, and random (or unexplained) influences are also present in the color choices, results suggest regularities as well. As discussed in the Introduction, regularities can both be based in a "first-order" relationship (a category of color relates to a category of the letter/day inducer) and "second-order" relationships (relative differences in inducer relates to relative differences in the concurrent). As expected, our results indicate a role for both types of regularities, in a complementary rather than excluding manner. A few specific day-to-color preferences appear to be particularly strong and consistent, such as red/Monday, blue/Monday, and white/Sunday. The day-to-color preference patterns were furthermore shared over different language groups, and also present in participants with a cultural background other than USA/English. In letter-to-color preferences, we also obtained certain specific preferences. The strongest effect is the red/A bias, which has previously been reported in English (Marks, 1975; Simner et al., 2005) and is now extended to English, Dutch and Hindi language subgroups. Furthermore, biases for blue/B, and green or blue T were obtained. The overall letter-to-color preference patterns are at least to some extent shared across the language groups (and were still present in participants with a cultural background other than USA/English). The similarities in the patterns of color preferences between Hindi and English/Dutch indicates that orthography of the letters, and linguistic properties of the weekday names, are not the only or defining characteristics of the color preferences. As we will explain below, it is likely that different factors are simultaneously at play in generating color preferences. As expected, there are shared biases in color preferences of participants with conscious color experiences with letters ("putative synesthetes"), and participants who do not report such a conscious color association (non-synesthetes). Exploring the patterns of color preferences, we also obtained differences between the groups, as the participants without any explicit color associations showed stronger cross-language similarities (both with letters and days) than the participants who indicated to have conscious color associations with this material. Perhaps not surprising (and in replication of the results of Simner et al., 2005), the factor "ease of color generation" influenced color preference, and this effect was effect was somewhat stronger for the non-synesthetes (compared with putative synesthetes). Importantly, partialling out this factor still led to significant correspondences in cross-language color associations.

Exploratory analyses examined the factors driving the color preferences. Possibly, the preference of "red" with Monday or the letter A is to mark the start of a sequence (of the workweek/ of the alphabet). In line with obtained linguistic effects in synesthetes (Rich et al., 2005, but see Marks, 1975; Simner et al., 2005), there might be effects in non-synesthetes as well. An example of such linguistic-specific effects is the preference of blue with "b" (Simner et al., 2005), which in the current study was obtained both in English as in Dutch (the Dutch word for blue is "blauw") but not in Hindi (Hindi word for "blue" does not start with a b). These are not the only factors influencing color preference, however. Non-synesthetes showed an overall bias to select colors that are easy to generate (Battig and Montague, 1969). To a lesser extend, there was also a bias to choose colors that were learned earlier in life (Berlin and Kay, 1969).

Mechanisms that allow associations between seemingly unrelated sensations appear to be common to us all. This is illustrated by the mere existence of figurative speech and metaphors, prevalent in art but also ubiquitous in daily life. These cross-modal correspondences resemble synesthesia, in terms of making the same type of associations. One famous example is the association of made-up round vs. spiky shapes with their imaginary names: ("maluma" vs. "takete," Köhler, 1947; "bouba" vs. "kiki," Ramachandran and Hubbard, 2001). Another wellknown finding is the connection between numbers and space in the spatial-numerical association of response codes (SNARC, Dehaene et al., 1993; Fias et al., 1996). Cross-modal mappings exist across different types of information (e.g., pitch /musical intervals to brightness or shapes (Karwoski et al., 1942; Marks, 1974; Hubbard, 1996) or taste/flavors to sounds (Simner et al., 2010). Furthermore, increased intensity tends to be intuitively related to increased intensity across different types of media (e.g., loudness/brightness/size/ (Marks, 1974, 1987). For a review of cross-modal correspondences see Spence (2011). Clearly, the similarity between synesthetes and non-synesthetes exists in the type of associations made. The current study suggests that while participants without conscious color experiences often commented during the experiment that their answers were "completely random," in fact they were not. As synesthetes, they show patterns or biases in their patterns of color-to-concept associations. Furthermore, non-synesthetic biases in cross-modal correspondences can even be found cross-language/culture. We suggest that the biases that are likely to influence a particular color chosen with a particular letter or day might be more generally shared across cultures and languages as well as across individuals.

One limitation of the current study is a degree of similarity between cultures; Dutch and American participants share a shared influence of a culture in the structure of the work-week and weekend, and associations such as Sunday as a day of rest or religious day. Hindi-speaking participants, currently living in America, may share some of these influences, depending on the duration of their time in the United States. While the questionnaire itself was completely in Hindi and the participants were native speakers, this might have influenced their color choices. The commonalities in letter-sounds could be due to some shared cultural/language aspect, or translation of English influences into the native (Hindi) language. Alternatively, the commonalities are related to cross-language sound-to-color associations which are either biological or learned pre-literacy. Spector and Maurer (2008, 2011) found that some shape-to-color associations are present pre-literacy and depend on the shape of the letter, while others are later learned literacy effects (such as the red A and green G). These findings show the influence of shape of the letter (Brang et al., 2011), the current results furthermore show an influence of sound of the letter. Given possible shared language/cultural influences, it would be interesting to see if correspondences can also be found in other, additional languages/cultures.

In this and previous studies, similarities are obtained in the cross-linguistic associations in synesthetes and non-synesthetes. What does this tell us more generally about how "different" synesthetes are? As we have argued before (Rouw et al., 2011; "trait vs. type"), similarity between synesthesia and non-synesthetes lies in the exact correspondences made between apparently unrelated modalities ("type"). What is different between these "normal" cross-modal correspondences and synesthetic experiences ("trait"). Most important is the phenomenology of the experience, which is more explicit/conscious, precise, and consistent in synesthetic than in non-synesthetic experiences. Presumably, synesthetic concurrents (but not non-synesthetic color associations), are really qualitative experiences, rather than mere (semantic) associations. In line with these reports, functional and structural brain differences between synesthetes and non-synesthetes are found in sensory brain areas (Nunn et al., 2002; Hubbard et al., 2005a; Rouw and Scholte, 2007; Hupe et al., 2012). Second, non-synesthetes have to exert effort to generate associations (there were many comments during the experiment from the non-synesthetic participants; that it was a silly task, and their color choices felt "completely random"). In contrast, for synesthetes the associations are described as "automatic" in the sense that it takes little effort to produce the concurrent when presented with the inducer. This characteristic of synesthesia might very well be related to the overwhelming evidence (Esterman et al., 2006; Muggleton et al., 2007; Weiss and Fink, 2008; Jäncke and Langer, 2011; Rouw et al., 2011; Specht, 2012) of the role of the parietal cortex in synesthesia. This is most commonly interpreted as reflecting the "hyper binding" present in synesthetes (although the parietal cortex has also other roles (Hubbard et al., 2005b; Cohen Kadosh et al., 2008). Importantly, next to these behavioral and brain differences between synesthetes and non-synesthetes, there is evidence for a genetic predisposition for synesthesia (Asher et al., 2009; Brang and Ramachandran, 2011; Mitchell, 2011; Tomson et al., 2011) and synesthesia has been found "running in the family" (Barnett et al., 2008a). The studies on the patterns of color preferences show how the nature of the color experiences are different: the (putative) synesthetes show more diverse, more specific, more consistent color associations than the non-synesthetes (Palmeri et al., 2002; Hubbard and Ramachandran, 2005; Eagleman et al., 2007). Furthermore, nonsynesthetes are more influenced by general factors as evidenced in the current study (see also Rich et al., 2005; Simner et al., 2005; Barnett et al., 2009) by stronger cross-language effects, as well as a stronger effect of the factor "ease to generate color name." Thus, evidence so far indicates that the differences are found when examining the trait of having synesthesia (what underlies having these experiences). While similarities are found when examining which particular types of associations are made. The trait provides a predisposition to develop unusual experiences (Rouw et al., 2011). The types are shared biases leading to non-random patterns of associations. Furthermore, these biases are shared among different (language/culture) groups. Whether nature or nurture (or a combination of the two) underlie these associations, is a topic of future research.

As a final note, previous studies have examined synesthesia in non-English languages. Simner and colleagues (Simner et al., 2011) found that both native and non-native Chinese speakers with synesthesia experience colors with Chinese characters. Again, these linguistic-to-color associations show (non-random) patterns of associations. For at least some of the synesthetes, the color choices of the characters and words are influenced by the initial letters. Asano and Yokosawa (2012) studied synesthetic coloring of Kanji characters, which are acquired later in life than other types of graphemes in Japanese language. Synesthetic colors were found related to phonology and meaning more than to orthography. At least some influence from earlier learned languages on the later languages were obtained. Indeed, age of acquisition could influence color-letter associations (Asano and Yokosawa, 2011). Barnett et al. (2009) examined synesthetic colors with months, days, and numbers in bilingual synesthetes. The (across-language) words had more similar colors based on commonalities in visual form across languages. In particular months (as compared with numbers or days) had similar colors based on beginning with the same first grapheme, the authors suggest that this is possibly because months are learned later in life when the child is already learning to read and write. Again, these findings with synesthetic colors suggest a mixture of effects influencing linguistic-color associations. This does not necessarily mean that one finding conflicts with another: the different influences can point at different moments in life influencing color preferences, earlier for perceptual involvement of low-level perceptual processing (e.g., Brang et al., 2011) and later for more conceptual, semantic or linguistic influences (e.g., Barnett et al., 2009; Asano and Yokosawa, 2011, 2012).

While analyses of putative synesthetes vs. non-synesthetes should be interpreted with care, the results show similar patterns of color preferences between these groups, both cross- and within-language. Color choices are most likely driven by multiple factors, including linguistic/phonetic influences, orthographic properties, and sequence/ordinality effects. More general effects are also present, such as ease with which the color name is generated. We propose that both synesthetic and non-synesthetic associations show that abstract concepts are not represented in isolation, but are connected to other representational systems. The presence of connections across-representation is independent of whether participants report conscious associations (as occurs in synesthesia) or not. Interestingly, we found that patterns of day-color and letter-color associations are present in non-synesthetes, and biases in color preferences are shared across different language/cultural groups.

# **ACKNOWLEDGMENTS**

We would like to thank Jackie Thind. Thanks to Marcus Spaan for help with the data collection of Dutch participants. We thank Sharma Aditya, Chaintanya Bhatt, Bhuvan Molparia, for help with translation. We thank the participants for their effort and time. We thank the reviewers for their valuable comments and suggestions.

# **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fpsyg. 2014.00369/abstract

## **REFERENCES**


Cytowic, R. E. (1995). Synesthesia: phenomenology and neuropsychology. *Psyche* 2.


Marks, L. E. (1974). On associations of light and sound: the mediation of brightness, pitch, and loudness. *Am. J. Psychol. 87*, 173–188. doi: 10.2307/1422011


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 03 August 2013; accepted: 07 April 2014; published online: 27 May 2014. Citation: Rouw R, Case L, Gosavi R and Ramachandran V (2014) Color associations for days and letters across different languages. Front. Psychol. 5:369. doi: 10.3389/fpsyg. 2014.00369*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Rouw, Case, Gosavi and Ramachandran. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# How well do you see what you hear? The acuity of visual-to-auditory sensory substitution

### *Alastair Haigh1, David J. Brown1, Peter Meijer <sup>2</sup> and Michael J. Proulx3 \**

*<sup>1</sup> Crossmodal Cognition Laboratory, School of Biological and Chemical Sciences, Queen Mary University of London, London, UK*

*<sup>2</sup> Metamodal BV, Eindhoven, Netherlands*

*<sup>3</sup> Crossmodal Cognition Laboratory, Department of Psychology, University of Bath, Bath, UK*

### *Edited by:*

*Clare Jonas, University of East London, UK*

### *Reviewed by:*

*Malika Auvray, LIMSI-CNRS, France Thomas David Wright, University of Sussex, UK*

### *\*Correspondence:*

*Michael J. Proulx, Crossmodal Cognition Laboratory, Department of Psychology, University of Bath, 2 South, Bath BA2 7AY, UK e-mail: m.j.proulx@bath.ac.uk*

Sensory substitution devices (SSDs) aim to compensate for the loss of a sensory modality, typically vision, by converting information from the lost modality into stimuli in a remaining modality. "The vOICe" is a visual-to-auditory SSD which encodes images taken by a camera worn by the user into "soundscapes" such that experienced users can extract information about their surroundings. Here we investigated how much detail was resolvable during the early induction stages by testing the acuity of blindfolded sighted, naïve vOICe users. Initial performance was well above chance. Participants who took the test twice as a form of minimal training showed a marked improvement on the second test. Acuity was slightly but not significantly impaired when participants wore a camera and judged letter orientations "live". A positive correlation was found between participants' musical training and their acuity. The relationship between auditory expertise via musical training and the lack of a relationship with visual imagery, suggests that early use of a SSD draws primarily on the mechanisms of the sensory modality being used rather than the one being substituted. If vision is lost, audition represents the sensory channel of highest bandwidth of those remaining. The level of acuity found here, and the fact it was achieved with very little experience in sensory substitution by naïve users is promising.

**Keywords: sensory substitution, blindness, acuity, synthetic synesthesia, visual, auditory**

# **INTRODUCTION**

Do we see with the eyes or with the brain? Is vision a discrete form of perception, distinct from others such as audition and touch? Is it possible for those who have lost their eyesight or have been born without vision to experience visual sensation or perception? Questions such as these have occupied the minds of philosophers and scientists for centuries (Morgan, 1977) and now lie at the heart of modern cognitive neuroscience. Today, with current experimental techniques and technologies including high-resolution functional brain imaging and devices which purport to transduce information from a lost sensory modality into the brain via another modality, inroads are being made toward finding answers to these questions. Sensory substitution devices (SSDs) aim to compensate for the loss of a sensory modality, typically vision, by converting information from the lost modality into stimuli in a remaining modality (Bach-y-Rita and Kercel, 2003). Here we utilized sensory substitution to examine how the very first stages of learning to "see with sound" occurs, and the quality of the information transfer from vision to audition as assessed with a test of acuity. A more complete understanding of the way in which this occurs may assist in the development of such devices that not only replicate lost sensory functionality, particularly in the blind, but along with research on synesthesia and multisensory processing, also call into question our notion of sensory modalities as functionally discrete, non-overlapping entities.

### **CHANGES FOLLOWING SENSORY LOSS**

Major neuroplastic changes can occur in a brain that is undamaged but loses input from a sensory modality. Multisensory processes in which cues from multiple modalities unite to form a percept may also include a degree of redundancy: an object's shape can be discerned by the hands and eyes simultaneously or separately; the eyes and ears can be used in concert to determine the direction of a physical sound source more accurately than from sound alone. It may be this redundancy which helps the brain to compensate for sensory loss by enhancement of function of the remaining senses (Merabet and Pascual-Leone, 2010).

Blind individuals, particularly those born without sight or those who lost sight early in life often show superior performance in other modalities, including finer pitch discrimination and sound localization, more accurate tactile discrimination, better speech discrimination, and verbal recall (Merabet et al., 2005; Pasqualotto and Proulx, 2012; Pasqualotto et al., 2013). Blind individuals lack normal visual input to their occipital cortices, but brain imaging studies have shown that this area is nevertheless active during a number of tasks, including Braille reading, auditory localization tasks, speech comprehension, and verbal recall (Merabet et al., 2005).

Sensory loss need not have occurred early in life however, and changes can occur rapidly in adults following sensory deprivation. In one study, participants were blindfolded 24 h per day for 5 days and given intensive training in tactile and spatial discrimination tasks. Participants experienced visual hallucinations soon after blindfolding and functional magnetic resonance imaging (fMRI) scans showed occipital cortex activation when fingertips were stimulated, with primary and secondary visual cortices becoming increasingly active over the blindfolded period (Pascual-Leone and Hamilton, 2001). Tactile discrimination skills learnt during the experiment were disrupted when participants' occipital cortices were subjected to repetitive transcranial magnetic stimulation (rTMS). After the blindfold period, participants' brains were scanned again and occipital cortex activity linked to tactile stimulation was absent. Because the changes seen in this experiment manifested and then reversed so quickly, it cannot have been that new neuronal connections were established. Instead, existing connections between somatosensory, auditory, and visual cortices were "unmasked" when input from the eyes temporarily ceased. The authors of this study suggest that neuroplasticity in response to sensory loss is a two-stage process: rapid unmasking of existing cortico-cortical connections followed by slower and more permanent formation of new neuronal connections (Pascual-Leone and Hamilton, 2001).

# **SENSORY SUBSTITUTION**

Sensory substitution is the use of one modality (the substituting modality) to take the place of another (the substituted modality). The concept has been construed by some in a very broad sense to include, for example, reading, in which vision (the written word) takes the place of audition (the spoken word) (Bach-y-Rita and Kercel, 2003). More commonly, however, the term is used to refer to a means to allow a person who has suffered sensory loss to make use of their remaining senses to perform functions normally carried out using the lost sense. An obvious and widely used example of this is Braille, in which tactile perception via the fingers substitutes for vision (or, arguably, audition), allowing blind people to read. This system only replaces a specific aspect of a modality however, namely language; substitution on a general level represents a much greater technical challenge. This challenge has been met over the past four decades by a variety of systems and devices, most of which have been designed to replace vision, either by touch or audition.

# **AUDITORY-VISUAL SENSORY SUBSTITUTION**

In tactile-visual sensory substitution (TVSS) systems, the skin or tongue functions as an analog of the retina (Bach-y-Rita et al., 1969). However, by comparison it is very crude and low-resolution. Kokjer (1987) estimated the informational capacity of the human fingertip to be in the order of 100 bps. The eye, by comparison, has been estimated to deliver around 4*.*<sup>3</sup> <sup>×</sup> <sup>10</sup><sup>6</sup> bps (Jacobson, 1951), some four orders of magnitude greater bandwidth. The ear falls between these two limits, its capacity has been estimated at around 104 bps (Jacobson, 1950). So although parallels between the auditory and visual systems are not obvious in the way that the skin/retina analog is, the ear has the potential to provide a higher-throughput means of directing visual information to the brain than the skin.

The first general-purpose auditory-visual sensory substitution (AVSS) system was developed by Meijer (1992). It is known as "The vOICe" and is the system used in the present study. The vOICe converts images captured by a camera into "soundscapes" delivered to the user through headphones at a default rate of one soundscape per second. Each soundscape is a left to right scan of the visual scene with frequency representing the image's vertical axis and loudness representing brightness (these mappings are not arbitrary, see Evans and Treisman, 2010). The user therefore experiences a series of "snapshots" passing from the left to the right ear. Other AVSS devices have been developed: one which uses a similar encoding protocol as The vOICe but converts scenes into images resembling line drawings and produces a more "musical" output (Cronly-Dillon et al., 1999, 2000); another, the Prosthesis for Substitution of Vision by Audition (PVSA), does not scan the visual field but lets frequency increase both from bottom to top and from left to right of the captured image, using a higher density of auditory "pixels" in the center of the image to simulate the fovea (Capelle et al., 1998); and a third, the Vibe, also does not scan the visual field, instead dividing it into several "receptive fields" which are presented simultaneously, their position encoded by frequency and left-right audio channel balance (Auvray et al., 2005; Hanneton et al., 2010).

As with TVSS devices, users of AVSS systems report distal attribution (Auvray et al., 2005). Users have been shown to recognize patterns (Arno et al., 2001), recognize and locate objects in 3D space, including placing previously unseen objects into categories, such as "plant" or "boot" (Auvray et al., 2007; Merabet et al., 2009). One expert late-blind user of The vOICe, P.F., has provided repeated, detailed descriptions of her experiences which, she claims, have gradually improved and become more like vision. Depth perception, smooth movement (as opposed to 1 Hz "snapshots") and even experience of colors emerged with continued use of the device for P.F., suggesting that her brain had been gradually adapting to more efficiently process this novel kind of auditory information (Ward and Meijer, 2010).

# **ACUITY IN SENSORY SUBSTITUTION SYSTEMS**

An important factor in the usefulness of a system in which vision is the substituted modality is the limit on detail resolvable by the user. Finding this limit can be achieved in much the same way that visual acuity is conventionally measured. Some studies have measured acuity through indirect means, by assessing the ability of participants to either localize or recognize objects with AVSS devices (Auvray et al., 2007; Proulx et al., 2008; Brown et al., 2011). The study by Proulx et al. (2008) even used an ophthalmic perimeter, commonly used to map the visual field, as a means of assessing the speed and accuracy of spatial localization using sensory substitution. Other studies determined the acuity limits of TVSS devices directly. The acuity limit of legal blindness in the United States is 20/200; that is, a person with this level of acuity can read an eye chart located 20 feet away as well as a person with normal vision would read the same eye chart were it 200 feet away (Social Security Act. United States Social Security Administration, 2006). Normal vision thus corresponds to an acuity of 20/20.

The translation of visual acuity to sensory substitution is not entirely straightforward as the computation requires consideration of the field of view provided by the device. For example, it might be physically possible to provide 20/20 vision with a SSD through telescopic means. However, if this is accompanied by severe tunnel vision due to a restricted field of view, then the end result is still classified as a severe visual impairment. In fact, the definition of legal blindness in the United States specifies an acuity of 20/200 or less, or a field of view of 20◦ or less. A full explanation and demonstrations of the issues involved in defining acuity for sensory substitution are also available online <sup>1</sup> , but we will summarize the main points here. For our calculations of acuity we will assume a 60◦ field of view for the camera, as we used in the experiments reported here. This is a typical field of view for web-cams and similar devices, (and may, for lack of a suitable standard, serve as a ballpark figure for a practical field of view). The minimum number of pixels required to portray most optotypes used in acuity measurement would be 2–3 pixels horizontally. Assuming 176 horizontal pixels for the camera input, as we also use in our experiments, then every pixel subtends approximately 0.35◦ in width. The smallest discernable optotype then spans about one degree for 3 pixels horizontally (3 × 60◦/176∼=1◦), or 0.7◦ for 2 pixels (2 × 60◦/176). Normal vision under Snellen's definition is the ability to recognize one of the Snellen chart optotypes when it subtends 5 min of arc (Duke-Elder and Abrams, 1970). Functionally, this means that visual acuity under the above conditions is between 8 and 12 times less than that possible with normal human vision. At best, visual acuity could be in the range 20/160– 20/240. The crucial aspect of these calculations for comparisons with reports of visual acuity in the literature is that they are based on a horizontal resolution of 176 pixels for a 60◦ field of view. If the physical resolution of a sensory device provides much less than this, then the maximum visual acuity possible with that device is dramatically reduced for the same field of view.

The first study to assess visual acuity with sensory substitution was conducted in the domain of touch by Sampaio et al. (2001). Sampaio et al., used the Snellen tumbling E paradigm to test blind and sighted naïve participants' performance using a 3 cm<sup>2</sup> <sup>12</sup> <sup>×</sup> <sup>12</sup> electrotactile array or "tongue-display unit (TDU)." Their setup included a camera with a 54◦ horizontal and 40◦ vertical field of view, and its 280 × 180 pixel frames were down-sampled to the 12 × 12 tactile display resolution by averaging adjacent pixels. Judging acuity as performance at or near 100% in letter orientation discrimination they reported that all participants were able to achieve this to an acuity level of 20/860 and that two participants of median performance doubled their acuity after 9 h of training to 20/430. Because the device provided a resolution of 12 pixels horizontally, the actual functional acuity might be far less, with a maximum theoretical acuity of 20/2400 for a 2 pixel wide optotype and a 60◦ field of view, or 20/2160 when calculated for their camera's 54◦ field of view. For example, in the latter case the denominator is calculated as (2 pixels × 54◦/12 electrodes) × (60 min of arc per degree/5 min of arc for normal vision) × 20 for normal vision = 2160.

The second study to assess acuity was conducted by Chebat et al. (2007), who tested a larger sample of early blind and sighted participants on a 4 cm<sup>2</sup> <sup>10</sup> <sup>×</sup> 10-array TDU. After a period of training participants were tested also using the Snellen tumbling E. The criterion for passing each level was 70% correct responses. Acuity scores ranged between 20/1800 and 20/8400 for an estimated 29◦ field of view, and it was found that blind participants were overrepresented at higher acuity scores with 8.4% of sighted and 31.3% of blind participants achieving the highest score. Again, by using the calculations and limitations described earlier, the maximum theoretical acuity for a 10 pixel device such as this would be 20/2880 for a 2 pixel wide optotype and a 60◦ field of view, or 20/1392 when calculated for their 29◦ field of view. The latter is consistent with the range of acuity scores reported by Chebat et al. (2007) for their narrower field of view.

Acuity using The vOICe AVSS device has recently been reported by Striem-Amit et al. (2012) for nine fully blind participants who had already been trained to use the device. Participants were trained for between 55 and 101 h and tested on Snellen tumbling Es. Using a criterion of 60% correct responses, participants' acuity is reported to have varied between 20/200 and 20/600 using a 66◦ field of view camera. The present study was designed to assess a number of additional issues beyond the scope of the study by Striem-Amit et al. First, their study was conducted only with expert users of the SSD who were also blind. It is thus unclear whether the acuity levels achieved reveal the resolution of the device, or rather the compensatory neural plasticity of the blind participants combined with their expertise in using the device. Furthermore, the mechanisms that give rise to the acuity performance are also unclear. To provide a benchmark measure of acuity, we here employed naïve sighted participants without previous experience of the device. Furthermore we tested them under different conditions (static and active use of a camera), and with additional experiments and questionnaires to determine the possible correlates of individual differences in acuity performance.

The present study also used the Snellen tumbling E in two separate experiments: in the first, The vOICe software was used to turn letter images of decreasing size into sound files offline which were played to participants as static soundscapes; in the second, blindfolded participants used a sunglasses-mounted camera and headphones to "read" letters from a screen. Acuity in present tongue-based TVSS devices is limited by the number of electrodes on the array (144 in Sampaio et al., 2001 and 100 in Chebat et al., 2007). The vOICe software, by contrast, produces an equivalent resolution of 11,264 "voicels" or auditory pixels in the default setting. This fact, along with the higher informational capacity of the ear (Jacobson, 1951) suggests that higher acuity scores with audition should be possible than those in the tactile studies cited above (see, e.g., Sampaio et al., 2001; Chebat et al., 2007; Striem-Amit et al., 2012).

As well as assessing the mean acuity of a sample group, the present study also takes an individual differences approach to determine whether any correlations can be found between performance on acuity tests with a SSD and other metrics. It has been shown, for example, that musical training correlates with improved ability to extract information from sound preattentively (Koelsch et al., 1999), and to extract speech from noise (Parbery-Clark et al., 2009). Many of the participants also took part in additional experiments to explore such individual differences. First we assessed whether there was any relationship between acuity and another form of auditory expertise, musical training.

<sup>1</sup>www.seeingwithsound.com/acuity.htm

Their ability to discriminate between similar musical phrases and their pitch discrimination abilities was also tested. This study also considered whether early, naïve use of sensory substitution immediately draws upon the substituted modality (vision) or only the substituting modality (hearing). Work on synesthesia, a crosswiring of the senses where a sound might evoke a visual experience, such as music evoking different colors (Hubbard et al., 2011), suggests that the sensory modalities are not always distinct, independent modules. Certainly one broad goal for work on sensory substitution is to ultimately provide the phenomenological experience of vision in a form of synthetic synesthesia (Proulx and Stoerig, 2006; Proulx, 2010). Along this line of interest, here participants also took a vividness of visual imagery questionnaire (Marks, 1973), as well as a psychophysical test designed by Cui et al. (2007) to correlate with the vividness of imagery reported by participants. If individual differences such as these can be found to correlate with acuity performance they may be useful as proxies to gage a person's likelihood of making successful use of an AVSS device such as The vOICe and to reveal potential mechanisms for such performance. This also assesses whether visual imagery evoked by the device, as a form or synthetic synesthesia (Proulx and Stoerig, 2006; Proulx, 2010), is related to measures of the functional resolution possible with the device.

# **EXPERIMENT 1**

### **PARTICIPANTS**

Adult volunteers without experience with The vOICe took part (4 male, 22 female, mean age 22.6 years, range 19–32 years). All reported normal vision (with corrective lenses in some cases).

### **APPARATUS**

For Experiment 1a Dell Optiplex 760 PC (Intel Core 2 Duo @3 GHz; 3.2 GB RAM; Microsoft Windows XP Professional) ran The vOICe software (Learning Edition v1.91) <sup>2</sup> , with the foveal enlargement mapping disabled. The program was run on "slow motion" setting, images being scanned from left to right producing soundscapes with a duration of 2 s, and in "negative video" mode whereby dark areas correspond to loud sounds and white areas produce no sound. Sennheiser HD555 open-back supraaural headphones were used for all tests involving an auditory component. The program's foveal enlargement option was kept disabled in all experiments.

For Experiment 1b FrACT visual acuity software (Bach, 1996; v3.7.1b, obtained from michaelbach.de/fract/download.html) was used, running the tumbling E experiment. Four orientations and differing sizes of the letter E in black on a white background were shown on an LCD screen with resolution 1440 × 900, and each image was followed by a 200 ms mask. Participants sat 175 cm from the monitor. All instructions and requirements were followed according to the FrACT specifications.

### **MATERIALS**

Digital images of the Snellen E in four orientations (left, right, up, and down, **Figure 1**) and 10 sizes (**Table 1**) were converted by

**Table 1 | Size and acuity values of the optotypes used in each block.**


The vOICe software first into a 176 × 64 pixel resolution and then into soundscapes, with the optotypes set as white to be sonified and the background black and silent. These values were used to calculate the Snellen acuity for each letter size following procedure detailed on The vOICe website (see text footnote 1), assuming a 66◦ camera field of view as used in Experiment 2. Optotypes have also been assigned an estimated "width" in mm assuming a 66◦ camera viewing angle at a distance of 1 m in order to compare results with those of Striem-Amit et al. (2012) (**Table 1**). A questionnaire about the experience of using The vOICe software and any strategies employed by participants to detect optotype direction was used.

# **PROCEDURE**

The concept of The vOICe was explained to participants and they were asked to read an explanation of the image-to-sound conversion protocol and the experimental procedure. They were asked if they understood what they had read and that they consented to taking part in the experiment. They were then asked put on a blindfold and headphones. The first experiment took a total of 40 min per participant.

### *Experiment 1a*

The experiment was conducted as blocks of trials with 12 trials per block. In each trial the soundscape was played to the participant, who had to state which direction they thought the optotype was facing (i.e., the tines of the E, see **Figure 1**). They were allowed to ask for the soundscape to be repeated up to ten times. Optotypes were presented in pseudorandom order with each direction featuring three times. The threshold for passing each block was 9/12 correct optotype directions, with the exception of Block 1

<sup>2</sup>www.seeingwithsound.com

which they could repeat up to five times if they failed to reach the threshold (with different orders of optotypes each time). Thereafter blocks were presented in order until the threshold was not reached at which point the experiment ended and the highest successfully completed block was recorded as the participant's vOICe acuity score.

### *Experiment 1b*

Participants then took the FrACT visual acuity test. The test was explained to them and they were asked to sit 1.75 m away from the computer screen. They called out the direction of each optotype presented which was entered into the computer by the experimenter who sat behind the display and thus did not have exposure to the stimuli. When the test was complete the Snellen fraction displayed on the screen was recorded as their visual acuity score. The test consisted of 30 trials. Finally, participants were asked to fill in a questionnaire about their experience.

### **EXPERIMENT 1A RESULTS**

Participants (*n* = 26) achieved vOICe acuity scores between 20/13965 and 20/1882. Scores of 20/2464 and 20/4682 were achieved by the highest number of participants (**Figure 2**). Median acuity was 20/4682. Thirteen participants completed the first block on the first attempt, nine on the second attempt, three on the third attempt, and one on the fourth attempt. There was a non-significant negative correlation between number of Block 1 attempts and final acuity score (Spearman's rank correlation *rs* = −0*.*37, *p* = 0*.*07).

### **EXPERIMENT 1B RESULTS**

Median visual acuity as measured by the FrACT test was 20/13. No correlation was found between visual acuity and vOICe acuity. Participants who had musical experience achieved higher vOICe acuity scores (Mann–Whitney test, *U* = 19*.*5, *p* = 0*.*03, *r* = 0*.*43).

### **EXPERIMENT 1 QUESTIONNAIRE RESULTS**

Participants were first asked about the experience of using The vOICe for the acuity task: "How would you describe your experience of the task? Can you compare it to a sensory modality/experience." A fifth of the participants described the auditory nature of the tasks (19%); for example, one participant noted its similarity to a music test. A few also noted that it felt like they used a general spatial sense to carry out the task (15%), interestingly described by one as "Like figuring out where you're walking in the dark." Nearly one-third mentioned experiences in the visual modality (30%), such as visualizing or imagining the letter E while trying to complete the task.

Participants were next asked about any strategies used to complete the task: "Did you use a particular strategy to identify the different orientations? Can you describe this strategy?" Half provided very specific, detailed strategies that were used, while half provided just a vague description. All responses were classifiable as having relation to either determining an orientation difference, using visual descriptions, and whether the participants attempted to either memorize the sounds, work out how the sounds related to the image that produced them, or actively tried to imagine the stimuli visually. First, 58% reported a strategy of determining the orientation difference. This is noteworthy considering the task could have been approached as an auditory discrimination task that did not have anything orientation-specific about it. Only 23% reported using an explicitly "visual" strategy, however, and instead spoke of the sounds as having a spatial quality that was implicitly related to the visual image; for example, one participant noted that the sounds had sides, rather than beginnings and ends (emphasis

added) "The pitch change at either *side* of the sound indicated a left or right orientation." The task was not automatically carried out, as 61% of the participants noted they had to deliberately work out the response after listening to the sounds. Another 15% said that they attempted to memorize the sounds and the correct response; a difficult feat considering that the range and modulation of pitch changed as a function of image size, thus making the sounds different at each level of acuity. However 23% attempted to imagine the image of the E that created the sound in order to respond.

## **EXPERIMENT 2**

### **PARTICIPANTS**

A subset of volunteers from Experiment 1 returned for additional experiments (2 male, 15 female, mean age 23 years, range 20–32 years). Procedures were approved by the Queen Mary Research Ethics Committee, and all participants gave informed, written consent.

### **APPARATUS**

"Spy" sunglasses (Mini DVR Spy sunglasses, Camera Audio Video Recorder, Multister, Hong Kong) with an image sensor similar to a webcam built into the bridge were used, in conjunction with The vOICe software, to convert Snellen Es displayed on a 1440 × 900 LCD screen into soundscapes as participants faced the screen. Movement was possible, thus the participants may have centered the stimuli however they preferred.

## **MATERIALS**

### *Experiment 2*

The same optotype soundscapes were used as in Experiment 1. For the part of the experiment in which subjects wore camera glasses, optotypes were displayed on screen in white on black. The same sizes (original, in pixels) as those shown in **Table 1** were used. Participants sat so that the camera glasses were 30.5 cm from the screen. At this distance a Block 1 letter occupied the equivalent area of the camera's field of view as a Block 2 letter occupied of The vOICe's "screen" size (i.e., 141 pixels out of 176 on the horizontal axis). Accordingly, in this part of the experiment the first block represented the same acuity score as Block 2 in **Table 1**, therefore there were nine blocks in this part of the experiment.

### **PROCEDURE**

### *Experiment 2*

The same procedure as Experiment 1 was carried out for the optotype soundscapes, except in this case participants completed all blocks regardless of their performance. Experiments 2, 3, and 4 were all carried out in a single session, approximately 2 months after Experiment 1 was conducted, and took approximately 60–70 min per participant.

### **EXPERIMENT 2 RESULTS**

Participants (*n* = 17) achieved vOICe acuity scores between 20/4682 and 20/737 (based on the first experimental criterion of 9/12 correct responses). A score of 20/2464 was achieved by the highest number of participants (**Figure 3**). Median acuity was 20/2464. Participants who took part in both experiments achieved higher acuity scores in the second test (Wilcoxon signed ranks test, *Z* = −2*.*99, *p* = 0*.*002; **Figure 4**), with the group as a whole improving from an acuity level of 5 on the first test to over level 6 on the second (6.3). Two individuals who had only reached levels 1 and 2 during the first experiment were able to attain levels 5 and 7 in the second.

Using the same experimental criterion as Striem-Amit et al. (2012) of 8/12 correct responses, participants achieved

correct responses) is shown as a function of increasing acuity score.

scores between 20/2464 and 20/737. Scores of 20/1882 and 20/1308 were achieved by the highest number of participants (**Figure 5**). Median acuity was 20/1308 Participants collectively were significantly above the threshold of 8/12 correct responses up to 20/2464 and above chance (4/12) up to 20/737 (**Figure 6**).

Performing a two-way analysis of variance (ANOVA) on ranked acuity scores showed significant main effects of size (*F(*1*,*9*)* = 14*.*3, *p* = 0*.*004) and of orientation (*F(*9*,*9*)* = 6*.*25, *p* = 0*.*03). There appeared to be an interaction between size and orientation such that at large sizes performance was superior on left/right oriented optotypes, and at acuity level

20/1882 performance was superior on up/down oriented opto-types (**Figure 7**), however the interaction effect was not significant.

Participants' (*n* = 15) acuity when tested on the static soundscapes was slightly but not significantly higher than when using camera glasses to view optotypes (Wilcoxon signed ranks test, *Z* = −1*.*56, *p* = 0*.*06; **Figure 8**). Participants were significantly above the 8/12 threshold up to 20/4862 and above chance (4/12) up to 20/737 (**Figure 9**).

# **EXPERIMENT 3**

### **PARTICIPANTS**

The same subset of volunteers in Experiment 2 from Experiment 1 returned for this experiment (2 male, 15 female, mean age 23 years, range 20–32 years). Procedures were approved by the Queen Mary Research Ethics Committee, and all participants gave informed, written consent.

# **APPARATUS**

# *Experiment 3*

MATLAB R2009a (v7.8.0) software with Psychophysics Toolbox extension (Brainard, 1997; Pelli, 1997) was used to run a Stroopstyle psychophysical test "EagleStroop" (Cui et al., 2007; used with permission) in which a 32 ms presentation of a colored background (orange, yellow, purple) was followed by a 32 ms appearance of one of three corresponding color words (in black) or no word. Color words were sometimes congruent with preceding background color. The test was presented on an 1440 × 900 LCD screen, as also used in Experiment 2.

# **PROCEDURE**

### *Experiment 3*

The "EagleStroop" psychophysical test was carried out, consisting of five practice trials in which colored backgrounds and color words appeared for 1000 ms each, followed by 120 recorded trials in which stimuli appeared for 32 ms. Participants entered the initial letter of the color word they thought they had seen by pressing that key on the computer keyboard or pressed the space bar if they saw no word. Scores were automatically recorded by the MATLAB software and the difference between correct responses when color and word were congruent and when incongruent for each participant were calculated.

# **EXPERIMENT 3 RESULTS**

No significant correlations were found between vividness of imagery as measured either by the vividness of visual imagery questionnaire (VVIQ) test or the "EagleStroop" psychophysical test and vOICe acuity; nor was a correlation found between VVIQ scores and the EagleStroop test. On the EagleStroop test there was no difference in performance when color and word were congruent and when they were incongruent across participants although there were differences in individual participants.

# **EXPERIMENT 4 PARTICIPANTS**

The same subset of volunteers in Experiments 2 and 3, originally from Experiment 1, returned for this experiment (2 male, 15 female, mean age 23 years, range 20–32 years). Procedures were approved by the Queen Mary Research Ethics Committee, and all participants gave informed, written consent.

# **APPARATUS**

# *Experiment 4*

Participants' ability to discern pitch and to differentiate between similar musical phrases was carried out using online tests on the Tonometric website<sup>3</sup> . The "adaptive pitch" and "tonedeaf" tests were used.

3www.tonometric.com

### **MATERIALS**

A "Vividness of Visual Imagery Questionnaire" (Marks, 1973) was used.

### **PROCEDURE**

### *Experiment 4*

For the "Tonometric" auditory perception tests, participants wore headphones and used the computer mouse to enter "higher" or "lower" in each trial of the "adaptive pitch" test, and "same" or "different" for the "tonedeaf" test. Their scores were recorded. For the next part of the experiment, the use of camera glasses in conjunction with The vOICe software was explained to participants. They were then asked to put on a blindfold, followed by the camera glasses which were connected to the computer by a USB cable. Headphones were placed on top to keep the glasses fixed in place. They were seated so the image sensor was 30.5 cm from the computer screen and asked to keep their head as still as possible but centering the stimuli as they preferred (**Figure 10**). Optotypes were then displayed on the screen and the same protocol was used as with the previous part of the experiment in which soundscapes were played, except that as The vOICe software was converting images to sound "live," the soundscapes played repeatedly until the participant stated the direction of the optotype. Finally, participants were then given a copy of the VVIQ and instructions on how to complete it.

### **EXPERIMENT 4 RESULTS**

On the Tonometric tests of auditory perception participants' mean pitch discernment was 8.17 Hz at 500 Hz (SD = 9*.*00, *n* = 16). Their mean score on the "tonedeaf" test was 72.73% correct (SD = 8*.*86, *n* = 17). No significant correlations were found between vOICe acuity score (static letter soundscapes) and either of these tests.

# **DISCUSSION**

The results of this study demonstrate firstly that very little training or explanation is required to carry out tests of this nature: every participant understood the concept of The vOICe, even if it was an entirely novel one to them. Half of the participants in Experiment 1 (13 of 26) completed the first block without needing to repeat it and no participant failed this initial training session. There was no link between performance on The vOICe acuity test and participants' actual visual acuity.

The repeat of the original vOICe test (Experiment 2), although it took place around 2 months after the first and without further training (Experiment 1), revealed a significant improvement in performance amongst participants who returned; there is no reason to suspect any link between initial performance and likelihood of returning. Returning participants needed little reminding of the procedure, and only one of the 17 required more than one attempt at the first block. Furthermore, all participants achieved a score of 20/4682 or greater (**Figure 3**), in contrast to the first experiment in which all scores below 20/4682 were represented by at least one participant (using the same criterion of 9 correct trials out of 12; **Figure 2**). The final acuity category of 20/408 appears to represent the upper limit of performance for all participants with scores in both iterations of this test no better than chance (**Figure 4**). Given that the optotypes were reduced to 5 pixels in width this limitation is likely explained by the physical representation provided being a sub-sampling of the original image. This makes it even more impressive that some participants were able to discriminate

**soundscape produced from the camera's view of the optotype displayed on screen.**

the orientation of the stimulus with only two vertical pixels being sonified. It is interesting to compare this to the study by Sampaio et al. (2001), where one blind and one sighted participant each received 9 h of training in using the TDU; they both doubled the acuity level after that training. Here, without additional training and a 2 month gap, we found that two participants went from basic level (1 and 2) performance in the first attempt, improved to higher levels (5 and 7, respectively) in the second attempt.

These results are comparable with tongue-based tactile acuity tests carried out on 15 blind and 25 sighted participants by Chebat et al. (2007). Using a criterion of 70% correct responses per acuity category, they reported acuity scores ranging between 20/1800 and 20/8400 for a 29◦ field of view, with most sighted participants scoring 20/3600. As noted in the Introduction, if these reported scores are corrected for the limited field of view provided, then the maximum acuity of 20/1800 would be calculated as approximately 20/3600 in a manner most comparable to the scores with The vOICe reported here where we found a maximum of 20/408 for a 60◦ field of view. The present study's slightly more stringent criterion of 75% correct saw most participants scoring 20/2464 or 20/4682 in Experiment 1. An earlier study (Sampaio et al., 2001) found considerably higher acuity among blind and sighted participants than the later study by Chebat et al. (2007), but from the estimates for maximum theoretical acuity we suspect errors in those early results. Acuity scores in the present study were not as high as those reported by Striem-Amit et al. (2012) although there is some overlap: the lowest score found in their

study was 20/1000 and the highest was 20/200, which equates to optotypes of width 73 mm and 15 mm respectively at a distance of 1 m. One possible explanation is that the limitation with the smallest optotype presented at 5 pixels by 2 pixels may not have been present in that study. Moreover, participants in that study had received a great deal of training (over 100 h in some cases), and thus perceptual learning processes might further account for the performance difference (Proulx et al., 2013). Although Stiem-Amit et al. do not report any correlation between training time and final performance, the difference in performance between the first and second acuity test in the present study shows a clear effect of training, which in this case amounted to *<*1% of that received by participants in the study by Striem-Amit et al. (2012). Also, all of their participants were blind, all but one of them congenitally, and this may also have been a factor in their high performance (see Chebat et al., 2007).

Interestingly, the orientation of the optotypes affected performance and did so differentially depending on the size of the optotype. When optotypes were large participants found it easier to discriminate left and right; as they became smaller, up/down discriminations briefly became easier even with the greater reduction in the number of pixels representing the vertical dimension (see **Figure 7**). The difference between a left and right-facing E is in the temporal dimension; when the E is facing up or down, differences are distinguishable by pitch. Because this pitch difference continues for the time corresponding to the entire width of the E, it is presumably easier to detect when optotypes are smaller, whereas the difference between left and right exists for only one fifth of this time. This is a property of The vOICe and is in no way comparable with TVSS devices. It would be very interesting for future studies to go further in determining whether differences exist between these temporal and pitch-based components of auditory acuity, both between and within individuals. If significant differences are found within individuals, this may be used as a basis for tailoring The vOICe to an individual user's own capabilities in these two separate abilities. Furthermore, it would be interesting to see whether this differential ease-of-discernment bias continues to hold as users become more proficient with training. Alternatively, it may be possible to eliminate this bias by using other optotypes; for example, Snellen Es angled at 45◦ or Landolt Cs (see Bach, 1996). This would also provide a test of how well the acuity of the device generalizes to other measures and features.

There was no statistically significant difference between acuity scores using static soundscapes and using camera glasses to "view" the letters (see **Figures 9** and **10**). Performance using the camera glasses may be expected to be slightly impaired, firstly because this was each participant's first attempt at using this equipment and secondly because "live" sonifications will inevitably be "rougher" than those created as static soundscapes, due to inconsistencies in lighting, movements made by participants (although they were instructed to remain as still as possible). However the use of a live camera feed should also provide superior resolution by allowing a dynamic sub-pixel displacement with the camera view, as well as a representation of the stimulus via different frequency ranges. The anti-aliasing of the image due to the representation in pixels would change with movement and perhaps provide a clearer view of the optotype. Also it might be easier to discriminate spectral differences in the higher frequency range than in the lower range, and moving the camera would allow for such sampling by elevation.

A link was found in Experiment 1b between participants' reported musical experience and their vOICe acuity: those with experience attained higher scores. An attempt was made to further subdivide participants into those with some experience and those with extensive experience but significance was lost, which may have been due to the small sizes of these resultant categories. However, there were no significant correlations between vOICe acuity and either pitch discrimination or the ability to discriminate between two similar musical phrases. It might tentatively be inferred therefore, that it is musical training, or experience in fine sound discrimination, that makes a difference in this test of auditory acuity, rather than aptitude. In fact most reported some experience, although without more detailed questioning it is difficult to quantify the degree of their training, so results should be treated with caution. They do accord with previous studies linking musical experience with improved auditory skills (Koelsch et al., 1999; Parbery-Clark et al., 2009).

There was no link between participants' subjective assessment of the vividness of their visual imagery and their vOICe acuity scores. A major drawback with using a questionnaire to measure this is inter-subject comparability, due to the highly subjective nature of the measure. A given individual may believe her visual imagination to be very vivid, perhaps because she can remember specific details of a familiar scene, but has no reliable way of comparing it with that of another who may actually imagine that scene in much more visual-phenomenological way. Cui et al. (2007) reported a correlation between subjective vividness of imagery and performance on their Stroop-style psychophysical

# **REFERENCES**


Scadden, L. (1969). Vision substitution by tactile image projection. *Nature* 221, 963–964. doi:10.1038/ 221963a0


test using a sample size of 8 and concluded that it could be used as an objective measure of vividness of imagery. The present study failed to replicate this result using a sample size of 15, calling that conclusion into question.

The approach of assessing the perceptual resolution, or acuity, of sensory substitution suggests that such assessment might be made in other areas of research. For example, the resolution at which attention operates has become a recent area of interest (He et al., 1996, 1997; Intriligator and Cavanagh, 2001; Moore et al., 2008). Such an approach can determine the precision with which sensory information can be processed by the different modalities, and for different functions. Measures of resolution such as acuity might also provide a benchmark measure for comparing sensory substitution algorithms, sensory modalities, and also might predict individual differences in sensory substitution acquisition. Considering that some long-term users of sensory substitution experience a form of synthetic synesthesia (Ward and Meijer, 2010), it might also be possible to assess the resolution and precision of synesthetic experiences. For example, when a sound evokes the perception of color for a synesthete, what is the spatial extent of that experience? How constrained is the experience by color, size, and shape? The investigation of the resolution of perceptual experience can provide important information about the nature of information processing across and within sensory modalities, and the further exploration of this issue for sensory substitution and other areas of perception will help to reveal the underlying mechanisms that allow one to see what is heard.

### **ACKNOWLEDGMENTS**

The authors thank the reviewers Malika Auvray and Thomas Wright for their helpful suggestions, and support from EPSRC (EP/J017205/1) to Michael J. Proulx.

acuity of the tongue in early blind individuals. *Neuroreport* 18, 1901–1904. doi:10.1097/WNR. 0b013e3282f2a63


and refraction," in *System of Ophthalmology*, Vol. V, ed W. S. Duke-Elder (London: Henry Kimpton).


152–177. doi:10.1111/j.1748- 6653.2011.02014.x


20, 132–138. doi:10.1097/ WNR.0b013e32832104dc


Visual experience facilitates allocentric spatial representation. *Behav. Brain Res.* 236, 175–179. doi:10.1016/j.bbr.2012.08.042


substitution. *PLoS ONE* 7:e33136. doi:10.1371/journal.pone.0033136


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 January 2013; accepted: 21 May 2013; published online: 18 June 2013.*

*Citation: Haigh A, Brown DJ, Meijer P and Proulx MJ (2013) How well do you see what you hear? The acuity of visual-to-auditory sensory substitution. Front. Psychol. 4:330. doi: 10.3389/fpsyg. 2013.00330*

*This article was submitted to Frontiers in Cognitive Science, a specialty of Frontiers in Psychology.*

*Copyright © 2013 Haigh, Brown, Meijer and Proulx. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.*

# Combined structural and functional imaging reveals cortical deactivations in grapheme-color synaesthesia

### *Erik O'Hanlon1, Fiona N. Newell <sup>1</sup> and Kevin J. Mitchell <sup>2</sup> \**

*<sup>1</sup> School of Psychology and Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland*

*<sup>2</sup> Smurfit Institute of Genetics and Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland*

### *Edited by:*

*Michael Banissy, Goldsmiths University of London, UK*

### *Reviewed by:*

*H. Steven Scholte, University of Amsterdam, Netherlands Jürgen Hänggi, University of Zurich, Switzerland*

### *\*Correspondence:*

*Kevin J. Mitchell, Smurfit Institute of Genetics, Trinity College Dublin, Dublin 2, Ireland e-mail: kevin.mitchell@tcd.ie*

Synaesthesia is a heritable condition in which particular stimuli generate specific and consistent sensory percepts or associations in another modality or processing stream. Functional neuroimaging studies have identified potential correlates of these experiences, including, in some but not all cases, the hyperactivation of visuotemporal areas and of parietal areas thought to be involved in perceptual binding. Structural studies have identified a similarly variable spectrum of differences between synaesthetes and controls. However, it remains unclear the extent to which these neural correlates reflect the synaesthetic experience itself or additional phenotypes associated with the condition. Here, we acquired both structural and functional neuroimaging data comparing thirteen grapheme-color synaesthetes with eleven non-synaesthetes. Using voxel-based morphometry and diffusion tensor imaging, we identify a number of clusters of increased volume of gray matter, of white matter or of increased fractional anisotropy in synaesthetes vs. controls. To assess the possible involvement of these areas in the synaesthetic experience, we used nine areas of increased gray matter volume as regions of interest in an fMRI experiment that characterized the contrast in response to stimuli which induced synaesthesia (i.e., letters) vs. those which did not (non-meaningful symbols). Four of these areas showed sensitivity to this contrast in synaesthetes but not controls. Unexpectedly, in two of them, in left lateral occipital cortex and in postcentral gyrus, the letter stimuli produced a strong negative BOLD signal in synaesthetes. An additional whole-brain fMRI analysis identified 14 areas, three of which were driven mainly by a negative BOLD response to letters in synaesthetes. Our findings suggest that cortical deactivations may be involved in the conscious experience of internally generated synaesthetic percepts.

**Keywords: VBM, fMRI, diffusion, DTI, structural, negative BOLD, deactivation, synesthesia**

# **INTRODUCTION**

Synaesthesia is a heritable condition in which particular stimuli generate specific and consistent sensory percepts or associations in another modality or processing stream (Galton, 1883; Cytowic, 1989/2002; Baron-Cohen et al., 1993). Many different forms exist, including colored letters or words (grapheme- or linguistic-color synaesthesia), "colored hearing," words to taste, tastes to shapes, music to color or shapes, the association of numbers or calendar units with spatial locations and many others (Rich and Mattingley, 2002; Ward, 2013). The condition is quite common, with between 1 and 4% of the population estimated to have the condition (Simner et al., 2006; Simner, 2012).

Though originally defined as a cross-sensory phenomenon, many cases involve cognitive or higher-level conceptual inducers and/or concurrents (Barnett et al., 2008a; Simner, 2012). Synaesthesia may thus, be better conceptualized as the association of additional attributes into the schema of the inducing object (Mitchell, 2011). The synaesthetic experience is characterized by conscious awareness of the concurrent, either as a vivid sensory percept—perceived externally (for "projector" synaesthetes) or "in the mind's eye"—or as an integral attribute brought to mind by the inducing stimulus (in the way that yellow is brought to mind by thinking of a banana) (for "associator" synaesthetes).

The mechanism driving these additional percepts or associations is unknown. In theory, it could involve cross-activation from a cortical area representing the inducing stimulus to one representing the concurrent percept or association. This crossactivation could be mediated by direct connections (Hubbard et al., 2005; Hubbard, 2007a) or indirectly, via an additional area or areas, possibly through feedback connections (Grossenbacher and Lovelace, 2001; Ward and Mattingley, 2006; Neufeld et al., 2012). Alternatively, the synaestheic experience could involve "hyper-binding" between cortical areas (Weiss and Fink, 2009; Van Leeuwen et al., 2010; Rouw et al., 2011), where some mechanism such as the synchronization of cortical oscillations drives the co-activation and thus, the mental association of patterns

**Abbreviations:** DTI, diffusion tensor imaging; FA, fractional anisotropy; GM, gray matter; VBM, voxel based morphometry; WM, white matter.

of activity representing the inducer and concurrent. While such synchronization may indeed be required it seems insufficient to explain the arbitrary, idiosyncratic, and stable nature of the synaesthetic associations—for example, for A to be bound to olive green, the representation of olive green would still have to be activated in the first place. An integrated model proposes that the synaesthetic experience may need both cross-activation and perceptual binding in order to engage frontal areas required for conscious awareness (Dehaene and Changeux, 2011; Hubbard et al., 2011).

A number of functional and structural neuroimaging experiments have been performed to try to define the neural correlates of synaesthetic experiences and to characterize structural differences associated with the condition. Functional magnetic resonance imaging (fMRI) and electroencephalography studies have provided some insights into the neural basis of synaesthesia but their findings are quite variable (Paulesu et al., 1995; Aleman et al., 2001; Nunn et al., 2002; Hubbard et al., 2005; Weiss et al., 2005; Gray et al., 2006; Rich et al., 2006; Sperling et al., 2006; Steven et al., 2006; Beeli et al., 2008; Barnett et al., 2008b; Goller et al., 2009; Brang et al., 2010). Many fMRI studies have found some anomalous activation of visual areas in response to the presentation of the "inducer"—either aurally presented sounds or visually presented achromatic graphemes. However, in addition to this major conclusion, equally remarkable is the variability of findings, even across studies that investigated the same form of (grapheme-color) synaesthesia. Several studies (Nunn et al., 2002; Hubbard et al., 2005; Sperling et al., 2006; Van Leeuwen et al., 2010) have found extra activation in the region of visual area V4, for example—a region involved in color perception (Lueck et al., 1989; Mckeefry and Zeki, 1997)—but others have not observed this and have seen activation or functional connectivity differences in other visual areas (Aleman et al., 2001; Rich et al., 2006) or in other areas, such as parietal cortex (Weiss et al., 2005; Van Leeuwen et al., 2010; Neufeld et al., 2012). Others have observed no additional activation correlating with the synaesthetic experience (Hupe et al., 2012). A positron emission tomography (PET) study also found, in addition to some areas of extra activation in colored-hearing synaesthetes, greater cortical *deactivation* in other areas in response to spoken words which induced a synaesthetic experience of color. These differential effects were induced selectively by words but not tones, in synaesthetes but not controls (Paulesu et al., 1995).

Phenotypic heterogeneity, including between "projector" and "associator" synaesthetes may explain some of the variation in these results (Hubbard et al., 2005; Rouw and Scholte, 2010). Nevertheless, a simple model of excess cross-activation between highly restricted cortical areas seems too minimal to accommodate all these findings. Rather, these findings suggest that differences in connectivity—either due to structural or functional changes—may be quite extensive in the brains of synaesthetes, a hypothesis which is supported by structural neuroimaging studies.

Several studies have identified structural differences in the brains of synaesthetes compared to controls (Rouw and Scholte, 2007, 2010; Hanggi et al., 2008; Jancke et al., 2009; Weiss and Fink, 2009; Banissy et al., 2012b). In almost all cases, synaesthetes showed greater volumes of areas of gray or white matter or greater fractional anisotropy (FA) within certain white matter tracts than controls [see Banissy et al. (2012b) for an exception]. Some of these differences are in the general region of visual areas thought to be involved in the synaesthetic experience but others are more widespread, in parietal or even frontal regions. A recent study analyzed global connectivity patterns in the brains of synaesthetes, using networks derived from correlations in cortical thickness (Hanggi et al., 2011). The global network topology was significantly different between synaesthetes and controls, with synaesthetes showing increased clustering, suggesting global hyperconnectivity. The differences driving these effects were widespread and not confined to areas hypothesized to be involved in the grapheme-color synaesthetic experience itself. Widespread functional connectivity differences have also been observed in a study using resting-state fMRI (Dovern et al., 2012).

Given the variability of functional imaging findings in synaesthesia, in particular the inconsistency of activation of specific visual areas such as V4, or indeed of any visual areas, we adopted an unbiased approach to look for differences in functional responses in synaesthetes vs. controls. We first carried out a whole-brain volumetric and diffusion tensor imaging (DTI) analysis to identify regions of structural differences between groups of synaesthetes and controls. We then used the clusters of gray matter difference as regions of interest for functional analyses, identifying differential sensitivity to the contrast between letters and non-meaningful characters in synaesthetes compared to controls. In parallel, we conducted a whole-brain functional analysis based on responses to visual stimuli that do or do not induce synaesthetic percepts between synaesthetes and non-synaesthete controls. Surprisingly, several of the functional differences we observe are driven by negative blood oxygen level-dependent (BOLD) response, reflecting unexpected cortical deactivations in this sample of synaesthetes in response to letters.

# **METHODS**

# **PARTICIPANTS**

We recruited 24 right handed participants for our study through local and media advertising, and from the student population at Trinity College Dublin. All participants were female, with a mean age of 38.76 years, and an age range of between 20–58 years. This sample included 13 synaesthetes and 11 non-synaesthete controls and both groups were matched for age (mean ages of 38.1 years, *SE* = 3*.*6 and 37.1 years, *SE* = 4*.*2, for the synaesthete and nonsynaesthete groups, respectively). Synaesthetes were identified by repeated testing for consistency of their letter-color associations over time. The details of the consistency tests used are described in Barnett et al. (2008a). None of our participants reported a history of neurological disorders or psychiatric diagnoses, substance abuse, or were treated at any time with psychiatric medications. All had normal or corrected-to-normal vision, and none reported any auditory deficits. The experimental protocol was approved by the School of Psychology Ethics Committee, Trinity College Dublin and all participants gave written informed consent to participate prior to the study.

# **MRI SCANNING PARAMETERS**

### *Anatomical scanning protocol*

All scanning was conducted on a Philips Achieva 3.0 T scanner, fitted with an eight channel head coil, equipped with a mirror that reflected the display projected on a 640 × 480 panel. This panel was placed behind the participant's head, outside the magnet. The mirror was mounted on the head coil in the participant's line of vision. 180 axial high-resolution T1-weighted anatomical Spoiled Gradient Echo (SPGR) images [Echo Time *(TE)* = 3*.*8 ms, Relaxation Time *(TR)* = 8*.*4 ms, Field Of View (FOV) <sup>=</sup> 230 mm, 162 mm, 0*.*<sup>898</sup> <sup>×</sup> <sup>0</sup>*.*898 mm<sup>2</sup> in-plane resolution, slice thickness 0.9 mm, flip angle α = 8◦] were acquired before the first functional imaging, to allow for subsequent activation localization and spatial normalization of fMRI data and for the purpose of Voxel Based Morphometry (VBM) analyses.

# *DTI scanning protocol*

Diffusion weighted images were obtained using spin-echo, echoplanar imaging (SE EPI) pulse sequence (*TE* = 52 ms, *TR* = 11*,*260 ms, flip angle α = 90◦, FOV = 224 mm, 149.7 mm). 60 axial slices were acquired with an in-plane resolution of 1*.*75 × <sup>1</sup>*.*75 mm2, a slice thickness 2.5 mm, and a gap <sup>=</sup> <sup>0</sup>*.*3 mm. Data with a b-value <sup>=</sup> 800 s mm−<sup>2</sup> and 15 non-collinear gradient directions was collected. The start of each series of directions was preceded by the acquisition of a non-diffusion-weighted volume (*b* = 0) for the purpose of image registration and motion correction.

# *fMRI scanning protocol*

The task was preceded by approximately 20 min of standard scout images, (including shimming to reduce the EPI image artifacts), and SPGR structural acquisitions. Thirty-two, non-contiguous (10% gap) 3.5 mm axial slices covering the entire brain were collected using a T2∗ weighted echo-planar imaging sequence (*TE* = 35 ms, *TR* = 2000 ms, flip angle α = 90◦, FOV = 224 mm, 122.85 mm, 64 <sup>×</sup> 64 matrix size, 1*.*<sup>75</sup> <sup>×</sup> <sup>1</sup>*.*75 mm<sup>2</sup> in-plane resolution). Imaging used a parallel SENSitivity Encoding (SENSE) approach with a reduction factor = 2.

# **MRI DATA ANALYSIS PROTOCOLS**

# *Image pre-processing*

The MR images were collected in Philips PAR and REC format and were converted to NIFTI file format for the purposes of VBM and diffusion tensor image analyses including tractography. Data acquired for the purpose of functional MRI analysis was converted to AFNI HEAD and BRIK file format and analyzed using the AFNI software tools (http://afni*.*nimh*.*nih*.*gov).

# *Whole-brain structural analysis using FSL-VBM*

To investigate possible structural brain differences between synaesthetes and non-synaesthetes a voxel-wise whole brain "optimized" VBM-style analysis was performed using FMRIB FSL VBM tools (Smith et al., 2004). VBM is a voxel-wise automated analysis technique performed on high resolution structural images to investigate differences in local concentrations or volumes (with the inclusion of a modulation step) of gray and white matter [see Ashburner and Friston (2000); Good et al. (2001) for detailed descriptions of the standard and optimized VBM methods]. Briefly, structural images were brain-extracted using BET (Smith, 2002) and segmented before being registered to the MNI 152 standard space using non-linear registration (Andersson et al., 2007b). The resulting images were averaged and flipped along the x-axis to create a left-right symmetric, study-specific gray matter, white matter, and cerebrospinal fluid templates. Second, all native gray and white matter images were non-linearly registered to these study-specific templates and "modulated" to correct for local expansion (or contraction) due to the non-linear component of the spatial transformation. The modulated gray and white matter images were then smoothed with an isotropic Gaussian kernel with a sigma of 2.5 mm (5.75 FWHM).

# *VBM statistical analysis*

MRICron was used for the purpose of voxelwise non-parametric statistical tests featuring Brunner Munzel tests for the purpose of group comparisons and correction for multiple comparisons using False Discovery Rate (FDR). These were conducted for the purpose of group comparisons and the identification of global gray and white matter volume differences. Significant voxels passed a voxelwise statistical FDR threshold of *p* = 0*.*01 corrected and a minimum cluster size threshhold of 10 mm3.

# *Whole-brain DTI white matter analysis using FSL*

Data analysis was performed with FSL. Pre-processing featured eddy current and movement correction. Voxelwise statistical analysis of the FA data was carried out using (TBSS) Tract-Based Spatial Statistics, (Smith et al., 2006), which is part of FSL (Smith et al., 2004). First, FA images were created by fitting a tensor model to the raw diffusion data using FDT, and then brainextracted using BET (Smith, 2002). All participants' FA data were then aligned into a common space using the non-linear registration tool FNIRT (Andersson et al., 2007a,b), which uses a b-spline representation of the registration warp field (Rueckert et al., 1999). Next, the mean FA image was created and thinned to create a mean FA skeleton, which represents the centers of all tracts common to the group. Each participant's aligned FA data was then projected onto this skeleton and the resulting data fed into voxelwise, cross-subject statistics.

# *DTI statistical analysis*

MRICron was used for the purpose of voxelwise non-parametric statistical tests featuring Brunner Munzel tests for the purpose of group comparisons and correction for multiple comparisons using FDR. Significant voxels passed a voxelwise statistical threshold of *p* ≤ 0*.*01 with a cluster size criterion based on the skeletonized images of 5 mm3.

# *Structure/function region of interest (ROI) anlayses (GM VBM-derived).*

The gray matter VBM results, (ROIs), showing significant volume differences between synaesthetes and non synaesthete controls were used as mask regions and applied to the functional MRI activation measures for each of the four independent fMRI stimulus conditions (color, achromatic, letter and non meaningful character). Separate 2 × 2 mixed ANOVAs were conducted based on responses to each of the stimulus conditions (either letters vs. non-letters or colors vs. achromatic) with group (synaesthetes or controls) as a between-subjects factor and each of the stimulus conditions as within-subjects factors to examine a main effect for group, for condition and possible group × stimulus condition interactions. *Post-hoc* analyses were carried out to determine the directions of any effects using IBM SPSS version 19 statistical software.

# **EXPERIMENTAL DESIGN FOR fMRI TEST**

Two independent functional MRI tasks were employed to assess the BOLD response to chromatic or achromatic check patterns and graphemes which were meaningful or "non-meaningful" letters. Our grapheme stimulus set were adopted from those previously described by Pesenti et al. (2000). Stimulus presentations were projected onto a viewing screen behind the magnet bore and viewed by the participants via a mirror attached to the head coil, The fMRI sessions consisted of two conditions, namely the Color Images session and the Grapheme Images session. Each session featured two runs of 15 blocks (each block lasting 30 s) with each run containing equal quantities of three conditions, namely rest, chromatic check patterns and achromatic check patterns and rest, letters and non-meaningful characters, respectively. The presentation order of the condition blocks was pseudo randomized across participants. The images were presented at a rate of 0.5 Hz and alternated with a gray background image to reduce after-image effects (**Figures 1**, **2**). Each gray background image for both interstimulus and rest condition blocks featured a fixation cross at the center, upon which the participant was instructed to gaze.

# *Color images session*

BOLD activations were acquired whilst the participant viewed chromatic and achromatic check pattern stimuli. The chromatic blocks consisted of images of check patterns, viewed in either red/green or blue/yellow combinations. Each check pattern featured a fixation cross at the center, upon which the participant was instructed to gaze. In order to ensure the participant viewed the center of the images, they were instructed to respond whenever a change occurred to the shape of the fixation cross (i.e., from the normal cross to a star shape). Participants responded via a button-press response mechanism was utilized to avoid head motion associated with verbal responses. The change in fixation cross occurred at a rate of four times per block in a pseudo random order used to avoid predictability.

# *Grapheme images session*

Changes in BOLD activations were acquired whilst the participants viewed images of graphemes (meaningful letters and non-meaningful characters which shared many of the same visual features as the meaningful letters). As before in the color stimulus presentations, each gray background image for both inter-stimulus and rest condition blocks featured a fixation cross at the center, upon which the participant was instructed to gaze. Participants were required to respond when the letters or non-meaningful characters appeared in an "italic" font format. The change in letter font occurred at a rate of four times per block in a pseudo random presentation to avoid predictability.

# **fMRI DATA ANALYSIS**

The fMRI data were analyzed using the AFNI software tools (http://afni*.*nimh*.*nih*.*gov) (Cox, 1996). Following image reconstruction, both runs for each session, (i.e., color and grapheme stimuli) were concatenated and motion corrected using 3-D volume registration (least-squares alignment of three translational and three rotational parameters). Noise related artifacts outside the brain were also removed using edge detection techniques.

A block analysis was performed to estimate the activation for the chromatic and achromatic conditions (for the color task) and for the letters and non-meaningful characters conditions (for the grapheme task) separately. The on–off block regressors were convolved with a standard haemodynamic response to accommodate the lag time of the BOLD response. Multiple regression analyses were then used to determine the average level of BOLD activation relative to the rest state periods (baseline). The baseline activations were also derived by averaging the rest periods in each block over both runs for each separate task.

The percentage change map (block activation) voxels were resampled to 1 × 1× 1 mm voxel resolution, aligned to their T1 anatomy image and then warped into Talairach standard space and spatially blurred with a 3-mm isotropic rms Gaussian kernel.

# *fMRI "whole-brain" voxelwise analysis*

Separate whole-brain voxelwise analyses, based on each of the stimulus conditions (i.e., color/achromatic and letter/character contrasts) were conducted based on a Two-Way mixed ANOVA with group as the between-subject factor and stimulus condition as the within-subject factor. Each 2 × 2 ANOVA was used to assess for main effects of group (synaesthete or controls), or stimulus condition (color/achromatic or letters/characters) and for any Group × Condition interactions. Significant voxels passed a voxelwise statistical threshold (*t* = 17*.*19, *p <* 0*.*0005, *N* = 22) and were required to be part of a minimum cluster of 134 mm<sup>3</sup> of contintiguous significant voxels. Thresholding was determined through 5000 Monte Carlo simulations and resulted in a 1% probability of a cluster surviving due to chance, fully corrected for multiple corrections. Resultant thresholded cluster maps were examined through *post-hoc* statistical analysis using a cluster based ROI approach to determine the effect of each condition per significant region of interest. This entailed extracting the mean activation levels for each task condition per cluster and conducting between group *t*-tests per condition. These values were then extracted to SPSS for the purposes of cluster-level statistical analysis and featured Bonferroni correction at *P* = 0*.*05. In the case of the color/achromatic contrast, the thresholded maps were inspected to ensure activation of color selective regions in response to task stimuli at a threshold of *p* = 0*.*05 corrected using the same method described above (*t* = 18*.*68, *p <* 0*.*005, *N* = 22).

# **RESULTS**

# **WHOLE BRAIN ANALYSES**

# *Gray matter volume comparison*

Gray matter volumetric analysis results are presented in **Table 1**. The non-parametric FDR *t*-test (*p* ≤ 0*.*01 and a minimum cluster size criteria of 10 mm3) revealed nine regions of increased gray matter volume in synaesthetes relative to controls (see **Figure 3** and **Table 1**). These regions included bilateral cerebellum, left lateral occipital cortex including the precuneus, the left lateral occipital cortex/fusiform gyrus, bilateral occipital cortex/fusiform gyrus, right lingual gyrus, the right posterior division of the postcentral gyrus and the right post-central/pre-central gyrus. No clusters showed a greater volume in controls.

# *White matter volume comparison*

White matter volumetric analysis results are presented in **Table 2**. FDR statistical comparisons (*p* = 0*.*01 with a minimum cluster criteria of 10 mm3) revealed six regions of white matter volume increases in synaesthetes compared to controls (see **Figure 3** and **Table 2**). Similar to the gray matter differences, the white matter volume differences again showed increased volumes in the synaesthetes relative to controls and featured the right occiptal pole including the precuneus, left middle occiptal gyrus, left temporal gyrus, left temporal/fusiform gyrus and bilateral lingual gyrus. No clusters showed a greater volume in controls.

# *White matter FA comparison*

Whole brain white matter FA was examined and 14 clusters showed significantly increased FA in synaesthetes compared to controls (presented with the GM and WM clusters in **Figure 3** and by themselves on a skeletonized framework of axonal tracts in **Supplemental Figure 1**. See also **Table 3**). Three regions of increased FA were identified along the right superior longitudinal fasiculus and included parietal/subgyral and supramarginal areas. Two clusters of increased FA were identified along the right inferior longitudinal fasiculus featuring the insula and frontooccipital and lingual gyrus. The right thalamus showed two clusters of increased FA along the anterior thalamic radiation. There were no clusters showing increased FA in controls compared to synaesthetes.

# *fMRI region of interest analysis featuring GM VBM clusters*

Using the nine GM VBM clusters, the mean activation levels for each of the fMRI conditions were extracted for the purpose of an ROI analysis to investigate the functional response to the letter/non-meaningful character stimuli at each of these cluster locations. The values were entered into a 2 × 2 ANOVA in SPSS [with group as the between-group measure and condition (letter, non-meaningful characters) as the within-group measure]. The ANOVA did not identify any regions with a significant main effect for either group or condition. Four of the nine clusters showed a significant group × condition interaction for the letter/non-meaningful characters contrast (see **Table 4** and **Figure 4**). Bonferroni correction for multiple testing divides the *p*-value significance cut-off (*<*0.05) by the number of tests (9 in this case), to generate a corrected significance value for any single test of 0.0056. None of the single clusters pass this corrected significance threshold. However, this only corrects for any one test showing a significant result at *p <* 0*.*05. The binomial distribution can be used to calculate the likelihood of seeing × out of nine tests being significant at *p <* 0*.*05, by chance. While the chance of seeing at least 1/9 tests significant at this level is quite high (0.37), the chance of seeing 4/9 tests significant by chance is

### **Table 1 | Gray matter volume difference (whole brain VBM analysis).**


*<sup>P</sup>* <sup>=</sup> *0.01 FDR corrected for multiple comparisons with minimum cluster size* <sup>=</sup> *10 mm3. The XYZ coordinates relate to the approximate Center Of Mass (COM) in each case.*

**FIGURE 3 | Structural differences between synaesthetes and non-synaesthete controls.** Whole brain VBM analysis results are shown with significant between-group differences (*p* = 0*.*01 FDR corrected) for gray matter shown in red (refer to **Table 1**) and white matter (*p* = 0*.*01 FDR corrected) shown in blue (refer to **Table 2**). The slice location of each volumetric difference is indicated on the right hand side of the image. In addition, significant

between-group differences from the whole-brain DTI TBSS FA analysis (*p* = 0*.*05 FDR corrected) are also shown in cyan (refer to **Table 3**). For all measures (GM, WM, and FA), the effect direction for the significant between-group differences identified was: synaesthetes *>* non-synaesthete controls. All significant clusters of between-group differences are overlaid on the standard MNI 152 T1 image provided within the FSL toolbox.



*<sup>P</sup>* <sup>=</sup> *0.01 FDR corrected for multiple comparisons with minimum cluster size* <sup>=</sup> *10 mm3. The XYZ coordinates relate to the approximate Center Of Mass (COM) in each case.*

### **Table 3 | White matter fractional anisotropy FA (whole brain analysis).**


*<sup>P</sup>* <sup>=</sup> *0.01 corrected for multiple comparisons using FDR. A minimum cluster size criteria of 5 mm<sup>3</sup> was applied to the fully corrected maps. The XYZ coordinates relate to the approximate Center Of Mass (COM) in each case.*

### **Table 4 | ROI analysis for fMRI conditions using gray matter VBM defined clusters.**


*ROIs determined by independent VBM analysis. Cluster-based ANCOVA (age corrected) based on mean activations per cluster. Bold text indicated significant cluster. See text for discussion of significance based on multiple tests.*

**interest.** Four clusters of significantly increased GM volume in synaestehetes showed a significant group × condition interaction for the letter/non-meaningful characters contrast. Mean fMRI task activations within these ROIs are detailed for each task condition (letter/character and color/achromatic) in synaesthetes and controls. Two clusters, in left lateral occipital cortex/precuneus panel **(A)** and in left post-central gyrus panel **(D)** show a main group effect in response to letters (blue bars). These differences are driven by a strong negative BOLD response to letters in synaesthetes. The other two clusters, in left occipital fusiform/lingual gyrus **(B)** and right postcentral/precentral gyrus **(C)** did not show a main group effect for letters.

very low (0.0006). The overall pattern of effects observed is thus highly significant.

*Post hoc* statistical analyses were used to investigate conditionspecific group comparisons. *Post hoc* parameter estimates (letter and non meaningful character conditions) identified two clusters where a greater response for the "letter" condition was found in the synaesthete group, namely the left lateral occipital cortex/precuneus and the right post-central/pre-central gyrus. In both clusters the mean activation levels revealed significantly greater negative BOLD response to letters in synaesthetes compared to controls (**Figure 4**).

# *fMRI "whole-brain" voxelwise analyses*

To determine whether any other brain areas showed a similar response to letters, or, alternatively, showed the more expected positive BOLD increase, we performed voxelwise whole-brain analyses, looking again for regions showing differential responsiveness to letters vs. non-meaningful characters, in synaesthetes compared to controls. A mixed, 2 × 2 ANOVA using matched samples revealed 14 significant areas showing a group × condition interaction, fully corrected for multiple corrections at a threshold of *P* = 0*.*01 as detailed in the methods above. *Post hoc* cluster based statistical analyses were carried out to determine the direction of responses driving these group × condition interaction effects. None of these areas showed increased responsiveness to letters vs. characters in synaesthetes but not controls. Three showed a main effect group difference in the response to letters, with synaesthetes having a lower and negative average BOLD response in all three (**Table 5**). These clusters are in the left and right inferior parietal lobules and the left transverse temporal gyrus (**Figure 5**). These regions did not overlap the previously defined clusters of increased gray matter volume.

# **DISCUSSION**

Several studies have now looked for structural differences in the brains of synaesthetes, using various modalities, with results that show general consistencies but that vary considerably in the details (Rouw and Scholte, 2007, 2010; Hanggi et al., 2008, 2011; Jancke et al., 2009; Weiss and Fink, 2009; Hupe et al., 2012; Zamm et al., 2013). One general point of agreement is that almost all the structural differences observed between synaesthetes and controls, across all studies—whether in cortical thickness, surface area, volume of gray or white matter clusters or FA of white matter—are increases in synaesthetes (Rouw et al., 2011; Hupe et al., 2012; Zamm et al., 2013); [see Banissy et al. (2012b) for an exception, where decreases as well as increases were observed]. This is also true in our study and argues strongly for the validity and generality of these findings on the basis that random differences would be expected to be observed in both directions.

The other trend that is evident across these studies is that though the observed structural differences are concentrated in occipital regions, they are also observed in temporal, parietal, and frontal areas and have even been reported in hippocampus, cerebellum, and thalamus. We see a similar distribution with structural volumetric differences observed most prominently in occipital and temporal areas (including cuneus, lateral occipital cortex, fusiform and lingual gyri), as well as post-central/precentral gyrus and cerebellum and FA differences apparent in occipital, but also parietal areas and even in thalamic radiations. Though the overall pattern is fairly consistent, no particular locations emerge as a general finding across all these studies.

While the details may vary, the primary picture is quite consistent: synaesthetes strongly tend to show greater gray and white matter volume and greater FA in many regions of the brain. We and others have previously argued that these data provide evidence for a structural difference as the primary cause of synaesthesia (Hubbard, 2007b; Rouw and Scholte, 2007; Bargary and Mitchell, 2008). While these findings are clearly consistent with that model, it has been rightly pointed out that structural differences could of course alternatively arise secondarily due to altered activity patterns in particular brain regions and circuits (Cohen Kadosh and Walsh, 2008). The structural findings thus do not


# **Table 5 | Voxelwise ANOVA—Group × Condition Interaction (letters/characters).**

*Clusters reported were determined through whole brain voxelwise analysis and threshold at p* = *0.01 and fully corrected for multiple comparisons. The post hoc between-group t-tests were performed at a cluster-based level and significant values (p < 0.05) are shown in bold.*

yet lay to rest the question of whether the primary alteration in synaesthesia is anatomical or neurochemical (Grossenbacher and Lovelace, 2001; Ramachandran and Hubbard, 2001; Ward et al., 2006; Hubbard, 2007a; Bargary and Mitchell, 2008).

The fact that the structural differences are quite widespread is consistent with the view that the synaesthetic experience may be just one manifestation of a wider profile of differences between synaesthetes and non-synaesthetes (Bargary and Mitchell, 2008; Barnett et al., 2008a; Hanggi et al., 2011; Dovern et al., 2012). A variety of additional evidence supports this hypothesis. First, different types of synaesthesia can co-occur in individuals (Ward and Simner, 2005; Asher et al., 2009) or in different members of the same family (Barnett et al., 2008a). This led us to propose that an individual genetic mutation may either probabilistically affect wiring across the brain, giving a distinct profile in each individual, or may cause initially widespread differences in wiring, which could be resolved differently in different individuals through experience-dependent processes (Barnett et al., 2008a). Second, several studies have found differences in more general psychological characteristics between synaesthetes and controls, including mental imagery (Barnett and Newell, 2008), sensory sensitivity (Banissy et al., 2009) and significantly higher scores on positive and disorganized schizotypy (Banissy et al., 2012a). Third, we and others have detected differences in very early stages of sensory processing in visual and auditory evoked potentials (Beeli et al., 2008; Barnett et al., 2008b; Goller et al., 2009; Jancke et al., 2012), apparently independent of synaesthetic cross-activation *per se*. Finally, synaesthetes showed widespread differences in global network topology based on cortical thickness correlations (Hanggi et al., 2011) or on functional connectivity measures (Dovern et al., 2012), which were not confined to areas hypothesized to be involved in the grapheme-color synaesthetic experience itself.

To attempt to address whether the areas of structural differences are involved in the synaesthetic experience *per se*, we used the clusters of increased gray matter volume as regions of interest in an fMRI experiment. The goal was to identify regions showing differential responsiveness to letters vs. non-meaningful characters, in synaesthetes but not in controls. Our expectation was that any such differences would be caused by an increased response specifically to letters in synaesthetes, reflecting the supposed "extra activation" associated with the concurrent percept. To our surprise, the areas that did show a difference showed the opposite effect—the response to letters in synaesthetes was not just lower than to meaningless characters (and also lower than in controls), it was negative in sign. This is calculated relative to the baseline activity in the individual voxels of each cluster over the course of the experiment. It is thus not an artifact of averaging across the whole brain. As a control, we examined whole-brain responses to colors and achromatic stimuli to ensure that we could detect an expected positive BOLD response to this contrast in our experiment. We did indeed detect such a signal in the generally expected regions of ventral occipitotemporal cortex (Lueck et al., 1989; Mckeefry and Zeki, 1997; Brouwer and Heeger, 2009) (as well as a number of other regions) in both synaesthetes and controls (**Table 6** and **Supplementary Figure 2**).

Our whole-brain analyses also revealed several areas showing a negative BOLD response to letters, but not characters, in synaesthetes, but not controls. No areas showed a similarly selective greater positive BOLD response to letters in synaesthetes. The lack of positive BOLD signals reflecting the synaesthetic experience is similar to the results of Hupe et al. (2012) and reflects the general variability in fMRI findings where such signals are not reliably found in any specific brain region (Rouw et al., 2011). The negative BOLD signals should be interpreted with caution, given that these findings were unexpected, arose in a modest sample size, and differ from most previous reports.

Nevertheless, the observation of negative BOLD signals is congruent with an early PET imaging study of auditory-color synaesthesia that also reported cortical deactivations, in addition

to letters. Mean fMRI task activations within these clusters are detailed for each task condition (letter/character and color/achromatic) in synaesthetes and controls. In each case, synaesthetes show a mean negative BOLD response to letters. **(A)** Left inferior parietal lobule/post-central gyrus; **(B)** left transverse temporal gyrus; **(C)** right inferior parietal lobule/post-central gyrus.

to activations, in synaesthetes but not controls, in response to stimuli inducing synaesthesia (Paulesu et al., 1995). It would be interesting to know whether previous fMRI studies of synaesthesia did not report negative BOLD signals because they did not arise or because they were not detected due to differences in experimental design (such as focus on specific regions of interest) or analysis methods.

With the caveat that the generalizability of these findings will have to be confirmed in future studies, it is interesting to speculate on what they might mean. Negative BOLD is associated with decreased neuronal activity (Shmuel et al., 2006; Pasley et al., 2007; Wade and Rowland, 2010; Keller et al., 2013) and increased GABA concentrations (Northoff et al., 2007) and thus seems to reflect true deactivation or inhibition, as opposed to physiological artifacts such as blood stealing by nearby active areas. The induction of negative BOLD in the somatosensory cortex ipsilateral to a peripheral stimulus correlates with a reduction in perceptual sensitivity of the non-stimulated hand (Kastrup et al., 2008), reflecting neuronal hyperpolarization and increased inhibition (Devor et al., 2007). The physiological importance of such mechanisms is supported by their involvement in somatosensory habituation (Klingner et al., 2012) and implication in the enhancement of contrast between stimulated regions of visual cortex and surrounding regions with adjacent receptive fields (Wade and Rowland, 2010). In the latter case, active long-range suppressive mechanisms have been invoked to explain the emergence of negative BOLD signals.

One possible, though speculative, explanation for these observations relates to the fact that the synaesthetic percept or association is internally generated and often reported as being "in the mind's eye." A number of studies have shown that generation of an internal sensory representation induces deactivation of regions which might compete for attention or provide conflicting information. For example, visual imagery induces negative BOLD in auditory cortex (Amedi et al., 2005), verbal memory induces deactivation across auditory and visual cortices (Azulay et al., 2009) and imagery of visual motion induces deactivation of early visual cortices (V1-3) (Kaas et al., 2010). Amedi and colleagues found a strong correlation across subjects between the deactivation of auditory cortex during visual mental imagery and their score on the vividness of visual imagery questionnaire (VVIQ). We have previously reported that synaesthetes tend to score higher on this imagery measure (Barnett and Newell, 2008). This is not to suggest that the synaesthetic percepts arise from the same processes as mental imagery *per se*—there is evidence from functional imaging that this is not the case (Rich et al., 2006; Steven et al., 2006). But it is possible that the vividness of a mental image and of a synaesthetic percept both rely on deactivation of other areas.

Such a conclusion is supported by findings from a transcranial direct current stimulation (tCDS) study. Terhune and colleagues found that synaesthetes showed enhanced cortical excitability of primary visual cortex, with a 3-fold lower phosphene detection threshold in response to activation by tCDS (Terhune et al., 2011). They tested whether this hyperexcitability of primary cortex could be either a contributing source to the generation of the synaesthetic percept, or, alternatively, a competing signal, which would interfere with the conscious perception of the synaesthetic percept. They show strong evidence that the latter is the case—stimulation or inhibition of primary visual cortical activity diminished or enhanced, respectively, the synaesthetic experience, based on both self-reports and behavioral interference measures. It thus seems plausible that the cortical deactivations we observe in response to stimuli that induce the synaesthetic experience could be an important part of that response, possibly involved in reducing the signals of competing percepts and allowing the internally generated synaesthetic percept to reach conscious awareness. Future experiments will be required to determine whether such deactivations are indeed a replicable finding and what their functional roles may be.

# **ACKNOWLEDGMENTS**

We are deeply grateful to Kylie Barnett and Hugh Garavan for their invaluable input into the design of these experiments. We would like to thank Danuta Lisiecka for excellent technical support. We are grateful to Edward Hubbard for providing the


# **Table 6 | Voxelwise ANOVA main effect for condition (color/achromatic)** *P* **= 0***.***05 corrected.**

non-meaningful character symbols. This work was supported in part by grant HRB RP/2004/191 from the Health Research Board of Ireland to Kevin J. Mitchell and Fiona N. Newell (principal investigators) and by the Higher Education Authority grant (PRTLI grant, cycle 3) awarded to Fiona N. Newell as a member of Trinity College Institute of Neuroscience. We would also like to acknowledge the Trinity College High Performance Computer facilities for computational support.

# **Supplemental Figure 1 | Results for the TBSS whole brain skeletonized white matter analysis showing significant between-group increases in synaesthetes compared to non-synaesthete controls (***p* **= 0.01 FDR corrected).** The significant clusters of FA difference are enhanced for visualization purposes using standard FSL TBSS visualization tools and are shown on the typical mean white matter skeleton image on the MNI 152 TI image within FSL. The slice locations for significant clusters are in shown in blue on the right of the image.

**Supplemental Figure 2 | Responses to color/achromatic contrast.** Two cortical clusters, in **(A)** right fusiform/middle occipital gyrus and **(B)** left inferior occipital/lingual gyri, show increased positive BOLD responses to colored compared to achromatic stimuli, in both synaesthetes and

# **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fpsyg. 2013.00755/abstract

# **REFERENCES**


859–872. doi: 10.1016/j.neuron. 2005.10.032


Available online at: http://www. fmrib.ox.ac.uk/analysis/techrep

controls.


5q33, 6p12, and 12p12. *Am. J. Hum. Genet.* 84, 279–285. doi: 10.1016/j.ajhg.2009.01.012


and disorganised schizotypy in synaesthetes who experience colour from letters and tones. *Cortex* 48, 1085–1087. doi: 10.1016/j.cortex. 2011.06.009


*J. Neurosci.* 29, 13992–14003. doi: 10.1523/JNEUROSCI.3577-09.2009


*Cogn. Sci.* 5, 36–41. doi: 10.1016/S1364-6613(00)01571-0


Craddock, R. C., et al. (2013). Neurophysiological investigation of spontaneous correlated and anticorrelated fluctuations of the BOLD signal. *J. Neurosci.* 33, 6333–6342. doi: 10.1523/JNEUROSCI.4837-12.2013


*Brain* 118(Pt 3), 661–676. doi: 10.1093/brain/118.3.661


*Med. Imaging* 18, 712–721. doi: 10.1109/42.796284


42, 304–308. doi: 10.1016/S0010- 9452(08)70356-3


06.052

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 28 May 2013; accepted: 27 September 2013; published online: 30 October 2013.*

*Citation: O'Hanlon E, Newell FN and Mitchell KJ (2013) Combined structural and functional imaging reveals cortical deactivations in grapheme-color synaesthesia. Front. Psychol. 4:755. doi: 10.3389/fpsyg.2013.00755*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 O'Hanlon, Newell and Mitchell. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# A taste for words and sounds: a case of lexical-gustatory and sound-gustatory synesthesia

# *Olympia Colizoli\*, Jaap M. J. Murre and Romke Rouw*

*Department of Psychology, Brain and Cognition, University of Amsterdam, Amsterdam, Netherlands*

### *Edited by:*

*Michael Banissy, Goldsmiths University of London, UK*

### *Reviewed by:*

*Jürgen Hänggi, University of Zurich, Switzerland Catherine L. Jones, University of Exeter, UK*

### *\*Correspondence:*

*Olympia Colizoli, Department of Psychology, Brain and Cognition, University of Amsterdam, Weesperplein 4, 1018 XA Amsterdam, Netherlands e-mail: o.colizoli@uva.nl*

Gustatory forms of synesthesia involve the automatic and consistent experience of tastes that are triggered by non-taste related inducers. We present a case of lexical-gustatory and sound-gustatory synesthesia within one individual, SC. Most words and a subset of non-linguistic sounds induce the experience of taste, smell and physical sensations for SC. SC's lexical-gustatory associations were significantly more consistent than those of a group of controls. We tested for effects of presentation modality (visual vs. auditory), taste-related congruency, and synesthetic inducer-concurrent direction using a priming task. SC's performance did not differ significantly from a trained control group. We used functional magnetic resonance imaging to investigate the neural correlates of SC's synesthetic experiences by comparing her brain activation to the literature on brain networks related to language, music, and sound processing, in addition to synesthesia. Words that induced a strong taste were contrasted to words that induced weak-to-no tastes ("tasty" vs. "tasteless" words). Brain activation was also measured during passive listening to music and environmental sounds. Brain activation patterns showed evidence that two regions are implicated in SC's synesthetic experience of taste and smell: the left anterior insula and left superior parietal lobe. Anterior insula activation may reflect the synesthetic taste experience. The superior parietal lobe is proposed to be involved in binding sensory information across sub-types of synesthetes. We conclude that SC's synesthesia is genuine and reflected in her brain activation. The type of inducer (visual-lexical, auditory-lexical, and non-lexical auditory stimuli) could be differentiated based on patterns of brain activity.

**Keywords: synesthesia, fMRI, priming, perception, memory, gustation, olfaction**

# **INTRODUCTION**

*Synesthesia* refers to the experience of cross-modal sensory (and conceptual) mappings, in which the corresponding external stimulation of the additional perceived sense is absent. The term *lexical-gustatory* synesthesia has been used to refer to the automatic and consistent experience of complex taste induced by spoken and written language (Pierce, 1907; Ferrari, 1910; Ward and Simner, 2003; Ward et al., 2005; Simner and Ward, 2006; Gendle, 2007; Simner and Logie, 2008; Simner and Haywood, 2009; Jones et al., 2011; Richer et al., 2011). We present a case study on a rare form of synesthesia in participant SC. SC reported consistently and automatically experiencing tastes, smells, and feelings of texture in her mouth and throat upon hearing, speaking and reading language in addition to hearing many musical and environmental sounds. Upon investigation into SC's synesthesia, it appeared that her synesthetic experiences were similar to those previously reported (Ward and Simner, 2003; Ward et al., 2005; Simner and Haywood, 2009; Richer et al., 2011).

The prevalence of synesthesia is estimated to be about 4% of the general population (Simner et al., 2006). The prevalence of lexical-gustatory synesthesia is unknown, but may be estimated to be less than 0.2% of the population, as it was not found when a random sample was tested for the presence of different types of synesthesia (Simner et al., 2006). For a summary of reported cases of synesthesia related to taste, we refer the reader to Ward and Simner (2003). Lexical-gustatory types of synesthesia have been reported in scientific journals since at least 1907 (Pierce, 1907). In lexical-gustatory synesthesia, the synesthetic percept of taste is typically as complex as veridical taste (e.g., potatoes with gravy), while generic tastes (e.g., bitter, sweet) are notably absent (Pierce, 1907; Ward and Simner, 2003; Richer et al., 2011). We follow the suggestion of Pierce (1907) when he said that in this case *gustatory* must be taken to mean any experiences in or related to the mouth, including pressure and texture in addition to tastes and smells. SC's synesthetic experiences are commonly not only tastes, but also smells and non-olfactory related sensations of objects in the mouth and throat. As is the case with all sub-types of synesthesia, individual differences in the specific experiences are reported and multiple types of synesthesia may co-exist within one individual (e.g., Beeli et al., 2005; Hänggi et al., 2008).

It is not entirely clear how many of the previous reported cases of lexical-gustatory synesthesia co-occurred with some form of non-linguistic sound-gustatory synesthesia. Pierce's (1907) participant did not report having been aware that non-vocal sounds induced taste. However, when systematically tested, she reported experiencing synesthesia elicited by non-vocal sounds, although it was notably less specific than those elicited by words. Richer et al. (2011) reported that non-verbal sounds induced gustatory sensations in synesthete PS, such as the sound of keys on a keyboard (tasted like tomatoes). Beeli et al. (2005) presented a case of (colored) tone interval-taste synesthesia in a trained female musician, who also reported having generic tastes for certain tone intervals while she reported having specific tastes for others. Ward and Simner's (2003) participant, JIW, who has been extensively tested on his lexical-gustatory synesthesia, reported that he did not experience synesthesia for environmental sounds.

Although inter-individual differences between lexicalgustatory synesthetes exist, there is still evidence for common mechanisms between these types of synesthetes. Phonology plays an important role linking words with synesthetic tastes, because similar sounding words tend to taste alike, more so than being visually similar (Ward and Simner, 2003). There is a growing body of evidence suggesting that word-taste pairs are linked by conceptual, semantic and learned experiences as well as by their phonology (Ward et al., 2005; Simner and Ward, 2006; Gendle, 2007; Simner and Haywood, 2009; Richer et al., 2011). The importance of the conceptual level of connection between words and tastes points toward more complex relationships than would be expected if phonology alone were the connection between words and their synesthetic tastes.

A stimulus that elicits a synesthetic experience is termed an *inducer*, and the corresponding synesthetic percept is the *concurrent* (Grossenbacher and Lovelace, 2001). External stimulation corresponding to the synesthetic concurrent is by definition absent in synesthetes. Synesthesia is often unidirectional: inducers elicit synesthetic concurrent percepts, but when the same synesthetic individual is presented with stimuli corresponding to the concurrent sense, no percepts of the inducers are experienced. Typically, words will induce taste concurrents, but tastes will not induce distinct words for lexical-gustatory synesthetes (Ward et al., 2005). A case of bidirectional lexical-gustatory synesthesia was reported by Richer et al. (2011). For this synesthete PS, some tastes evoked one or several inducer words, such as thinking of the name *Valery* while eating celery. In contrast, SC cannot name words that are associated to given tastes, with the exception of some words that have more obvious emotional significance (e.g., remembering the words that produce disgusting tastes or textures). It may be the case that bidirectional effects are typically only present under the conscious threshold in lexical-gustatory synesthetes. There is evidence showing that in grapheme-color synesthesia, bidirectional effects of colors on graphemes are present in synesthetes at a sub-conscious level (Cohen Kadosh et al., 2005, 2007; Meier and Rothen, 2007; Rothen et al., 2010). Such evidence is suggestive of mechanisms underlying structural pathways between related neural networks.

The close proximity of the primary gustatory cortex (in insular cortex) to language-related cortex (e.g., Broca's area) has prompted researchers to suggest that these areas might have additional structural connections present that give rise to the synesthetic experience (Ward and Simner, 2003). This was found to be true in the inferior temporal cortices of grapheme-color synesthetes (Rouw and Scholte, 2007) and in the insular and auditory cortices of an interval-taste and tone-color synesthete (Hänggi et al., 2008). Alternatively, indirect connections from higher-level regions may feed into these taste and language related regions of cortex due to disinhibited feedback in a different manner than in non-synesthetes, while having no differences in structural connectivity (Cytowic and Wood, 1982; Grossenbacher and Lovelace, 2001; Ward and Simner, 2003). To date there is one study known to the authors that investigated brain activation related to lexicalgustatory synesthesia (Jones et al., 2011). Synesthete JIW's brain activation was compared to a group of controls. (We note that synesthete BW was also scanned in the same study. However, BW was presented with JIW's stimuli and this makes it challenging to interpret her data). The authors highlighted the involvement of the insula in processing gustatory (Small, 2010), olfactory (Carmichael et al., 1994), and linguistic information (Dronkers, 1996; Wise et al., 1999). They found that JIW's left anterior insula showed more activation in the presence of unpleasant words compared to neutral words and this differed for controls, while the precuneal cortex showed a difference between JIW and controls when comparing "tasty" and "tasteless" words.

In the present study, we investigated brain activation related to the synesthetic experience of taste and smell for SC. We refer to the literature to determine if SC shows normal language and sound-related brain activation as a within-subject control measure, because we were unable to scan a control group to compare to SC. In addition to the functional magnetic resonance imaging (fMRI) data of SC, a behavioral priming task was designed in order to test certain assumptions based on the description of her synesthesia. For the behavioral experiment, SC's behavior was compared to a group of matched controls. We present her case information before the specific hypotheses concerning this priming experiment.

# **CASE HISTORY OF SC**

At the time of testing, SC was a 29 year-old right-handed woman. She works as a musician, performing artist and teacher. SC's native language is Dutch. SC is fluent in English (exposed since at least age 10) and is also proficient in German and French (studied in school beginning at age 13), but does not consider herself fluent in either of these languages. Languages that she does not understand do not induce synesthetic experiences. SC reported experiencing tastes, smells, textures, as well as experiences which are "hard to describe" upon hearing, reading, and thinking about words, letters and a subset of non-linguistic sounds, including music and environmental noises. Her synesthesia appears to be unidirectional, meaning that tastes and smells do not induce the experience of certain words (i.e., in general she cannot answer a question such as "which words taste like potatoes?"). She reported having experienced this type of synesthesia all her life. SC realized that she was different from others when she was 7 years old. SC reported that her mother experiences days of the week in color but was unaware of the presence of any other types of synesthesia in her family. There did not seem to be any other type of synesthesia present in SC (e.g., related to color). She did report experiencing what has become known in the non-scientific literature and on YouTube as autonomous sensory meridian response (ASMR), which is a pleasurable, specific and intense tingling feeling in the head and body upon hearing "soft" or "crackling" sounds. SC has no history of neurological disease or trauma, reported no serious health complaints or sensory deficits. Although we did not test her IQ or memory, SC reported having been an above-average student and never having any learning disabilities.

SC reported that auditory linguistic stimuli are the strongest inducers of her synesthesia, including hearing her own voice. Each letter of the alphabet and most words elicit a synesthetic percept in SC. When reading, the synesthesia is only induced when she repeats the words mentally to herself as "inner speech." SC reported that she believes that the tastes of words are related by the phonology with the exception of food words. (Although we did not test this directly, we briefly examined this claim in relation to her multilingualism and it appeared to hold true). All food words taste like the foods they describe without any known exceptions. The types of non-linguistic sounds that induce synesthetic experiences in SC are discrete sounds, or "sounds apart from each other." For example, a song with complex components does not necessarily induce a taste or smell, however, the sound of just a bass drum does. SC is a trained musician and did not report that certain tones, chords, or types of instruments produced consistent types of tastes or patterns of tastes.

She described the synesthetic taste as being less intense or real as veridical taste even though she does feel the synesthetic taste on her tongue and in the mouth. The tastes are located on the tongue, mainly in the back. However, the specific location can change depending on the nature of the taste. The synesthetic concurrents are overall lacking the experience of temperature. For example, the Dutch word *alsof* tastes like soft-serve ice cream, however, it does not feel cold. SC has many synesthetic olfactory percepts in addition to tastes. She described the experience of synesthetic smells as the following: "It is as if it's still in my mouth after inhaling something that has smell." Some words and sounds induce sensations that are neither tastes nor smells, for example, the feeling of swallowing buttons, or stuffing her mouth full of marbles. These sensations are not strong enough to trigger her gag-reflex, but it does make her "a little bit nervous." SC described each synesthetic concurrent (tastes, smells, and physical sensations) as a very fast percept. When engaged in conversation, each percept is experienced one after the other at the rate of normal speech. SC reported that the tastes of words do not mix, and compound words (e.g., *bookend*) have two distinct tastes corresponding to the component words (e.g., *book* and *end*). However, she did report that two or more tastes at the same time do occur. Although they do not create a new taste, they can co-exist, often located on distinct parts of the tongue or mouth. SC reported feeling that the presence of real food or beverages in the mouth does not affect the experience of the synesthetic concurrent sense.

# **GOALS OF THE CURRENT RESEARCH**

We designed a behavioral priming task (**Figure 1**) in which the presentation modality, congruency of the prime related to the target taste, and the inducer-concurrent direction were manipulated. SC reported that auditory stimulation is stronger than visual stimulation in eliciting her synesthesia and we predicted a difference in her performance between presentation modalities. In addition, we were interested to know whether Stroop-like

**FIGURE 1 | A trial from the priming task.** This task was presented in both a visual and auditory modality. Congruency refers to whether the type of tastes of the prime and target words were congruent (e.g., sweet–sweet) or incongruent (e.g., sweet–sour). Direction refers to whether the prime and target words were presented in SC's inducer-concurrent (I-C) direction or concurrent-inducer (C-I) direction.

effects of congruency (Stroop, 1935) related to the type of tastes (sweet, bitter, sour, and salty) would be evident in SC. This was possible because the concurrent words themselves taste like the foods they describe (e.g., *apple* tastes like "apple"). Although SC does not report generic tastes or smells as concurrent sensations, she was given a forced-choice task in which she had to classify each concurrent as either sweet, salty, bitter, or sour. In order to test for an effect of congruency, the relationship of the prime to the target word was manipulated in two ways: the type of taste of prime and target words was either congruent (e.g., sweet followed by sweet) or incongruent (e.g., sweet followed by sour). Lastly, we tested for bidirectional effects between the inducer and concurrent words by including two prime-target directions: SC's original inducer-concurrent direction (I-C direction) or the current-inducer direction (C-I direction). This task consisted of a 2 × 2 × 2 factorial within-subjects design. We tested for a significant difference in SC's performance compared to a group of trained controls.

Using fMRI, we tested whether we could successfully localize written and spoken language as well as non-linguistic soundrelated activation in SC's brain based on the fMRI localization literature. Furthermore, we contrasted activation related to the experience of synesthesia (i.e., "tasty") to unrelated activation (i.e., "tasteless") in order to uncover synesthesia-related brain processes. This contrast was compared and contrasted to the case of synesthete JIW reported in Jones et al. (2011).

# **MATERIALS AND METHODS PARTICIPANTS**

SC is a right-handed woman and was 28–29 years old across the course of the testing period. SC has been educated at the university level. The control group that took part in the behavioral priming task was composed of 16 non-synesthetic participants (*M* = 29*.*25 years, *SD* = 1*.*72). Controls were matched to SC for native language, age, gender, and education. Eleven of these control participants completed the consistency test for wordtaste associations (12 participants indicated that they would be available for a retest at a later unknown date. One of these participants did not respond to our further communications). None of the control participants reported experiencing lexicalgustatory synesthesia, grapheme-color synesthesia, or any other types of synesthesia, although not all types of synesthesia were exhaustively ruled out. All participants were informed that they could terminate their participation at any time and gave written informed consent before participating in the research. All participants received C20 for participating in the behavioral session and it took about 1.5 h to complete. SC received an additional C40 for filling in questionnaires about her synesthesia, which were completed outside of the lab. The fMRI experiment was done for a Dutch TV series, and SC was not compensated financially for the fMRI experiment. SC was screened with a standard magnetic resonance imaging (MRI) protocol and gave written informed consent before participating in the MRI study.

This research was approved by the Ethics Committee of the Department of Psychology at the University of Amsterdam.

### **CONSISTENCY TEST MATERIALS AND PROCEDURE**

The list used for the consistency test of word-taste associations for SC consisted of 110 Dutch words taken from the CELEX database (Max Plank Institute: http://celex*.*mpi*.*nl). Both frequent and infrequent words were chosen for this list. The frequency measure used was occurrences per 1 million words. The consistency test for the controls consisted of a subset of 30 random words from SC's list.

For both SC and the controls, the 1st version of each word list was in randomized order (non-alphabetical) and the 2nd version of each list was again randomized. SC was not informed that she would be given the list again (after 9 months), while the controls were informed that they would be given a "pop-quiz" at an unknown time in the following few weeks to test their memory for these associations. The instructions for SC were to respond freely while describing the synesthetic experience and to be as detailed as possible. SC rated the intensity of the experience on a scale of 1 to 5, where 1 indicated "not intense" and 5 indicated the "most intense" experience. In addition, SC was asked to categorize the type of each taste or smell as: sweet, bitter, sour, or salty, and these ratings were used to design the stimuli for the behavioral priming task. The control group was instructed to generate taste and/or smell associations for the list of 30 words and to be as specific as possible.

### *POST-HOC* **CLASSIFICATION AS TASTE OR SMELL**

After the consistency test had been completed, SC classified each word (the 104 inducing words from the consistency test) as inducing a smell, taste, or other type of feeling. In addition, we asked her to classify each concurrent sensation as pleasant or unpleasant. She was permitted to give combinations of these categories (e.g., smell and taste).

### **PRIMING TASK MATERIALS AND PROCEDURE**

The Dutch words and the English translations of the eight word pairs used in the behavioral task are given in **Table 1**. These 16 words were used as the stimuli in the behavioral task. These 8 word pairs were all identically consistent except for *durned*, which was added later in order to balance the stimuli.

All control participants began the experiment with a computerized training task. They were instructed to learn eight pairs of associations between words and specific tastes and smells. In addition, the controls were trained to learn which type of taste or smell (sweet, bitter, sour, and salty) each of these eight pairs had. The instructions were to indicate if the pair of words on screen were "correct" or "incorrect," meaning that they are correctly associated with each other or not. After each "correct" pair, a screen with four taste choices appeared, and participants were instructed to indicate if the pair of words was sweet, bitter, sour, or salty. They received feedback on accuracy after each response was made. The word pairs were presented in both inducer-concurrent directions (I-C and C-I). The control participants performed rounds of the training task until they reached 95% accuracy or higher by the end of a round. Thereafter, they were given a paper test in which they had to fill in the eight pairs of words correctly in a blank table as well as the corresponding taste (in the I-C direction). This was done in order to see if the participants could recall the correct associations freely. If not, they repeated a round of the training task and the free-recall test.

Each trial of the word-pair training task for the control group began with a green centered fixation cross for 1000 ms, followed by one of the word pairs, underneath which two response options were visible until a response was made: correct or incorrect (word-pair association). Feedback on accuracy was given for 1000 ms following each trial. When the pair of words was the correct association, a screen appeared thereafter asking what type of taste the word-pair association had and remained present until a response was made. The four possibilities were: sweet, bitter, sour, and salty. Feedback was given for 1000 ms on accuracy of the type of taste of the word pair. One round of the training task consisted of 160 correct-incorrect association trials and 80 type-of-taste trials. From the 160 correct-incorrect trials, 80 trials were in SC's

**Table 1 | Stimuli used in the behavioral priming task.**


*The word-taste/smell pairs used were based on SC's synesthetic associations. The closest English translations are in parentheses. \*Zuurring should be spelled as zuring, however, we kept it the way SC spelled it.*

I-C direction and 80 were in the C-I direction. The words were presented in white Arial font against a black background.

After this word-taste training, the priming experiment began. For SC, the beginning of the experiment was the priming task. The priming task consisted of two modality conditions: visual and auditory. The task was identical in the two cases, except for the modality of the word presentation. In the visual condition, two words were presented on the computer screen in rapid succession. In the auditory condition, two words were presented via headphones in rapid succession. The instructions were to indicate the taste (sweet, bitter, sour, or salty) of the second word. The first word in each trial was the prime, while the second was the target. The tastes of the first and second words were either congruent or incongruent with each other (i.e., salty followed by salty is congruent, while sweet followed by salty is incongruent). An equal number of congruent and incongruent trials were given within each modality (visual and auditory). The directionality of SC's inducing word and taste-related concurrent word could differ (i.e., the inducer followed by the concurrent vs. the concurrent followed by the inducer), and the directions (I-C vs. C-I directions) of the word-taste pairs were balanced within each modality (visual and auditory). Each participant received the same experiment, meaning that the conditions were not counterbalanced between participants in order to better compare SC to the control group. Participants started with the visual modality followed by the auditory modality. Trials belonging to word-taste congruency and inducer-concurrent directionality were pseudorandomized within participants but not between them. The taste and response options were: sweet, bitter, sour, or salty. These four tastes corresponded to the first two and last two buttons on a seven-button response box, respectively, (the middle three buttons were unused). Participants used the index and middle fingers of each hand to respond.

In the visual condition, the beginning of a trial was indicated by a green centered fixation cross for 1000 ms, followed by a forward mask (hash tags corresponding to each letter in the prime word) for 100 ms, followed by the prime for 1000 ms, followed by a backward mask (identical to the forward mask) for 100 ms, followed lastly by the target word. The target word remained on screen until a response was made. The words were presented in Arial Black font printed in white typeface color against a black background.

In the auditory condition, the beginning of a trial was indicated by a green centered fixation cross for 1000 ms, followed by a forward mask (hash tags corresponding to each letter in the word) for 100 ms, followed by the prime for 1000 ms (via the headphones), followed by a backward mask (identical to the forward mask) for 100 ms, followed lastly by the target word (via the headphones). The target word was heard once. At the onset of the target word, a blue fixation cross appeared and remained on screen until a response was made.

There were 192 trials in each modality (visual and auditory). From these 192 trials, 96 trials were congruent and 96 trials were incongruent (e.g., a sweet prime word followed by a sweet target word was congruent, while a sour prime word followed by a sweet target word was incongruent). In addition, congruency was balanced with word-pair direction (i.e., inducer or concurrent as prime): Ninety-six trials were in the I-C word-pair direction, and 96 trials were in the C-I word-pair direction. The priming task consisted of 384 trials in total.

Before the priming task began, and after the word-pair training task was completed, participants completed a short button training task in order to be able to respond to the four different tastes as fast as possible. The button task consisted of 80 trials; 20 trials per taste. A trial began with a centered fixation cross for 500 ms, followed by one of the four taste words until a response was made or 3000 ms (missed trial). After each trial, a feedback screen appeared for 1000 ms indicating correct and incorrect responses. All stimuli were presented in white Arial font against a black background.

During the entire behavioral session, participants were seated ∼42 cm in front of the computer monitor. All stimuli were presented on a PC using Presentation (version 14.1; www.neurobs.com) on a 23-inch monitor. The screen resolution was 1280 × 1024 pixels, 32-bit color depth. The refresh rate of the screen was 60 Hz. All responses were recorded with a USB 7-button box.

# **fMRI PROCEDURE**

Written words and auditory stimuli were presented in separate runs. SC was presented with two visual and two auditory runs, which were interleaved. SC was not instructed to make any responses in the scanner. During the visual runs, SC was instructed to actively view all the words that appeared on screen in addition to the periods when only a fixation cross was presented. Visual stimuli were projected onto a screen at the rear of the scanner. SC viewed the stimuli through a mirror placed above her head on the head coil. During the auditory runs, SC was instructed to keep her eyes closed, while paying close attention to the sounds through the headphones. SC wore earplugs and headphones, adhering to the standard MRI safety protocol, and foam pads were used to minimize head motion. The volume of the auditory stimuli was adjusted prior to scanning to ensure that she could hear all sounds over the noise of the scanner and through the ear protection. We instructed SC to ignore the scanner noises as much as possible. Before scanning, SC listened to audio recordings of typical scanner noises. SC reported that most of the scanner noises did not induce synesthesia. The few sounds that elicited synesthesia in SC were not intense, and SC reported that it was easy for her to ignore them. We verified that this was the case with SC after scanning. Importantly, scanner noises in the functional runs were the same in all conditions.

# **WRITTEN LANGUAGE LOCALIZER**

In order to localize the (VWFA) and related experiences of synesthetic taste, a 16-second blocked design with four conditions was used: (1) 80 Dutch words with four letters and equal average frequencies chosen at random; (2) The 80 words from (1) presented as Chinese characters in Hanzi Kaishu font; (3) 80 Dutch words that evoked an intense synesthetic experience for SC (all words had a rating of 4 or 5 on a 5-pt. Likert scale); (4) 10 Dutch words that evoked a weak-to-no synesthetic experience or no synesthesia at all for SC (all words had a rating of 1 on a 5-pt. Likert scale). Each of these four conditions was presented five times, for a total of 20 stimulus blocks within one run. The order of these four conditions was pseudo-randomized. Stimuli were visually presented in 16-second blocks. A 16-second baseline (rest) period was presented between each stimulus block in which a centered fixation cross was presented on screen. Sixteen trials of words were presented within each stimulus block in random order. Each trial consisted of a centered fixation cross for 500 ms, followed by a word for 500 ms. There were 320 word trials in each run (80 trials per condition). Each run lasted 11 min. Two runs of this localizer were presented to SC in the MRI scanner. A total of 640 trials were used in the final analysis (160 trials per condition). All words were presented in black Courier New font against a white background.

### **VERBAL AND NON-VERBAL SOUND LOCALIZER**

In order to localize verbal and non-verbal sounds and the related experiences of synesthetic taste, a 16-second blocked design with four conditions was used: (1) Spoken Dutch audio from TV news broadcasts; (2) Spoken Turkish and Russian audio from TV news broadcasts; (3) Musical instruments (bagpipes, drums, guitar, oboe, trumpet, triangle, accordion, harmonica, and xylophone); (4) Environmental sounds (applause, rain, siren, wind, electric fan, keyboard, typewriter, and race car). Each of these four conditions was presented five times, for a total of 20 stimulus blocks within one run. The order of these four conditions was pseudo-randomized. Stimuli were presented through headphones in 16-second blocks. The auditory stimuli were continuously presented within each block, with no breaks in between the different sounds. In each environmental sounds block, each of the eight sounds was played for 2 s. In each musical instruments block, two of the ten instruments were played for 8 s. The order of the different musical instruments and environmental sounds was randomized within a block. A 16-second baseline (rest) period was presented between each stimulus block in which no sounds were heard. During the auditory runs, the screen was black with a centered white fixation cross. SC was instructed to keep her eyes closed during the entire run. Each run lasted 11 min. Two runs of this localizer were presented to SC in the MRI scanner. In this condition, one trial was considered one 16-second block. Therefore, 40 trials were included in total.

# **fMRI DATA ACQUISITION**

Scans were acquired on a Philips 3 Tesla Achieva TX scanner, located at the Spinoza Center, Amsterdam, the Netherlands. Whole brain gradient-echo echo-planar images (voxel size = 3 × 3 × 3 mm, FOV = 240 × 240, matrix = 80 × 80, *TR* = 2000 ms, *TE* = 27*.*63 ms, flip angle = 76.1◦, slice thickness = 3 mm, slice gap = 0.3 mm, 38 slices per volume, sensitivity encoding factor of 2) were acquired to measure blood oxygen level-dependent (BOLD) magnetic resonance images with a 32-channel SENSE head coil. Each functional run consisted of 337 volumes and lasted ∼11 min.

A T1 anatomical scan was acquired (voxel size = 1 × 1 × 1 mm, FOV = 256 × 256, matrix = 256 × 256, *TR* = 8090 ms, *TE* = 3*.*71 ms, flip angle = 8◦, slice thickness = 1 mm, no slice gap, 160 slices per volume, 1 volume was acquired that lasted ∼5 min) so that functional images could be registered to native anatomical space and normalized to the Montreal Neurological Institute (MNI) standard space.

## **fMRI PRE-PROCESSING AND STATISTICAL ANALYSIS**

Analyses of the MRI images were carried out using the Oxford Centre for Functional MRI of the Brain (FMRIB) Software Library (FSL) version 5.0.4, Oxford, UK: http://www*.*fmrib*.*ox*.* ac*.*uk/fsl (Smith et al., 2004; Woolrich et al., 2009; Jenkinson et al., 2012). Statistical analyses were conducted using FSL's fMRI Expert Analysis Tool, (FEAT version 6.00). Preprocessing steps included motion correcting using MCFLIRT (Jenkinson et al., 2002), slice-timing correction (temporal sinc interpolation), pre-whitening (FILM algorithm), spatial smoothing (a 5 mm Gaussian kernel of full-width at half-maximum), grandmean intensity normalization of the entire 4D dataset by a single multiplicative factor, and high-pass temporal filtering (cutoff at σ = 50 s). Voxels belonging to brain tissue were extracted from non-brain tissue voxels using the Brain Extraction Tool (BET; Smith, 2002). No runs were discarded due to motion or other artifacts.

In the first-level analysis, the time course of each run was composed of a blocked design convolved with the double gamma hemodynamic response function and tested with an uncorrected voxel threshold of *P* = 0*.*05. Resulting contrast images were linearly registered to the T1-weighted image using FLIRT with 7◦ of freedom and the full search space (Jenkinson and Smith, 2001; Jenkinson et al., 2002; Greve and Fischl, 2009), then spatially normalized to the T1-weighted MNI-152 stereotaxic space template (2 mm) using FNIRT with 12◦ of freedom and the full search space (http://fsl*.*fmrib*.*ox*.*ac*.*uk/fsl/fslwiki/FNIRT).

In the higher-level analysis, the mean of the first-level runs was computed using a fixed-effects model of variance. *Z*-statistic (Gaussianized T/F) images were thresholded using clusters determined by *Z >* 2*.*3 and a (corrected) cluster significance threshold of *P* = 0*.*05 (Worsley, 2001), controlling the family-wise error rate.

# **RESULTS**

In order to compare the behavioral data of SC to a small sample of controls, we followed the methods developed and published in Crawford and Howell (1998); Crawford and Garthwaite (2002), and Crawford et al. (2010). This methodology is in essence a modified independent samples *t*-test, where the individual case is treated as a sample of *N* = 1 (thereby not contributing to the estimate of the within-group variance) and the null hypothesis is that the case's score is an observation from the control sample's distribution. This methodology was more appropriate to the present circumstance than other standard methods (e.g., comparison of *Z*-scores), because it does not treat the statistics of the normative sample as population parameters, but rather as sample statistics. With small sample sizes (*N <* 50), *Z*-scores overestimate the abnormality of the individual's case by assuming that the variance is known, when it is not, increasing Type 1 error rates (e.g., Crawford and Howell, 1998). Furthermore, by using this modified *t*-test, we avoid violating important assumptions of other widely used statistical tests, such as the assumption of data independence in the Chi-square test.

The modified *t*-test is available as open-source software from the website: http://homepages*.*abdn*.*ac*.*uk/j*.*crawford/pages/dept/ SingleCaseMethodsComputerPrograms*.*HTM*.* The test used in the current study was Singlims\_ES.exe. The output of this test includes hypothesis testing in the form of *t*-test statistics, a point estimate of the abnormality of the case's score as a percentage of the population falling below it, confidence intervals associated with the uncertainty of the abnormality estimate, effect size (*Zcc*), and confidence intervals associated with the uncertainty of the effect size (effect size is comparable to Cohen's *d*; see Crawford et al., 2010).

The significance level was set at α = 0*.*05. As our comparisons between SC and the control group involved multiple *t*-tests, we corrected for multiple comparisons using the Bonferroni method.

### **CONSISTENCY OF SC'S WORD-TASTE ASSOCIATIONS**

The 1st and 2nd versions of the word lists were completed by SC with a period of 9 months in between tests. Although all words were in the Dutch language, SC gave her answers in English on both versions of the list. Following Gendle (2007), we computed consistency based on different levels of identity: identical, nearly identical, conceptually related, and unrelated. We feel that this method is more sensitive to the complexity of the synesthetic experiences and allows for a better understanding of the consistency score.

Of the 110 words presented to SC, 94.5% of the words induced a synesthetic experience. Six words (5.5%) did not induce a synesthetic experience in either the first or second list. We did not include these six words in the calculation of the percentage of consistent answers given. Therefore, the consistency scores given below were calculated based on 104 words. If SC gave an indication of taste/smell in one of the lists, but not in the other, this was counted in the consistency analysis as an unrelated response.

Four different scores for consistency were calculated based on the four levels of response categorization (**Figure 2**): Identical responses, 53/104 = 51%; Nearly identical responses, 23/104 = 22%; Conceptually related responses, 6/104 = 6%; Unrelated responses, 22/104 = 21%.

A majority of the words (51%) had identical responses, 73% of the words had identical and nearly identical responses (cumulative), and 79% of the words had identical, nearly

identical, and conceptually related responses (cumulative), while 21% of the words had unrelated responses.

The 22 instances of unrelated responses consisted of three words (2.7%) that induced "hardly anything" in the first list only, seven different words (6.4%) that induced "hardly anything" in the second list only, and twelve words (11%) that induced two unrelated synesthetic experiences between the first and second lists.

The intensity ratings of the identical and nearly identical responses between the two lists were positively correlated, *rs(*74*)* = 0*.*49, *p <* 0*.*001, indicating that if a word induced an intense experience while filling in the list the first time, it also induced an intense experience while filling in the second list, and vice versa. This result was the same when correlating the intensity ratings of all words, including the six words that never induced a synesthetic experience an intensity rating of 1 in both lists, *rs(*108*)* = 0*.*60, *p <* 0*.*001. The intensity ratings of these 22 unrelatedwordswerealsopositivelycorrelated,*rs(*20*)* = 0*.*46,*p* = 0*.*032, indicating that if a word induced an intense experience while filling in the first list, it also induced an intense experience while filling in the second list even if the description of the synesthetic experiences were unrelated.

# **CONSISTENCY OF THE CONTROL GROUP'S WORD-TASTE ASSOCIATIONS**

The 1st and 2nd version of the lists were completed with a period of three to four weeks in between tests. Four different scores for consistency were calculated based on the four levels of response categorization (*N* = 11; **Figure 2**): Identical responses, 2.27/30, *M* = 7*.*57% (*SD* = 8*.*31);Nearlyidentical responses,3.18/30,*M* = 10*.*60% (*SD* = 5*.*54); Conceptually related responses, 3.82/30, *M* = 12*.*73% (*SD* = 11*.*14); Unrelated responses, 20.73/30, *M* = 69*.*10% (*SD* = 8*.*70).

Aminorityof thewords(7.57%)hadidenticalresponses,18.17% of thewords hadidentical or nearlyidentical responses, and30.90% of the words had identical, nearly identical, or conceptually related responses, while 69.09% of the words had unrelated responses.

# **IS SC MORE CONSISTENT THAN THE CONTROL GROUP?**

A comparison between SC and the controls' consistency scores are given in **Figure 2**. If SC's synesthesia is genuine, we expected SC to have significantly more identical responses and significantly less unrelated responses compared to the controls (in percentages). Resultsarepresentedin**Table 2**.Asexpected,SC's scoreforidentical responses was significantly greater than that of the control group. In addition, SC's score for unrelated responses was significantly lower than that of the control group. No significant differences in scores were found between SC and the controls for nearly identical or conceptually related responses.

SC completed a word-taste association list that was more than three times as long as that of the controls. Furthermore, SC's retest was given 9 months after the first test, while the control group was given their retest within three to four weeks after the first test. SC was not informed that we would retest her word-taste associations, while the control group was instructed to remember the associations they gave. SC gave more identical responses than the control group, while the control group gave more unrelated responses compared to SC. The nearly identical and conceptually


**Table 2 | Differences in consistency scores for word-taste associations between SC and controls.**

*A significant difference (1-tailed) is indicated (\*) and corrected for multiple comparisons.*

related responses did not differ between SC and the control group. Given the above, we conclude that SC's word-taste associations are significantly more consistent than that of the control group, implying that her synesthesia is highly consistent over time and therefore, genuine.

# *POST-HOC* **CLASSIFICATION AS TASTE OR SMELL**

SC classified 104 words as inducing a smell, taste or other type of experience, in addition to it being a pleasant or unpleasant experience.

SC classified 60/104 (57.69%) words as taste: 46 (76.67%) were rated as pleasant and 14 (23.33%) as unpleasant. The majority of tastes induced were pleasant to SC.

SC classified 33/104 (31.73%) words as smell: 12 (36.36%) were rated as pleasant and 21 (63.64%) as unpleasant. The majority of smells induced were unpleasant to SC.

SC classified 6/104 (5.77%) words as "other" (indicating a physicalfeeling or bodily sensation): 3 (50%)were rated as pleasant and 3 (50%) as unpleasant. These sensations were equally likely to be pleasant or unpleasant to SC.

SC classified 1/104 (0.96%) words as inducing both smell and taste. This word was rated as both pleasant and unpleasant.

SC classified 4/104 (3.85%) words as inducing both a taste and "other." These four words (100%) were all rated as unpleasant.

In total,61/104 (58.65%)wordswere ratedasinducingapleasant experience, 42/104 (40.39%) as unpleasant, and 1/104 (0.96%) as both pleasant and unpleasant simultaneously (SC explained that this word can induce two distinct sensations depending on the context). The majority of the words tested induced a pleasant experience to SC.

# **PRIMING TASK**

The following abbreviations are used: *M* = mean, *SD* = standard deviation from the mean (*N* = 16), RT = reaction times. Only correct trials were included in the RT analyses. All RT data is reported in milliseconds. The significance level used was α = 0*.*05.

# **TRAINED CONTROL GROUP**

A 2 × 2 × 2 repeated measures ANOVA was carried out on the data of the control group (i.e., SC's data was not included in the following repeated measures analysis). The factors of interest were: modality (visual vs. auditory), congruency (incongruent vs. congruent), and inducer-concurrent direction (I-C vs. C-I).

There was a significant effect of modality on accuracy scores, *<sup>F</sup>(*1*,* <sup>15</sup>*)* <sup>=</sup> <sup>165</sup>*.*97, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0*.*917 (**Figure 3A**). Controls scored higher on visual trials (*M* = 97*.*01%, *SE* = 2*.*19) compared to auditory trials (*M* = 91*.*41%, *SD* = 1*.*35). A trend was found in RT for the effect of modality, *<sup>F</sup>(*1*,* <sup>15</sup>*)* <sup>=</sup> <sup>3</sup>*.*20, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>*.*094, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0*.*18 (**Figure 3B**). Controls were faster on visual trials (*M* = 878*.*05 ms, *SD* = 127*.*52) thanauditory trials(*M* = 920*.*28 ms,*SD* = 145*.*61), but this difference failed to reach significance in RT.

There was no evidence of a congruency effect in RT or accuracy in the controls (*F <* 1*.*5; **Figures 3C,D**).

A significant effect of direction was found in both accuracy, *<sup>F</sup>(*1*,* <sup>15</sup>*)* <sup>=</sup> <sup>344</sup>*.*70, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0*.*96 (**Figure 3E**), and RT, *<sup>F</sup>(*1*,* <sup>15</sup>*)* <sup>=</sup> <sup>51</sup>*.*43, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0*.*77 (**Figure 3F**). The controls were more accurate (*M* = 98*.*70%, *SD* = 1*.*40) and faster (*M* = 824*.*24 ms, *SD* = 123*.*08) in the I-C direction compared to the C-I direction (*M* = 89*.*71%, *SD* = 2*.*24; *M* = 980*.*08 ms, *SD* = 147*.*72).

The interaction between modality and congruency was significant for RT, *<sup>F</sup>(*1*,* <sup>15</sup>*)* <sup>=</sup> <sup>11</sup>*.*72, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>*.*004, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0*.*44 (**Figure 4B)**, but not in accuracy (*F <* 1; **Figure 4A**). The congruency effect in RT (incongruent vs. congruent) was bigger in the auditory condition (*M* = 50*.*85 ms, *SD* = 65*.*31) than the visual condition (*M* = −14*.*86 ms, *SD* = 75*.*27).

The interaction between direction and congruency was marginally significant in accuracy, *F(*1*,* <sup>15</sup>*)* = 4*.*37, *p* = 0*.*054, η2 *<sup>p</sup>* = 0*.*23 (**Figure 4C**), while no interaction was found in RT (**Figure 4D**). The difference between directions (I-C vs. C-I) was bigger on congruent (*M* = 10*.*03%, *SD* = 3*.*04) trials compared to incongruent trials (*M* = 7*.*94%, *SD* = 2*.*49).

The interaction between modality and direction was significant in both accuracy *<sup>F</sup>(*1*,* <sup>15</sup>*)* <sup>=</sup> <sup>259</sup>*.*20, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0*.*945 (**Figure 4E**), and RT, *<sup>F</sup>(*1*,* <sup>15</sup>*)* <sup>=</sup> <sup>35</sup>*.*23, *<sup>p</sup> <sup>&</sup>lt;* <sup>0</sup>*.*001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0*.*70 (**Figure 4F**). In accuracy, the difference between directions (I-C vs. C-I) was bigger in the auditory condition (*M* = 14*.*58%, *SD* = 2*.*47) compared with the visual condition (*M* = 3*.*39%, *SD* = 2*.*30). This pattern is reversed in the RT data, the difference between directions (C-I vs. I-C) was bigger for visual trials (*M* = 255*.*65 ms, *SD* = 130*.*36) compared to auditory trials (*M* = 52*.*42 ms, *SD* = 84*.*66).

No three-way interactions were found (*F <* 2*.*5).

# **COMPARING SC TO CONTROLS ON THE PRIMING TASK**

The priming task was designed based on SC's report of her synesthetic experiencesand therefore,we expected that shewould exhibit behavior that was statistically differentfrom that of a trained group of controls. We compared SC to the control group in order to

**FIGURE 3 | Priming task: main effects of synesthete SC and controls (***N* **= 16).** Main effects are plotted for three factors of interest: modality refers to the visual or auditory presentation of the stimuli. Effects of modality are presented for accuracy **(A)** and reaction time **(B)**. Congruency refers to whether the type of tastes of the prime and target words were congruent (e.g., sweet–sweet) or incongruent (e.g., sweet–sour). Effects of congruency are presented for accuracy **(C)** and reaction time **(D)**. Direction refers to whether the prime and target words were presented in SC's inducer-concurrent (I-C) direction or concurrent-inducer (C-I) direction. Effects of inducer-concurrent direction are presented for accuracy **(E)** and reaction time **(F)**.

objectively assess the statistical significance of her behavior in each condition. In order to compare SC's data to the control group and furthermore, restrict the number of comparisons made, we calculated the scores in accuracy and RT for each of the main effects and 2 × 2 interaction effects and tested whether SC's score was significantly different from that of the control group in each condition. One-tailed *t*-tests were used because we expected only larger effects forSCcompared tocontrolsforallconditions(i.e.,wedidnotexpect that effects would be larger in the control group). Significance levels were corrected for multiple comparisons using the Bonferroni method.

The results of the *t*-tests are reported in **Table 3**. No differences between SC and the control group were found for RT or accuracy. Therefore, we could not confirm our hypothesis that SC behaves significantly different on this task compared to a group of matched controls who had been briefly trained on a subset of SC's associations. For interpretation, the data of both SC and the control group is illustrated in **Figure 3** (main effects) and **Figure 4** (interaction effects).

### **fMRI ACTIVATION**

Whole brain *Z*-statistic values, MNI coordinates, cluster size, and brain regions are reported for each contrast of interest. Brain regions are based on the Harvard-Oxford Cortical Structural Atlas, the Juelich Histological Atlas, and Brodmann areas are reported from the Talairach Daemon when available.

**FIGURE 4 | Priming task: 2 × 2 interaction effects of synesthete SC and controls (***N* **= 16).** Interactions are plotted for three factors of interest: modality refers to the visual or auditory presentation of the stimuli. Congruency refers to whether the type of tastes of the prime and target words were congruent (e.g., sweet–sweet) or incongruent (e.g., sweet–sour). Direction refers to whether the prime and target words were presented in SC's inducer-concurrent (I-C) direction or concurrent-inducer (C-I) direction. The interaction between modality and congruency are presented for accuracy **(A)** and reaction time **(B)**. The interaction between inducer-concurrent direction and congruency are presented for accuracy **(C)** and reaction time **(D)**. The interaction between modality and inducer-concurrent direction are presented for accuracy **(E)** and reaction time **(F)**.

# **WRITTEN LANGUAGE LOCALIZER**

Ina passive-viewinglocalizer paradigm,SCwas presentedwithfour types of visual stimuli: written Dutch words, written Chinese characters, Dutch words that induced strong synesthetic experiences for SC (i.e., "tasty"), and Dutch words that induced weak-to-no synesthetic experiences for SC (i.e., "tasteless").

The contrast *Dutch words > Chinese characters* is typically used to localize the VWFA in inferior temporal cortex, a brain region selective for the written form of words (e.g., Baker et al., 2007). Contrasting written language to unrecognizable written symbols is useful in order to isolate activation related to comprehension and semantics. The contrast revealed activation over much of SC's brain (**Figure 5**). Considering the range of functions involved in language comprehension and recognition, it is not surprising to find such widespread activation using our *a priori* threshold. Therefore, we increased the thresholding (cluster-based threshold of *Z >* 3*.*1 and corrected cluster-based threshold of *P* = 0*.*05) of this contrast *post hoc*, because of the large amount of activation found at the original level of thresholding. Significant activation and local maxima at the *post-hoc* threshold for the contrast of are given in **Table 4**. The VWFA was the second most significant cluster, after Broca's area, both in the left hemisphere. Activation in the first cluster encapsulated most of the left orbitofrontal cortex, the left supramarginal gyrus and extended into the left middle temporal gyrus. Activation unique to the left hemisphere


**Table 3 | Differences in performance on the priming between SC and controls for main effects and 2 × 2 interactions.**

*Modality refers to the difference in the visual and auditory modalities. Congruency refers to the difference between incongruent and congruent prime-target taste relationships. Direction refers to the difference between the concurrent-inducer direction (C-I) and the inducer-concurrent direction (I-C) of the prime-target relationship. Tests were corrected for multiple comparisons. No significant differences (1-tailed) were found.*

**FIGURE 5 | Written language localizer for synesthete SC.** Significant activation for the contrasts *Dutch words > Chinese characters* (yellow) and *tasty > tasteless words* (red) are shown. Masks of whole-brain *Z*-statistic values are shown in MNI space, with SC's normalized brain as the

background image. The mask in this image of the contrast *Dutch words > Chinese characters* represents the original level of cluster-based thresholding (*Z >* 2*.*3) not the *post-hoc* increased threshold of (*Z >* 3*.*1) and a corrected cluster-based significance threshold of *P* = 0.05.

was in extrastriate cortex and inferior occipital-temporal cortex (including the VWFA).

We next tested the comparison between written Dutch words that elicited a strong synesthetic experience ("tasty") to written Dutch words that elicited weak-to-no synesthetic experience ("tasteless"; **Table 3**). This contrast is better suited to investigate brain activation in SC that was related to the synesthetic experience, because it subtracted out (most) differences in language that were present in the VWFA localizer. Significant activation for the contrast *tasty > tasteless words* (all written Dutch words) is presented in**Table 4**. This contrast yielded three distinct clusters in left hemisphere of SC's brain (**Figure 5**): a frontal region including inferior frontal gyrus, frontal pole, and orbitofrontal cortex along the anterior border with the insular cortex, an inferior temporal cluster corresponding to the VWFA, and a cluster in superior parietal lobe along the supramarginal gyrus. It should be noted that all three clusters in this contrast are overlapping entirely with areas in the contrast of *Dutch words > Chinese characters*, indicating that they are indeed part of SC's language processing network and not independent brain regions. The opposite contrast, *tasteless > tasty words*, again showed three significant clusters of activation that are presented in **Table 4**. These brain areas included right precuneal cortex next to the cingulate gyrus, right posterior cingulate cortex, left cerebellum, and right frontal pole (and notably did not overlap with activation in the other contrasts tested).

SC experiences smell in addition to taste. The whole-brain contrast *tasty > tasteless* words was inspected for evidence of activation in the primary olfactory cortex that is considered to be located in the piriform cortex (Gottfried et al., 2002; Small et al.,


**Table 4 | Significant clusters of fMRI activation in SC related to written language for the contrasts (A)** *Dutch words > Chinese characters,* **(B)** *tasty > tasteless words***, and (C)** *tasteless > tasty words***.**

*\*This contrast was thresholded (voxel-wise) post hoc using Z > 3.1 and a corrected cluster significance threshold of P* = *0.05.*

*Whole brain Z-statistic values, MNI coordinates, cluster size, and brain regions are reported for each contrast of interest. Brain regions are based on the Harvard-Oxford Cortical Structural Atlas, the Juelich Histological Atlas, and Brodmann areas are reported from the Talairach Daemon when available. A higher cluster-based Z-threshold (Z > 3.1) was chosen post hoc for the contrast Dutch words > Chinese characters, because of the large extent of activation found at the original threshold (Z > 2.3). The corrected cluster significance threshold remained P* = *0.05.*

2005). We did not find significant activation that corresponded to the coordinates given for the frontal or temporal piriform cortices (Gottfried et al., 2002). Uponinvestigation of the uncorrected data, small clusters of activation in the left hippocampus and amygdala werefound.Theprocessingofodorsinvolves severalbrainnetworks and furthermore, can be modulated based on context (Gottfried et al., 2002; Small et al., 2005). This experiment was not sensitive enough to separate olfactory and gustatory processes.

# **VERBAL AND NON-VERBAL SOUND LOCALIZER**

In a passive-listening paradigm,SCwas presentedwithfour types of auditory stimuli: spoken Dutch, spoken foreign language, musical instruments, and environmental sounds. Sounds were localized in separate runs from visual stimuli, as SC was instructed to close her eyes during the presentation of auditory stimuli in the scanner. (Therefore, we cannot make a direct comparison between visually presented and spoken words).

Significant activation and local maxima for the contrast of *native language > foreign language* are presented in **Table 5**. The contrast revealed activation over much of SC's brain (**Figure 6**) and in many of the same regions as the written language localizer. The analysis classified ten clusters of activation in both the left hemisphere and right hemispheres. Activation was seen bilaterally spreading across most of the middle temporal gyri from posterior to anterior regions, in inferior frontal gyrus (corresponding to Broca's area), in the superior and inferior parietal lobe along the supramarginal gyrus to the angular gyrus, and in orbitofrontal cortex (anterior to the insular cortex). The spoken language contrast also evoked activation in the left inferior temporal gyrus (SC's VWFA).

# **ACTIVATION RELATED TO MUSIC AND ENVIRONMENTAL SOUNDS**

The contrast *musical instruments > baseline* evoked activation in bilateral primary auditory cortex, orbitofrontal cortex (along the anterior border with insular cortex), inferior frontal gyrus, extending into Broca's area, and paracingulate gyrus (**Figure 6**). Additional activation was seen in the left hemisphere, in the superior parietal lobe along the supramarginal gyrus, and inferior temporal gyrus, overlapping with SC's VWFA (**Table 5**).

The contrast*environmental sounds>baseline* evoked activation in bilateral primary auditory cortex, and rightinferiorfrontal gyrus near Broca's area. Activation related to environmental sounds was lesswidespread than that related tomusic, butitwas almost entirely overlapping with the *musical instruments > baseline* activation (**Table 5**).

In the current paradigm, we were able to contrast the sounds of musical instruments to environmental noises. These contrasts related more directly to the difference in eliciting the synesthetic experience than that of spoken Dutch vs. spokenforeign languages. All of the musical instruments evoked synesthetic experiences in SC while only a few of the environmental noises did. In both conditions, SC rated that the sounds elicited a synesthetic experience of 3/5 (5 being the highest) for level of intensity. Therefore, this differed from the visual presentation of inducing vs. non-inducing


**Table 5 | Significant clusters of fMRI activation in SC related to verbal and non-verbal sounds for the contrasts (A) native language** *>* **foreign language, (B) musical instruments** *>* **baseline, (C) environmental sounds** *>* **baseline, (D) musical instruments** *>* **environmental sounds, and (E) environmental sounds** *>* **musical instruments.**

*Whole brain Z-statistic values, MNI coordinates, cluster size, and brain regions are reported for each contrast of interest. Brain regions are based on the Harvard-Oxford Cortical Structural Atlas, the Juelich Histological Atlas, and Brodmann areas are reported from the Talairach Daemon when available.*

**FIGURE 6 | Verbal and non-verbal language localizer for synesthete SC.** Significant activation for the contrasts *musical instruments > baseline* (yellow) and *musical instruments > environmental sounds* (red) are shown. The contrast *musical instruments > environmental sounds* represents a

difference in synesthetic gustatory experience for SC (i.e., *"more" > "less" synesthesia*) in addition to differences in types of sounds. Masks of whole-brain *Z*-statistic values are shown in MNI space, with SC's normalized brain as the background image.

synesthetic words related to the amount of stimuli that induced the synesthesia between contrasts. It should be noted that activation in this contrast is also related to differences between the musical and environmental stimuli, as they were not balanced for low-level properties such as tone and temporal frequency. Still, we believe that these stimuli are better matched to subtract out largely confounding effects, as those present in the spoken language contrast. We refer to the literature to identify consistent activations (in non-synesthetes) for these types of auditory stimuli.

Significant activation for the contrast *musical instruments > environmental sounds* is reported in **Table 5** and shown in **Figure 6**. Four clusters of activation were found only in the left hemisphere: the superior parietal lobe around the supramarginal gyrus, angular gyrus, the superior temporal pole extending into the anterior insula and orbitofrontal cortex, and paracingulate gyrus in the frontal lobe. The opposite contrast, *environmental sounds > musical instruments* yielded activation in one cluster in the right angular gyrus (**Table 5**). The contrasts *musical instruments > environmental sounds* and *environmental sounds > musical instruments* yielded significant activation in left and right angular gyri, respectively. This cluster found in the angular gyrus of the right hemisphere was located posterior to the location of the cluster in the corresponding angular gyrus of the left hemisphere.

# **DISCUSSION**

We present a case report on synesthete SC, who experiences lexical-gustatory and sound-gustatory synesthesia. In addition to gustatory sensations (∼60% of words tested), ∼30% of words induced smells, and ∼10% of words induced another type of sensation or a combination of experiences. SC's synesthesia was stable over time and more consistent than a group of controls. Her long-term consistency for a large list of synesthetic inducers serves as the synesthetic "test of genuineness" and is comparable to previous reports of lexical-gustatory synesthesia (Baron-Cohen et al., 1987; Ward and Simner, 2003; Ward et al., 2005; Gendle, 2007; Simner andHaywood, 2009; Richer et al., 2011). SC exhibited behavior and reported experiences that are in line with previous reports of lexical-gustatory synesthesia. SC reported experiencing synesthesia her whole life; that it is induced by hearing people speak, speaking herself, reading and "inner" speech; the concurrent sensations are as complex as real tastes or smells, which can be pleasant and unpleasant; the concurrent sensations are felt on the tongue, mouth and throat; food words taste like the foods that they describe (Pierce, 1907; Ward and Simner, 2003; Gendle, 2007; Richer et al., 2011). SC shares some specific qualities with synesthete TD, who also reported going out of her way to avoid speaking "ugly words" that induce a very unpleasant taste concurrent. SC has concurrent sensations not entirely related to taste or smell, such as feelings of pressure and objects in the mouth, similar to Pierce's case study and synesthete PS. Still, differences between SC and what has so far been reported in detail in the literature exist. For instance, SC did not report experiencing temperature along with the concurrent sensations and experiences complex synesthesia for non-linguistic sounds, including musical instruments and environmental noises. Furthermore, SC experiences synesthesia for most of the words tested (94.5%), and this percentage varies widely between lexicalgustatory synesthetes: 46–100% across 5 synesthetes (Simner and Haywood, 2009), 31–86% across 7 synesthetes (Ward et al., 2005), 47.3% in synesthete TD (Gendle, 2007), 44% in synesthete JIW (Ward and Simner, 2003), 42% in synesthete PS (Richer et al., 2011), illustrating the range of inter-individual differences within a sub-type of synesthesia.

# **PRIMING TASK**

Based on the experiences reported by SC, a behavioral priming task was designed in order to test certain assumptions and to see if her behavior on this task stood out from a group of controls trained on a subset of SC's associations. In this priming task, participants indicated the type of taste of the target word (sweet, bitter, sour, or salty). The priming task was presented in both a visual and auditory modality. Furthermore, it was divided into two taste conditions and two word-pair directions, which refer to the relationship of the prime to the target word. In a congruent trial, the prime and target word had the same type of taste (e.g., sweet followed by sweet), while in an incongruent trial, the prime and target word had different types of taste (e.g., sweet followed by sour). The direction of the synesthetic inducer and concurrent was also balanced between the primes and target words. In an I-C direction trial, an inducer preceded a concurrent (e.g., *alsof* followed by *softijs*), while in a C-I direction trial, a concurrent preceded an inducer (e.g., *softijs* followed by *alsof*). We compared SC to the trained control group in order to objectively assess the statistical significance of her behavior in each condition.

Significant main effects of presentation modality (**Figure 3A**) and inducer-concurrent direction were found in the control group (**Figures 3E,F**). The main effect of congruency was not significant as expected, nor did it differ between SC and the controls (**Figures 3C,D**). However, congruency and presentation modalities interacted in such a way that the expected congruency effect (in RT) was present in the auditory modality but not in the visual modality for the control group (**Figure 4B**). SC did not differ significantly from controls in RT, however, SC performed better on incongruent trials in the visual modality (100%) compared to congruent trials (93.75%), and showed no difference between incongruent (93.75%) and congruent trials (93.75%) in the auditory modality. No such difference was present in accuracy for the control group. We expected SC to have higher accuracy on congruent compared to incongruent trials. The difference between SC and controls was not significant, nor did it differ as we expected. Therefore, any explanation for SC's higher accuracy on incongruent trials would be speculative at best. We conclude that the interaction between modality and congruency showed the closest dissociation in performance of SC compared to the group of controls.

An interesting and unexpected effect of inducer-concurrent direction in the control group was found (**Figures 3E,F**). The control participants were trained to associate the word pairs in both directions. Therefore, we did not expect an effect of inducerconcurrent direction to be present in the control group, and we expected that SC would differ significantly in this respect from the controls. In contrast to our expectations, SC performed similarly to the controls in both the I-C and C-I directions (**Figure 3F**). Accuracy was significantly higher in the I-C direction compared to the C-I direction (**Figure 3E**). This may be due to the fact that the original list from which the controls studied before training, as well as the paper test of the associations given after training, were only presented in the I-C direction. This may have been enough of an influence to cause the effect of direction in the controls. We note that one of the control participants spontaneously reported that the learned associations "feel natural" and noticed that the word pairs shared phonological properties, such as similar vowel tones. Associations between tastes and pitch are known to exist in non-synesthetes (Crisinel and Spence, 2009; Simner et al., 2010). Naturally occurring (but mainly unconscious) associations between the phonology of the word used and the foods that they were associated with may explain at least part of the effects found in the control group. However, this hypothesis is an interesting avenue for future research.

SC'sperformanceon thispriming taskdidnotdiffer significantly from the control group on any main effects or interaction effects. We conclude that SC's behavior followed the same pattern as that of the control group who were trained to associate this subset of SC's word-taste pairs. When such a control group is used, then it is apparent that this task should not be considered as a diagnostic tool in assessing behavioral differences related to the presence of lexical-gustatory synesthesia. The similarities between SC and the trained control group on the priming task are quite striking. The controls were trained briefly, especially in comparison to SC's life-long experiences. They showed a very strong pattern of behavior on the priming task, where almost each effect reached significance in RT, accuracy or both.We did not, however, compare SC's performance to a group of untrained matched controls. SC's behavior did reflect that of the trained control group, which is objective evidence that she does have word-taste associations, as she was never trained to associate the word pairs to each other as the controls were. The word pairs used were SC's own associations, which she describes as automatic, consistent and having been present as long as she can remember. The question remains open why the behavior of a synesthete so closely resembles that of controls briefly trained on semantic associations. Perhaps the task design was insensitive to key elements of SC's synesthetic experience in contrast to the semantic nature of her associations.

# **SC'S BRAIN ACTIVATION**

Using fMRI, we measured SC's brain activation while passively viewing and hearing different sets of stimuli. SC served as her own control in the fMRI experiment, as practical limitations prevented us from being able to scan a comparable control group. Therefore, the data presented here should be considered exploratoryin nature. In order to better understand the brain activation related to SC's synesthesia, we employed blocked designs as well as control conditions to evaluate SC's brain activation related to well-known functions, such as language processing (Danckert and Mirsattarri, 2012). We conclude that SC's brain activation related to language (both written and spoken) was localized successfully compared to the known literature on language related activation (Fedorenko et al., 2010, 2011, 2012), evoking activity in well-known language related structures, such as bilateral middle temporal gyri, inferior frontal gyri, orbitofrontal gyri, including Broca's area, and inferior temporal cortex in the fusiform gyrus (**Figure 5**). In addition, we tested whether SC showed normal brain activation upon hearingmusicalinstruments and environmental sounds compared to baseline (no stimulation). We can conclude that SC showed normal brain activation upon hearing non-linguistic auditory stimuli (Fedorenko et al., 2011). Activation related to musical instruments was more extended compared with environmental sounds. Activation related to environmental sounds was contained within regions activated by musical instruments (both wholebrain contrasts), implying a shared neural substrate between these categories of sound. These shared regions were bilateral superior temporal gyri and right inferior frontal gyrus (the region of the right hemisphere that corresponds to Broca's area in the left hemisphere).

A comparison between the "tasty" to "tasteless" stimuli allowed us to localize brain activation related to SC's synesthetic experiences by canceling out most of the activation related to language comprehension, because some visually presented words hardly induced any synesthesia in SC. Three clusters of activation were found in the whole-brain analysis (all in the left hemisphere): inferior frontal cortex (Broca's area) extending into orbitofrontal and anterior insular cortex, the VWFA in inferior temporal cortex, and the parietal lobe along the supramarginal gyrus (**Figure 5**). Interestingly, all these clusters of activation are "contained" within the activation found for written language, implying that the neural substrate of SC's synesthetic experience is not independent from the neural substrate of language. As we were not able to test a control group, we recognize that activation in these contrasts may not be entirely related to synesthesia. We present SC's activation as a supplement to Jones et al. (2011) so that it may serve as a useful reference for future fMRI studies on such rare cases of lexical-gustatory synesthesia.

The contrast used here (*tasty >tasteless words*) is similar but not entirely comparable to the one reported in the categorical analysis by Jones et al. (2011) for participant JIW. Jones et al. compared pleasant, unpleasant, and emotionally neutral words (balanced for frequency, age of acquisition and number of syllables) that induced tastes to tasteless words. In the current study, SC rated words as either pleasant or unpleasant as a *post-hoc* analysis; therefore, emotional affect was not balanced in the design. Due to the fact that SC experiences synesthesia for almost every word she hears, stimuli were not balanced for frequency, age of acquisition or number of syllables.

Significantwhole-brainactivationfor JIWwasfoundin the right middle occipital gyrus and left lingual gyrusfor the contrast*tasty> tasteless words* (Jones et al., 2011). We did not find corresponding activation for SC for the *tasty > tasteless words* contrast. When JIW's brain activation for the *tasty > tasteless words* contrast was compared to a control group, a difference in activation was found in JIW's precuneal cortex along the central commissure. We found precuneal cortex activation in SC's brain, but for the opposite contrast (whole brain) of *tasteless > tasty words*. The precuneal cortex is implicated in a wide variety of cognitive functions, such as integration of multimodal information and has shown up in both functional and structural imaging studies on graphemecolor synesthesia (Jones et al., 2011). The increase in activation in response to tasteless words compared to tasty words in these regions may reflect an interaction between processes that reflect the (missing) internal experience of the synesthetic concurrent and not the content of that experience. Alternatively, such activation could be reflective of default mode network activation (Fransson and Marrelec, 2008). Further research is needed as these findings were not expected and the explanations are purely exploratory.

Jones et al. (2011) used three regions of interest (ROIs) defined *a priori* based on their role in processing taste and affective components of taste (pleasant vs. unpleasant vs. neutral). These were anterior insula, orbitofrontal cortex, and anterior cingulate. In our whole brain analysis of SC, we found activation in the left orbitofrontal cortex along the border with the anterior insula. Jones and colleagues found that activation in this region related to the emotional content of the taste compared to neutral tastes, and specifically to unpleasant tastes. In JIW's data, pleasant and unpleasant words elicited more activity in left anterior insular cortex than words that induced affectively neutral tastes. The contrast of *unpleasant > neutral* stimuli differed between JIW and the control group in the anterior insula. We did not separate or balance conditions of stimuli based on SC's affective evaluation of the concurrent sensations, but *post-hoc* inspection validated the fact that both pleasant an unpleasant synesthetic experiences were related to the words used as stimuli (64% were pleasant).

Unlike JIW, SC experiences synesthesia induced by nonlinguistic and non-vocal sounds (Ward and Simner, 2003). Therefore, we were presented with a unique opportunity to investigate brain activation related to sound-gustatory synesthesia in SC. All the musical instruments used as stimuli induced a concurrent sensation in SC, while only a few of the environmental sounds did. Although these types of stimuli were not balanced to account for low-level differences in acoustic properties, we contrasted musical instruments to environmental sounds in order to better localize activation related to differences in SC's synesthetic experience (**Figure 6**). Furthermore, the fact that these types of sounds share a neural substrate implies that the same brain regions share functional mechanisms. The contrast *musical instruments > environmental sounds* can be compared to the contrast *tasty > tasteless words* (written form), because both contrasts involve a difference between "more" and "less" synesthetic experiences for SC (although these contrasts are not equivalent). We visually compared the contrast *musical instruments > environmental sounds* to the contrast *tasteless > tasty words*for written words, and we found that two regions of activation were overlapping: the left superior parietal cortex along the supramarginal gyrus and left anterior insular cortex along the border of the inferior orbitofrontal cortex.

# **REFERENCES**


The primary gustatory cortex is located in the anterior insular cortex (Ogawa et al., 2005; Veldhuizen et al., 2007; Small, 2010), and this activation may reflect SC's synesthetic experience of taste or the affective component of this experience. The region in the superior parietal lobe is activated at the group level and across studies of grapheme-color synesthesia, and a common role in the conscious and attentional binding of components of sensory stimuli across sub-types of synesthesia has been proposed (Rouw et al., 2011).

We cannot conclude that we have localized activation related to synesthesia alone, as we had no control group to compare SC's activation to. However, based on what is known about gustatory, linguistic, auditory and synesthetic brain activation, in addition the only other (known) publication on brain activation in a lexical-gustatory synesthete (Jones et al., 2011), we feel that the data obtained on SC was reliable and reflective of her synesthesia. Taken together these results showed that the synesthetic experience of taste was not only reflected in the brain activation of a gustatory-type synesthete, but furthermore, that the modality of the inducer (visual-lexical, auditory-lexical, and non-lexical auditory stimuli) could be differentiated from these patterns of brain activity.

# **CONCLUSION**

We were presented with a rare opportunity to investigate a case of lexical-gustatory and sound-gustatory synesthesia in SC. The authenticity of her experiences are reflected in the consistency of her word-taste associations as well as during an objective behavioral priming task, although, not always inways thatwere expected. Such a rare form of synesthesia provides us with the opportunity to understand neural differences at the individual level that underlie differences in phenomenological experiences.

## **ACKNOWLEDGMENTS**

We would like to give a special thank you to SC for her time, interest, and dedication to our experiments. We thank Ricardo van Buren for help with the experimental setup and data collection, all of the control participants, as well as Henk Bas and Jurjen Simmelink for introducing us to SC.

*J. Cogn. Neurosci*. 17, 1766–1773. doi: 10.1162/089892905774589181


*Cogn. Sci*. 5, 36–41. doi: 10.1016/S1364-6613(00)01571-0


in grapheme-color synesthesia. *Nat. Neurosci* 10, 792–797. doi: 10.1038/nn1906


early gustatory cortex by attention to taste. *Chem. Senses* 32, 569–581. doi: 10.1093/chemse/bjm025


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 27 July 2013; accepted: 02 October 2013; published online: 23 October 2013.*

*Citation: Colizoli O, Murre JMJ and Rouw R (2013) A taste for words and sounds: a case of lexical-gustatory and sound-gustatory synesthesia. Front. Psychol. 4:775. doi: 10.3389/fpsyg. 2013.00775*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Colizoli, Murre and Rouw. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# The induction of synaesthesia with chemical agents: a systematic review

### *David P. Luke1 \*† and Devin B. Terhune2 \*†*

*<sup>1</sup> Department of Counselling & Psychology, University of Greenwich, Eltham, UK <sup>2</sup> Department of Experimental Psychology, University of Oxford, Oxford, UK*

### *Edited by:*

*Michael Banissy, Goldsmiths University of London, UK*

### *Reviewed by:*

*Asifa Majid, Radboud University Nijmegen, Netherlands David Brang, Northwestern University, USA*

### *\*Correspondence:*

*David P. Luke, Department of Counselling & Psychology, University of Greenwich, Avery Hill Road, Eltham, London, SE9 2UG, UK e-mail: d.p.luke@gre.ac.uk; Devin B. Terhune, Department of Experimental Psychology, University of Oxford, South Parks Road, Oxford, OX1 3UD, UK e-mail: devin.terhune@psy.ox.ac.uk †These authors have contributed equally to this work.*

Despite the general consensus that synaesthesia emerges at an early developmental stage and is only rarely acquired during adulthood, the transient induction of synaesthesia with chemical agents has been frequently reported in research on different psychoactive substances. Nevertheless, these effects remain poorly understood and have not been systematically incorporated. Here we review the known published studies in which chemical agents were observed to elicit synaesthesia. Across studies there is consistent evidence that serotonin agonists elicit transient experiences of synaesthesia. Despite convergent results across studies, studies investigating the induction of synaesthesia with chemical agents have numerous methodological limitations and little experimental research has been conducted. Cumulatively, these studies implicate the serotonergic system in synaesthesia and have implications for the neurochemical mechanisms underlying this phenomenon but methodological limitations in this research area preclude making firm conclusions regarding whether chemical agents can induce genuine synaesthesia.

**Keywords: drugs, serotonin, synaesthesia**

# **INTRODUCTION**

Synaesthesia is an unusual condition in which a stimulus will consistently and involuntarily produce a second concurrent experience (Ward, 2013). An example includes grapheme-color synaesthesia, in which letters and numerals will involuntarily elicit experiences of color. There is emerging evidence that synaesthesia has a genetic basis (Brang and Ramachandran, 2011), but that the specific associations that an individual experiences are in part shaped by the environment (e.g., Witthoft and Winawer, 2013). Further research suggests that synaesthesia emerges at an early developmental stage, but there are isolated cases of adult-onset synaesthesia (Ro et al., 2007) and it remains unclear whether genuine synaesthesia can be induced in non-synaesthetes (Terhune et al., 2014).

Despite the consensus regarding the developmental origins of synaesthesia, the transient induction of synaesthesia with chemical agents has been known about since the beginning of scientific research on psychedelic drugs (e.g., Ellis, 1898). Since this time, numerous observations attest to a wide range of psychoactive substances that give rise to a range of synaesthesias, however, there has been scant systematic quantitative research conducted to explore this phenomenon, leaving somewhat of a lacuna in our understanding of the neurochemical factors involved and whether such phenomena constitute genuine synaesthesia. A number of recent theories of synaesthesia implicate particular neurochemicals and thus the possible pharmacological induction of synaesthesia may lend insights into the neurochemical basis of this condition. For instance, *disinhibition* theories, which propose that synaesthesia arises from a disruption in inhibitory activity, implicate attenuated γ-aminobutyric acid (GABA) in synaesthesia (Hubbard et al., 2011), whereas Brang and Ramachandran (2008) have specifically hypothesized a role for serotonin in synaesthesia. Furthermore, the chemical induction of synaesthesia may permit investigating experimental questions that have hitherto been impossible with congenital synaesthetes (see Terhune et al., 2014).

Despite the potential value in elucidating the induction of synaesthesia with chemical agents, there is a relative paucity of research on this topic and a systematic review of the literature is wanting. There is also an unfortunate tendency in the cognitive neuroscience literature to overstate or understate the possible induction of synaesthesia with chemical agents. The present review seeks to fill the gap in this research domain by summarizing research studies investigating the induction of synaesthesia with chemical agents. Specifically, our review suggests that psychoactive substances, in particular those targeting the serotonin system, may provide a valuable method for studying synaesthesia under laboratory conditions, but that methodological limitations in this research domain warrant that we interpret the chemical induction of synaesthesia with caution.

# **METHODS**

# **LITERATURE SEARCH AND INCLUSION CRITERIA**

A literature search in the English language was conducted using relevant databases (PubMed, PsychNet, Psychinfo) using the search terms synaesthesia, synesthesia, drug, psychedelic, LSD, psilocybin, mescaline, MDMA, ketamine, and cannabis and by following upstream the cascade of references found in those articles. Initially a meta-analysis of quantitative findings was planned, however, it became apparent that there had been only four *direct* experimental attempts to induce synaesthesia in the laboratory using psychoactive substances, making such an analysis unnecessary. A larger number of other papers exist, however, describing *indirect* experiments in which participants were administered a psychoactive substance under controlled conditions and asked via questionnaire, as part of a battery of phenomenological questions, if they experienced synaesthesia during the active period of the drug. Whilst these studies typically provide a non-drug state condition for comparison they did not set out to induce synaesthesia and so are less evidential than direct experimental studies. There also exist a number of case reports describing the induction of synaesthesia using chemical agents within various fields of study. Under this category, we include formal case studies as well as anecdotal observations. A final group of studies used survey methodologies, providing information regarding the prevalence and type of chemically-induced synaesthesias among substance users outside of the laboratory. Given the range of methodologies and quality of research, we summarize the studies within the context of different designs.

# **DRUG TYPES**

The majority of the studies and case reports relate to just three psychedelic substances—lysergic acid diethylamide (LSD), mescaline, and psilocybin. However, some data is also available for ketamine, ayahuasca, MDMA, as well as less common substances such as 4-HO-MET, ibogaine, *Ipomoea purpurea*, amyl nitrate, *Salvia divinorum*, in addition to the occasional reference to more commonly used drugs such as alcohol, caffeine, tobacco, cannabis, fluoxetine, and buproprion.

# **RESULTS**

The final search identified 35 studies, which are summarized in **Table 1**. Here we review the most salient results from the different studies.

# **EXPERIMENTAL STUDIES**

Among experimental studies, we distinguish between *direct* and *indirect* experiments as those that explicitly attempted to induce synaesthesia in a hypothesis-driven manner and those that explored the induction of synaesthesia as part of a larger battery, respectively.

# *Direct experiments*

We identified four published experimental studies formally attempting to induce synaesthesia with pharmacological agents (Kelly, 1934; Simpson and McKellar, 1955; Hartman and Hollister, 1963; Masters and Houston, 1966). In the first study (Kelly, 1934), four non-synaesthetes had previously taken part in a 7-week auditory tone-color associative training study in which they were presented with eight different tone-color pairs 2000 times. Although the participants demonstrated a tone-color association learning effect, no evidence of spontaneous color photisms in response to the tones were observed. One week after the last training session the participants were joined by a fifth who had the day before received 1000 single tone-color pairings but likewise demonstrated no synaesthesia, and they consumed 15 g of (presumably dried) peyote cactus. Although not specified, this amount of peyote provides an estimated dosage of between 0.15 and 1.2 g of mescaline (Crosby and McLaughlin, 1973; Bruhn et al., 1978), providing anything from a mild to a very strong dose depending on relative alkaloid content (Shulgin and Shulgin, 1991). One participant took a further 5 g after receiving no visual perceptual changes, but to little effect. Although four of the five participants perceived colorful visual imagery, due to the mescaline, none perceived the appropriate color when the tones were played. However, these four participants experienced other types of synaesthesia, including haptic-visual, kinaesthetic-visual (especially color), and algesic-color. This suggests a high prevalence of synaesthesia following the consumption of mescaline, but that mescaline does not seem to enhance trained associations.

The second study (Simpson and McKellar, 1955), included two congenital synaesthetes (auditory-visual and multiple types) with the researchers acting as non-synaesthete controls (McKellar, 1957). Participants were administered four mescaline doses between 0.3 and 0.5 g (considered strong doses; Shulgin and Shulgin, 1991) on separate occasions. During the active effects of the drugs, participants were presented with a variety of sensory stimuli (visual, auditory, tactile, gustatory, olfactory, kinaesthetic, thermal, and algesic). Participants reported several distinct types of novel synaesthesias: auditory-visual (all 4 participants), kinaesthetic-visual (3), tactile-visual (2), olfactoryvisual (2), algesic-visual (2), thermal-visual (1), olfactory-tactile (1), visual-tactile (1), and visual-thermal (1), with a relatively equal ratio of novel synaesthesias (10:7) between synaesthetes and non-synaesthetes. In addition, one of the congenital synaesthetes reported enhancement of their usual auditory-tactile and visualthermal synaesthesias, suggesting that mescaline can both induce synaesthesia among non-synaesthetes and enhance synaesthesia among congenital synaesthetes.

A third study by Hartman and Hollister (1963; see also Hollister and Hartman, 1962) compared the effects of mescaline (5 mg/kg), psilocybin (150 mcg/kg), and LSD (1 mcg/kg), considered to be light to moderate doses, administered under blind conditions. A total of 18 participants took both substances 1 week apart and received auditory stimulation from 16 pure sonic tones before and after drug administration. Overall, participants reported significantly more colors and other auditorily-driven visual effects (brightening of the visual field, shattering of patterns, and patterning of form) compared to baseline whilst under the influence of both LSD and mescaline, but not psilocybin, although there was a non-significant increase in such experiences with the latter. Fewer than 50% of the participants (exact proportion not reported) experienced auditorily-induced synaesthesia under the influence of the three psychedelics.

A fourth experimental study is described in Masters and Houston (1966), but only minimal details about the methodology and results were presented. Specifically, they report a series of informal experiments and interviews conducted with 214 participants in 204 sessions in which psychedelic drugs


### **Table 1 | Summary of studies reporting induction or modulation of synaesthesia through drug use.**

*(Continued)*

### **Table 1 | Continued**


### **Table 1 | Continued**


*(Continued)*



*Empty cells indicate that the respective information was not reported in the study. aThese two papers describe different analyses from the same data.*

were consumed. In the course of this work the authors report successfully intentionally inducing color-sound and auditorygustatory synaesthesia with LSD, but there is no information regarding the proportion of participants reporting such effects, the dosage, or other results. Insofar as few details are provided about this study, it is difficult to critically evaluate its methodology and results, but the reported results do converge with the three previous experimental studies.

These four studies suggest that synaesthesia can be induced in a controlled environment using chemical agents. Nevertheless, they suffer from a large number of limitations including a lack of placebo control, double-blinds, and randomized allocation. The absence of these experimental controls warrant concerns about demand characteristics (Orne, 1962)—participants may have expected to have synaesthesia under the influence of the drugs, thereby inflating their tendency to endorse that they had experienced synaesthesia. In addition, the studies identified the experience of synaesthesia using self-reports rather than behavioral measures of automaticity or consistency (for a review, see Ward, 2013).

# *Indirect experiments*

Despite the lack of direct experiments in the last 50 years, we identified nine studies that investigated the psychological effects of psychedelic drugs under controlled conditions, including placebo controls. As part of a broad assessment, these studies included questions regarding the experience of synaesthesia during the active period of the drug. Studerus (2013) reviews data from several studies (Studerus et al., 2010, 2011, 2012) that included nearly 600 psilocybin, MDMA, and ketamine test participants. Studerus (2013) provides prevalence rates of induced auditoryvisual synaesthesia (only), with sounds as inducers, for these substances (MDMA: 10%; ketamine: 27%; psilocybin: 37%) and demonstrates its linear dose-dependent nature with psilocybin, the most studied substance in this context. Averaged across studies, the percentage of positive responses to questionnaire items relating to auditory-visual synaesthesia induced by psilocybin ranged from 0% for placebo and 45 mcg/kg doses, up to 50% for the highest psilocybin dose of 315 mcg/kg. The linear dosesynaesthesia relationship with psilocybin is also supported by independent data on synaesthesia as one of a number of visual effects in a similar indirect experiment (Griffiths et al., 2011) and tallies with earlier reports of less than 50% of participants experiencing experimentally-induced auditory-visual synaesthesia with 150 mcg/kg of psilocybin (Hartman and Hollister, 1963).

A notable finding of these studies is that psilocybininduced auditory-visual synaesthesia is predicted by different demographic and individual difference variables, most notably *absorption*, the tendency to experience all-encompassing attentional states characterized by intense affective and imaginative involvement in an activity (Tellegen and Atkinson, 1974). Thus, individuals high in absorption appear to be more prone to drug-induced synaesthesia. Other studies administering psychedelics experimentally also report the prevalence of synaesthesia, usually defined as auditory-visual synaesthesia only. These studies have found prevalence rates ranging from 10 to 57% with various substances including ketamine, psilocybin, ayahuasca, *Salvia divinorum*, and *Ipomoea purpurea*, a plant containing LSDlike alkaloids (Savage et al., 1969; Lahti et al., 2001; Riba et al., 2004; Addy, 2010; Carhart-Harris et al., 2011).

Indirect experimental studies largely corroborate the results of the experimental studies. These studies possess a number of the same limitations, most notably the absence of behavioral measures to corroborate synaesthesia. In addition, they are exploratory in design and most studies only report on the prevalence of auditory-visual synaesthesias thereby obscuring our understanding of the types of synaesthesias induced by the chemical agents studied. However, some of the studies (Lahti et al., 2001; Riba et al., 2004; Addy, 2010; Carhart-Harris et al., 2011; Griffiths et al., 2011; Studerus, 2013) benefit from the inclusion of placebo controls, which in part circumvent confounds pertaining to demand characteristics.

### **CASE REPORTS**

We identified 17 case reports exploring the apparent induction of synaesthesia with chemical agents.

The majority of the case studies simply report the form of synaesthesia that is induced, adding only to the known types that may be reported (see **Table 1** and **Figure 1**), but a few studies provide further details regarding phenomenological characteristics of synaesthesia or information regarding chemical agents that can modulate congenital synaesthesia. One notable finding among case reports is Klüver's (1966) observation of variability in the perceived visuospatial location of visual concurrents in induced auditory-color synaesthesia. Specifically, he reported that some participants would experience concurrents as endogenous mental images or representations whereas others would experience concurrents as though they were localized in space. This variability closely mirrors individual differences among synaesthetes. In particular, there is evidence that one subset of grapheme-color synaesthetes experience colors as mental representations (*associators*) whereas another experiences colors as spatially co-localized with the inducing grapheme (*projectors*) (Dixon et al., 2004; Ward et al., 2007). Klüver (1966) further noted that mescaline-induced synaesthesia can sometimes lead to quite complex fusions of several sensory percepts and may even produce associations between abstract concepts (e.g., negation) and visual images (e.g., a white square metal plate). One case report describes the apparent consistent induction of grapheme-color synaesthesia with melatonin, the veracity of which was supported with a behavioral measure of texture segregation (Brang and Ramachandran, 2008).

Of special interest are case reports describing the modulation of congenital synaesthesias with chemical agents. Brang and Ramachandran (2008) described the inhibition of an unspecified form of congenital synaesthesia with two types of antidepressant: the selective serotonin reuptake inhibitor, *fluoxetine*, and the substituted amphetamine, *bupropion*. The former result provides further support for the role of serotonin in the modulation of synaesthesia and thus complements direct and indirect experimental studies pointing to serotonergic agonists as reliable inducers of synaesthesia in non-synaesthetes.

As Shanon (2003) notes, case reports of chemically-induced synaesthesia are typically of the auditory-visual variety, particularly auditory-shape and auditory-color, as occurs with mescaline (Smythies, 1953; Klüver, 1966; Marks, 1975), LSD (Pahnke and Richards, 1966; McKenna, 1982; Hofmann, 1983), cannabis (Marks, 1975), and ayahuasca (Shanon, 2003; Fotiou, 2012). Such auditory imagery is sometimes reported to be dynamic in nature fluctuating with the sounds as they change (e.g., Pahnke and Richards, 1966; Hofmann, 1983), as when listening to music.

# **SURVEYS**

A small number of surveys report on the prevalence and type of chemically-induced synaesthesias. These reports are typically indirect, sampling substance users and reporting on synaesthesia as one of a number of phenomena occurring under the influence of different substances. In one study (DeGracia, 1995), psychedelic substances in general were reported to induce synaesthesia in 50% of users with 62 respondents reporting 23 different types of synaesthesia, the most common of which was auditoryvisual (52% of those reporting synaesthesia), including music and voice inducers. Similar prevalence rates are found in a survey of cannabis users (56%) who predominantly reported musiccolor synaesthesia (Tart, 1975). Even surveys of users of obscure psychedelic substances like 4-HO-MET have reported the induction of synaesthesia, although prevalence rates were not reported (Kjellgren and Soussan, 2011). Notably, lifetime prevalence of chemical induction of synaesthesia is relatively higher than for those induced in individual experimental trials, as might be expected.

We found one study that surveyed synaesthetes regarding the impact of chemical agents on congenital synaesthesia (Cytowic and Eagleman, 2009). This study reports on the relative effects of commonplace drugs among 1279 verified grapheme-color synaesthetes. Alcohol and caffeine tended to enhance (9% both drugs) or reduce (3% caffeine, 6% alcohol) synaesthesia to relatively comparable degrees. Cytowic and Eagleman (2009) also report that of six synaesthetes ingesting LSD, two (33%) reported enhancing effects with the remainder reporting no effect. Although preliminary, these results suggest that LSD has stronger effects than commonplace psychoactive substances.

# **TYPES OF SYNAESTHESIAS**

All but a small number of the studies reporting induction of synaesthesia in non-synaesthetes specified the inducerconcurrent associations (see **Table 1** and **Figure 1**). The most commonly reported type was overwhelmingly auditory-visual

(19 reports; 23%). In turn, the most frequent inducer was auditory stimuli including music and voices (39 reports; 47%), and the most frequent concurrents were visual experiences (43 reports; 52%). However, these figures are somewhat inflated by the exclusive reporting of auditory-visual synaesthesia in the indirect experimental studies. Cumulatively, these results suggest that non-synaesthetes report similar forms of synaesthesia whilst under the influence of certain drugs as congenital synaesthetes, but also other inducer-concurrent associations that to our knowledge have not been reported in research studies of congenital synaesthesia, such as visual-thermal associations.

# **DISCUSSION**

Here we reviewed the known studies describing the apparent induction and modulation of synaesthesia with chemical agents. These studies strongly suggest that chemical agents are capable of producing synaesthesia-like experiences. Crucially, there is large convergence across studies with different methodologies in terms of the prevalence of chemical-induced synaesthesia, the types of drugs (and neurochemical mechanisms) involved, and, to a lesser extent, the types of synaesthesias reported. There is also preliminary evidence that the same drugs have compatible effects on congenital synaesthesia. In what follows, we consider the limitations of these studies, the types of chemical agents implicated, and the types of synaesthesia reported. We conclude by considering whether these phenomena are comparable to congenital synaesthesia and offer suggestions for future research.

# **STUDY LIMITATIONS**

Despite the importance of convergent results, this research literature suffers from a number of substantial limitations, which need to be considered when interpreting the veracity of the chemical induction of synaesthesia and any implications this research has for the neural mechanisms underlying congenital synaesthesia. First, there is a relative paucity of experimental studies. Of these, relatively few included placebo controls and some may be contaminated by demand characteristics. The absence of placebo controls in these studies is especially crucial because there is evidence that various psychedelic drugs enhance suggestibility (for a review, see Whalley and Brooks, 2009) and thus may augment participants' susceptibility to demand characteristics. Furthermore, relatively little information is available regarding dosage, which may be crucial (Studerus, 2013), or the time course of the phenomenon (i.e., onset and duration). There is also considerable variability across participants with only a subset reporting chemical-induced synaesthesia. This variability may be explained in part by individual differences in absorption (Studerus, 2013), but has been largely ignored by researchers. The majority of the reviewed studies describe case reports and surveys. These types of studies are valuable at suggesting research avenues and possible mechanisms, but are not sufficiently rigorous to enable firm conclusions regarding the veracity of the effects.

The most severe limitation of the reviewed studies is that all but one (Brang and Ramachandran, 2008) relied on self-reports of experiences of synaesthesia. Although reports by congenital synaesthetes have been consistently validated by behavioral measures (Ward, 2013), it cannot be assumed that experiential reports among non-synaesthetes under the influence of chemical agents would translate to similar behavioral response patterns as congenital synaesthetes. Across studies, synaesthesia may not have been properly defined to participants or authors may have been using different definitions. Relatedly, some of the reported forms of synaesthesia [e.g., semantic-olfactory (Klüver, 1966); gustatorysemantic, (Breslaw, 1961)] closely resemble states of absorption (e.g., feeling cold when viewing a picture of an iceberg) (Ott, 2007). Given these limitations, caution must be exerted regarding the interpretation of the chemical induction of synaesthesia until these phenomena can be verified using more rigorous behavioral measures in studies with stronger experimental controls.

# **TYPES OF CHEMICAL AGENTS**

The studies reviewed here suggest that a wide range of drugs can produce synaesthesia-like experiences, even in controlled settings. Most of the research has included LSD, mescaline, psilocybin, ayahuasca, or MDMA. A notable commonality among these substances is that they are all serotonin agonists (e.g., Nichols, 2004), specifically serotonin 2A subtype agonists, although some nonserotonergic substances have been reported to induce synaesthesia, such as ketamine (Lahti et al., 2001) and *Salvia divinorum* (Addy, 2010). Few studies have included rigorous comparisons of these different drug classes, but the available evidence suggests that the prevalence rates for synaesthesia under the influence of serotonin agonists is greater than for drugs that do not target serotonin (Luke et al., 2012; Studerus, 2013). Furthermore, Simpson and McKellar (1955) reported that serotonin agonists both induced synaesthesia in non-synaesthetes and enhanced existing synaesthesias in congenital synaesthetes. This result was recently replicated in a survey of psychedelic users and congenital synaesthetes (Luke et al., 2012). Cytowic and Eagleman (2009) also reported the enhancement of congenital synaesthesia with LSD. Cumulatively, these results clearly implicate serotonin in synaesthesia and warrant further research on the role of this neurochemical in synaesthesia.

Nevertheless, other non-serotonergic compounds can also induce synaesthesia, although the most prevalent ones are typically also psychedelic in character, such as ketamine, *Salvia divinorum*, cannabis, nitrous oxide, datura, and dextromethorphan (Tart, 1975; Lahti et al., 2001; Addy, 2010; Luke et al., 2012; Studerus, 2013). These divergent chemical compounds act primarily on different neurochemical systems to each other and yet they all elicit profound changes in consciousness and somewhat similar phenomenological syndromes, such as the experience of synaesthesia. Given that there are currently around 350 known psychedelic chemicals and potentially 2000 untested ones (Luke, 2012), all with different modes of action, understanding exactly what neurochemical processes are responsible for the different experiential effects is a complex conundrum that remains to be disentangled by psychopharmacologists (Presti, 2011). Nevertheless, systemic taxonomical research that relates specific chemicals to particular experiences could illuminate the neurochemistry involved (Luke et al., 2012).

# **TYPES OF SYNAESTHESIA**

There was consistent evidence across studies that auditory-visual synaesthesia was the most commonly experienced form of synaesthesia under the influence of drugs. In turn, auditory stimuli and visual experiences were the most common inducers and concurrents, respectively. However, there is also considerable variability across studies in the types of chemically-induced synaesthesias reported by participants. The source of this variability is at present unclear and may be driven by variability in the types of stimuli presented to participants and environment variability (most studies were not conducted in controlled laboratory environments). The preponderance of sound and music as inducers is likely due to the fact that participants commonly listen to music whilst consuming drugs. A number of studies also report only the prevalence of auditory-visual synaesthesias thereby inflating the differential prevalence rates between this type and other types of synaesthesia. Some phenomena labeled synaesthesia may actually be the result of other factors associated with the drug. For instance, we have found examples where drug-induced hallucinations (e.g., Cooles, 1980) that lack an unequivocal inducerconcurrent association pattern are incorrectly interpreted as synaesthesias (Ballesteros et al., 2006; see also Terhune and Cohen Kadosh, 2012). Finally, an increase in suggestibility following the ingestion of psychedelic drugs (Whalley and Brooks, 2009) may account for the occurrence of synaesthesia-like experiences that appear to be the product of absorption.

A common critique of this literature is that drug-induced synaesthesias tend to differ from congenital synaesthesias (Hubbard and Ramachandran, 2003; Sinke et al., 2012) in terms of the complexity and types of inducer-concurrent associations. The present review only partly supports this conclusion. On the one hand, there are reports of chemically-induced synaesthesias with unusual inducer-concurrent associations that, to our knowledge, have not been reported by congenital synaesthetes. One striking example is Klüver's (1966) report of a complex visual-tactile-conceptual-visual-gustatory-olfactory synaesthesia following the ingestion of mescaline. Similarly, we do not observe any reports of particular well-documented types of synaesthesia such as spatial-sequence synaesthesia (Cohen Kadosh et al., 2012). On the other hand, there are examples of types of induced synaesthesia that are well-documented in the literature including grapheme-color (Brang and Ramachandran, 2008; Luke et al., 2012) and auditory-color synaesthesias (e.g., Hartman and Hollister, 1963). As an aside, we find it especially noteworthy that auditory-visual is the most frequently reported drug-induced synaesthesia as well as the most common acquired form of this condition (Afra et al., 2009). The under-representation of specific types of synaesthesia (e.g., grapheme-color, spatial-sequence) may reflect a lack of exposure to alphanumeric stimuli during drug consumption or, alternatively, may suggest that such types of synaesthesia require exposure to particular associations (e.g., Witthoft and Winawer, 2013) and/or take time to develop (e.g., Simner et al., 2009).

# **CRITERIA OF SYNAESTHESIA**

An open question is whether chemically-induced synaesthesias are equivalent to congenital synaesthesias. Consensus has yet to emerge regarding the principal characteristics of synaesthesia and the ways by which ostensible synaesthesias can be confirmed as genuine. Nevertheless, there is considerable agreement that inducer-concurrent associations are automatic, consistent, specific, and accessible to consciousness (Ward and Mattingley, 2006; Deroy and Spence, 2013; Ward, 2013; Terhune et al., 2014). Here we use these criteria and other characteristics of congenital and chemically-induced synaesthesias to briefly consider the extent to which these two phenomena are similar.

Considered against these criteria, the available evidence indicates that chemically-induced synaesthesias do not as yet qualify as genuine synaesthesia. There is as yet no clear evidence that chemically-induced synaesthesias are automatic as all studies to date have relied on self-report except for one study (Brang and Ramachandran, 2008). The same goes for the criteria of consistency and specificity. There is one study that confirmed the consistency of melatonin-induced grapheme-color synaesthesia (Brang and Ramachandran, 2008), which is suggestive, but far from conclusive. The last criterion, accessibility to consciousness, however, appears to be overwhelmingly met by chemically-induced synaesthesias. These results clearly present a mixed picture but it should be noted that almost none of the studies have actually attempted to validate induced synaesthesias. Accordingly, any strong judgments regarding the veracity of chemically-induced synaesthesias in our opinion remain premature (for a different view, see Hochel and Milán, 2008).

Further evidence points to similarities and differences between congenital and chemically-induced synaesthesias. As previously noted, some researchers have emphasized that congenital synaesthesias tend to be relatively simple associations (Hubbard and Ramachandran, 2003; Sinke et al., 2012) whereas induced synaesthesias are often complex and sometimes reflect inducerconcurrent associations not observed in congenital synaesthesia. As observed in this review, there is considerable heterogeneity in the types of chemically-induced synaesthesias. As noted above, this variability may arise from different factors associated with the respective drug and environmental influences. Although the complexity and dynamism of psychedelic-induced synaesthesia is evident in some reports (e.g., Klüver, 1966) it is not mandatory (e.g., Simpson and McKellar, 1955), and indeed as Sinke et al. (2012) note, the complexity of the visual concurrent experience is related to dose and time from dosing (i.e., the intensity of the drug experience). It is notable that grapheme-color synaesthesia, the most well-studied form of congenital synaesthesia, was also reported by ∼1% of recreational psychedelic tryptamine (e.g., LSD, psilocybin) users in an online survey (Luke et al., 2012). Furthermore, music and other auditory stimuli, the most common inducers across the studies reviewed here, function as inducers in more than 25% of cases of congenital synaesthesia (Hochel and Milán, 2008) (see above). Other notable similarities include individual differences in the visuospatial phenomenology of color concurrents in chemically-induced (Klüver, 1966) and congenital (Dixon et al., 2004; Ward et al., 2007) synaesthesias. Finally, the result that individuals high in absorption are more prone to chemically-induced synaesthesias (Studerus, 2013) is notable because absorption is indiscriminable from fantasyproneness (Rhue and Lynn, 1989) and the fantasizing component of empathy is elevated among congenital synaesthetes (Banissy et al., 2013). Thus, individuals who are prone to chemicallyinduced synaesthesia may have a similar cognitive perceptual personality profile as congenital synaesthetes. To summarize, chemically-induced synaesthesias do not as yet meet accepted criteria for genuine synaesthesia, although no study has attempted to rigorously investigate this question. Induced synaesthesias do, however, display a number of striking similarities to congenital synaesthesias that warrant further attention.

# **FUTURE DIRECTIONS**

The present review shows that there is convergent evidence that particular chemical agents produce synaesthesia-like experiences. However, the studies conducted to date suffer from numerous limitations and many questions remain unaddressed. Here we briefly outline further directions for research on the chemical induction of synaesthesia.

Future research on chemically-induced synaesthesia will need to utilize up-to-date methodologies to confirm that inducedsynaesthesias are not the product of demand characteristics. In particular, there is a strong need for placebo-controlled, double-blind studies of these phenomena. There is consistent evidence that chemically-induced synaesthesias are more common with serotonin agonists, but experimental studies that directly compare a range of chemical agents are required before firm conclusions can be made. Future studies will also need to include established measures to verify the occurrence of synaesthesia such as measures of the automaticity (e.g., Dixon et al., 2004) and consistency (Eagleman et al., 2007; Rothen et al., 2013) of inducer-concurrent associations, rather than relying solely on self-reports as is the norm in the studies to date. Elsewhere, we have noted that some criteria (e.g., consistency) may not be applicable to genuine synaesthesia at an early stage because the associations may have yet to undergo consolidation (Terhune et al., 2014). Specifically, consistency of inducer-concurrent associations in congenital synaesthetes may arise through a consolidation process wherein the inducer and concurrent are repeatedly paired and the association is strengthened and becomes more specific over time. This hypothesis is consistent with research showing that inducer-concurrent consistency increases over time in children with synaesthesia (Simner et al., 2009). This should be considered when assessing the veracity of chemically-induced synaesthesias. Future studies would also benefit from the inclusion of more comprehensive phenomenological inventories in order to identify the similarities and differences between congenital and induced synaesthesias. Finally, it would be valuable to determine using transcranial magnetic stimulation whether drug-induced synaesthesias are dependent upon similar cortical structures (e.g., parietal cortex) as congenital synaesthesias (Esterman et al., 2006; Muggleton et al., 2007; Rothen et al., 2010). Pursuing these lines of investigation will help to elucidate whether chemically-induced synaesthesias are similar to congenital synaesthesias and thereby greatly inform further research on the neurophysiological and neurochemical mechanisms underlying congenital and chemically-induced synaesthesias.

# **SUMMARY AND CONCLUSIONS**

Although it is nearly 170 years since the first report of the pharmacological induction of synaesthesia (Gautier, 1843), research on this topic remains in its infancy. There is consistent, and

# **REFERENCES**


convergent, evidence that a variety of chemical agents, particularly serotonergic agonists, produce synaesthesia-like experiences, but the studies investigating this phenomenon suffer from numerous limitations. The wide array of suggestive findings to date are sufficiently compelling as to warrant future research regarding the characteristics and mechanisms of chemicallyinduced synaesthesias.

# **ACKNOWLEDGMENTS**

Devin B. Terhune is supported by a Marie Curie Intra-European Fellowship within the 7th European Community Framework Programme.

tices/entheogens/docs/kundalini\_ survey.html


hallucinogens," in *Hallucinations; Behavior, Experience, and Theory*, eds R. K. Siegel and L. J. West (New York, NY: Wiley), 9–52.


neurochemical and pharmacological actions of a putative antiaddictive drug. *Pharmacol. Rev.* 47, 235–253.


*New Essays on Synaesthesia*, ed O. Deroy (Oxford: Oxford University Press) (accepted).


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 27 June 2013; accepted: 27 September 2013; published online: 17 October 2013.*

*Citation: Luke DP and Terhune DB (2013) The induction of synaesthesia with chemical agents: a systematic review. Front. Psychol. 4:753. doi: 10.3389/fpsyg.2013.00753*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2013 Luke and Terhune. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# The role of conceptual knowledge in understanding synesthesia: evaluating contemporary findings from a "hub-and-spokes" perspective

# *Rocco Chiou1,2,3 \* and Anina N. Rich1,2,3 \**

<sup>1</sup> Department of Cognitive Science, Macquarie University, Sydney, NSW, Australia

<sup>2</sup> Perception in Action Research Centre, Faculty of Human Sciences, Macquarie University, Sydney, NSW, Australia

<sup>3</sup> ARC Centre of Excellence in Cognition and its Disorders, Macquarie University, Sydney, NSW, Australia

### *Edited by:*

Michael Banissy, Goldsmiths University of London, UK

### *Reviewed by:*

Jean-Michel Hupe, Centre National de la Recherche Scientifique-University Paul Sabatier, France Danko Nikolic, Max Planck Institute for Brain Research, Germany Matthew A. Lambon Ralph, University of Manchester, UK

### *\*Correspondence:*

Rocco Chiou and Anina N. Rich, Department of Cognitive Science, Macquarie University, Sydney, NSW 2019, Australia e-mail: roccochiou@gmail.com; anina.rich@mq.edu.au

Synesthesia is a phenomenon in which stimulation in one sensory modality triggers involuntary experiences typically not associated with that stimulation. Inducing stimuli (inducers) and synesthetic experiences (concurrents) may occur within the same modality (e.g., seeing colors while reading achromatic text) or span across different modalities (e.g., tasting flavors while listening to music). Although there has been considerable progress over the last decade in understanding the cognitive and neural mechanisms of synesthesia, the focus of current neurocognitive models of synesthesia does not encompass many crucial psychophysical characteristics documented in behavioral research. Prominent theories of the neurophysiological basis of synesthesia construe it as a perceptual phenomenon and hence focus primarily on the modality-specific brain regions for perception. Many behavioral studies, however, suggest an essential role for conceptual-level information in synesthesia. For example, there is evidence that synesthetic experience arises subsequent to identification of an inducing stimulus, differs substantially from real perceptual events, can be akin to perceptual memory, and is susceptible to lexical/semantic contexts. These data suggest that neural mechanisms lying beyond the realm of the perceptual cortex (especially the visual system), such as regions subserving conceptual knowledge, may play pivotal roles in the neural architecture of synesthesia. Here we discuss the significance of non-perceptual mechanisms that call for a re-evaluation of the emphasis on synesthesia as a perceptual phenomenon. We also review recent studies which hint that some aspects of synesthesia resemble our general conceptual knowledge for object attributes, at both psychophysical and neural levels. We then present a conceptual-mediation model of synesthesia in which the inducer and concurrent are linked within a conceptual-level representation. This "inducer-to-concurrent" nexus is maintained within a supramodal "hub," while the subjective (bodily) experience of its resultant concurrent (e.g., a color) may then require activation of "spokes" in the perception-related cortices. This hypothesized "hub-and-spoke" structure would engage a distributed network of cortical regions and may account for the full breadth of this intriguing phenomenon.

**Keywords: synesthesia, multisensory integration, attention, color memory, object knowledge**

# **INTRODUCTION**

Although subjective experiences (qualia) vary across individuals, the majority of people seem to perceive the world in a similar way, providing a common ground for communication. For example, we can stand together overlooking the ocean and agree that the water appears to be a shade of blue, we can discuss elements of a lawn being green and brown, and so on. For individuals with *synesthesia*, however, the experience of the environment can be quite different from the rest of us. A synesthete may experience the flavors of different food when reading words in a sentence, or different colors when thinking of the letters of the alphabet. The triggering stimulus (inducer) and resultant experience (concurrent) can be in different sensory modalities (e.g., listening to music elicits a taste), or within the same modality (e.g., reading achromatic text elicits

colors associated with each word or letter). Estimates of the prevalence of synesthesia vary considerably, depending on the definition of synesthesia and the sampling method. If we look specifically at grapheme–color synesthesia (graphemic stimuli, such as letters, digits, the names of days or months and other words, trigger colors), estimates vary between∼0.08% (Rich et al.,2005) and∼2.8% of the general population (Simner et al., 2006; Banissy et al., 2009). Simner et al. (2006) define day-color synesthesia differently from grapheme–color synesthesia, and estimate it occurs in up to 4% of the population, whereas Rich et al. (2005) consider individuals with only colored days as having a variant of grapheme–color synesthesia. Other types of synesthesia (e.g., music induces colors) are less common than grapheme–color synesthesia (for discussion, see Ward, 2013). Synesthesia has become a "hot topic" in

cognitive neuroscience. Although there is an intrinsic fascination with phenomena that reflect extraordinary subjective experiences, this focus on synesthesia also reflects the potential that synesthesia has to provide insights into "normal" mechanisms of cognition and perception.

The primary questions addressed by synesthesia research to date are the level at which an inducer has to be processed for synesthesia to be elicited, and the relationship of synesthetic colors to the "real" colors triggered by wavelengths of light, with neuroimaging studies particularly focusing on the involvement of color-selective regions of the brain (V4). In the present disquisition, we develop the idea that synesthesia may represent an anomaly of *object knowledge*, akin to knowing a banana is yellow. Our starting point is the growing evidence that conceptual-level information is critical for synesthesia to arise and that synesthetic colors may more closely resemble color memory than color perception.

We first examine some influential claims that synesthetic colors are the same as the color experiences induced by wavelengths of light (henceforth "real" or "actual" colors). These studies are often interpreted as demonstrating that synesthesia is a "genuinely perceptual" phenomenon. We clarify the relationship between these behavioral data and the dominant neurocognitive models in the field, with a scrutiny on some crucial aspects of the data that are not taken into account in the major models. We then review recent behavioral and neuroimaging studies that suggest conceptual knowledge is pivotal in our understanding of synesthesia. In particular, these data demonstrate that synesthesia and our general conceptual knowledge of object attributes show similar psychophysical characteristics and may recruit common neural mechanisms. Finally, building upon the literature of "ideasthesia" (i.e., synesthetic experience with its roots in ideas or concepts; see Jürgens and Nikoli´c, 2012), we propose a conceptualization of synesthesia as a benign anomaly of the general mechanisms for representing object concepts. Incorporating the current understanding of semantic memory and its neural substrates, this framework explains many contemporary findings in the field and provides testable hypotheses for future work.

# **PART 1: CONTROVERSIES OVER THE MECHANISMS UNDERLYING SYNESTHESIA**

There are two dominant proposals regarding the neurophysiological basis of synesthesia. The *disinhibited feedback* theory suggests that synesthesia results from a "glitch" of the inhibitory circuitry that fails to suppress crosstalk between brain areas, which is normally inhibited in the non-synesthetic brain. According to different versions of this view, the disinhibition may occur in the feedback from multi-modal regions (e.g., superior temporal sulcus; Grossenbacher and Lovelace, 2001) or from areas involved in executive control (e.g., prefrontal cortex; Cohen Kadosh et al., 2009) to unimodal sensory areas. Some early evidence showing synesthesia-like experience elicited by hallucinogens has been interpreted in favor of this hypothesis such that the drug stifles normal inhibitory neural mechanisms (Grossenbacher and Lovelace, 2001). More recently, Cohen Kadosh et al. (2009) used hypnosis to elicit synesthesia-like experiences in individuals without synesthesia. They hypnotized four non-synesthetes and gave them associations between colors and numbers. These participants were then given post-hypnotic suggestions and performed a visual detection task designed by Smilek et al. (2001). Results showed that hypnotically induced "colors" affected detection of digits. Cohen Kadosh et al. (2009) argue that hypnosis mitigates inhibitory mechanisms, hence allowing a temporary emergence of synesthesia. This intriguing result needs to be considered with a few caveats. First, although there is evidence showing that hypnosis leads to a reduction in regional activation levels of the anterior cingulate cortex (Rainville et al., 2002), it is still unclear how such modulation occurs at the neuronal "excitation vs. inhibition" level. By the same token, the behavioral effect of hypnosis, albeit curious, does not clearly demonstrate alteration in inhibitory circuitries. Second, the induced effect in the hypnotized participants was much larger than that reported by Smilek et al. (2001) in a single synesthete (discussed in more detail in the later Section "Does Synesthesia Depend on Identification of Inducing Stimuli?"), raising the concern that the effects may not be analogous. At this point, therefore, the disinhibited feedback theory remains a plausible mechanism for synesthesia yet lacks direct evidence.

The other dominant view, the *cross-activation* theory, suggests that excessive neural connections between cortical areas underlie synesthetic experiences. In the seminal version of this theory, Ramachandran and Hubbard (2001b) proposed that synesthesia arises from direct cross-wiring of neural tracts between the adjoining areas processing inducers and concurrents. Grapheme–color synesthesia, then, would occur due to the grapheme area directly triggering activation of color area V4 via anomalous connectivity. A revision of this original hypothesis, which acknowledges the contribution of attention in binding color and visual word form, is the two-stage cascaded cross-tuning model (Hubbard, 2007; Brang et al., 2010; Hubbard et al., 2011). The major change is that although the anomalous color is initially generated via crossactivation between the grapheme area and V4, this "feature" still needs subsequent binding to create a unified percept. This binding is proposed to occur at a later stage, mediated by attentional mechanisms of the parietal lobe. This two-stage processing was proposed primarily to explain the mechanisms of grapheme–color synesthesia, but has recently been extended to account for other forms of synesthesia (Hubbard et al., 2011). A similar proposal to the cross-activation model is the re-entrant processing model (Smilek et al., 2001). In addition to "cross-talk" between relevant brain areas (e.g., areas processing form and color), this proposal includes feedback from higher-level areas (such as the anterior inferior temporal lobe) that represent the concept or meaning of the inducer (Smilek et al., 2001; Hubbard et al., 2011). This version of the cross-activation hypothesis has received less attention to date, but serves as a starting point for our discussion of the potential for higher-level areas to link inducers and concurrents.

The emphasis in the two dominant theories has very much been on the perceptual side of the system. This is also reflected in the primary focus of many neuroimaging studies of synesthesia: many studies have been designed to test whether synesthetic color is handled by a color-selective section of the fusiform gyrus (i.e.,V4), with inconsistent results. Some studies find that synesthetic color correlates with V4 activation in some synesthetes and interpret

this as synesthetic experiences resembling actual perceptual events (e.g., Hubbard et al., 2005; Sperling et al., 2006), whereas other studies fail to find a V4 response to synesthetic color (e.g., Rich et al., 2006; Hupé et al., 2012; Sinke et al., 2012b). There is also inconsistency between behavioral studies, with some claiming that synesthetic color behaves just like real color (e.g., Ramachandran and Hubbard, 2001a; Kim et al., 2006) and others discovering clear differences between the two types of colors (e.g., Edquist et al., 2006; Hong and Blake, 2008; Arnold et al., 2012). We review the relevant behavioral and neuroimaging studies that contribute to this debate in the following sections.

Most of the evidence reviewed here is based on grapheme– color synesthesia, due to it being the most widely-studied form (Rich et al., 2005; Simner et al., 2006; Novich et al., 2011). We do not, as yet, have a good basis for determining the likelihood that different forms of synesthesia differ fundamentally in their underlying mechanisms, but here we attempt to conceptualize a general form of the phenomena.

# **DOES SYNESTHESIA DEPEND ON IDENTIFICATION OF INDUCING STIMULI?**

One of the key issues in gauging the role conceptual processing has in synesthesia is to determine how "deeply" an inducer needs to be processed for synesthesia to arise. If synesthesia is triggered by "low-level" physical features (e.g., Ramachandran and Hubbard, 2001b), changing the appearance of an inducer should result in alteration of synesthetic percepts. If, in contrast, it is the meaning or the concept that elicits synesthesia, any stimulus that is identified as a particular inducer (e.g., A, a, and index the same concept) should elicit the same color. Subjective reports are mixed on which of these descriptions is accurate.

Rich et al. (2005) reported, based on questionnaire responses from 150 grapheme–color synesthetes, that the anomalous color experiences typically occur regardless of the modality of the inducer (seen, heard, or even just thought about). Moreover, changes in the appearance of an inducer (font, size, etc.) were only occasionally reported to impact on synesthetic color, suggesting it may be the lexical representation triggering color experiences rather than the physical appearance. In addition, the semantic context of an item has a strong effect on the perceived color: an identical stimulus is reported to elicit different colors depending whether it is interpreted as a digit or a letter under different contexts (e.g., 8**o** vs. g**o**; Myles et al., 2003; also see Dixon et al., 2006). By contrast, other synesthetes have reported subtle differences in the saturation of color for different fonts or cases (e.g., E vs. E or A vs. a; see Witthoft and Winawer, 2006) or only have synesthesia for visually presented but not spoken letters (Arnold et al., 2012). In our large-scale surveys, however, this occurs for a very small proportion of synesthetes, as is also reported by Simner (2012). In general, for the majority of synesthetes, it is the concept of a letter that determines the color rather than its low-level visual characteristics.

Dixon et al. (2000) objectively verified that synesthesia could be elicited by the concept of an inducer. They presented a digit (e.g., 4), an operator (e.g., +), a second digit (e.g., 3), and finally a color patch. They had a synesthete, "C," name the color

patch as quickly as possible, and then report the solution to the arithmetic problem. C showed a reliable congruency effect – her naming times were slower when the display color of the target patch was incongruent with the synesthetic color induced by the arithmetic solution, compared to when they were congruent. This indicates that synesthetic color can be triggered by a mental concept, without the inducing stimulus being physically present.

In a subsequent study by the same research team, Smilek et al. (2001) briefly presented a black digit on a colored background that either matched or mismatched C's synesthetic color for the digit. C was poorer at identifying the digit when it was presented on a congruently colored background relative to an incongruent one, as if it was camouflaged by the background color. Some aspects of these data are puzzling. For example, if the digit could be effectively concealed by the synesthetic color matching the display background color, C should perform worse than non-synesthetic controls. However, C's performance in the congruent condition was actually comparable to controls; her performance was just better than controls in the incongruent condition. Based on their data, Smilek et al. (2001) propose that synesthesia arises when the concept/meaning of an inducer is activated but does *not* require conscious identification.

In contrast to the results of Smilek et al. (2001), Mattingley et al. (2001) found that synesthesia depended critically on *conscious* identification of the inducer. They presented 15 synesthetes with a variant of a synesthetic congruency task (Wollen and Ruggiero, 1983) in which the synesthetic color elicited by a letter inducer prime was either congruent or incongruent with a succeeding target color to be named. They showed a robust synesthetic priming effect from the letter prime on subsequent color naming times when the prime was clearly visible. This priming effect disappeared when the inducer prime was rendered invisible by a backward mask. The unseen primes still elicited a priming effect on a letter naming task, indicating some unconscious processing of the letter, at least to the level of orthographic representation. Mattingley et al. (2001) interpreted these results as synesthetic color only arising when the inducer is consciously identified. Later findings have been consistent with this claim, demonstrating that synesthetic congruency effects can be abolished when attentional resources are not sufficient for conscious report of a letter, such as in the middle of the attentional blink (Rich and Mattingley, 2010). Concurrently, other studies have demonstrated that reducing attentional resources available for processing the inducer (presumably hampering identification) attenuates the otherwise reliable impact of synesthetic color on color naming (Rich and Mattingley, 2003; Mattingley et al., 2006; Sagiv et al., 2006).

The studies reviewed above suggest that access to the conceptual representation of an inducer plays a critical role in eliciting synesthesia. However, the extent to which meaning without full conscious identification can elicit synesthesia is unclear. Smilek et al. (2001) emphasize that access to the concept/meaning of the inducer should be sufficient to trigger synesthetic colors. Access to lexical or pictorial meaning can occur prior to identification, leading to masked priming effects (e.g., Dell'Acqua and Grainger, 1999). In synesthesia, however, the masked priming data (Mattingley et al., 2001) suggest that implicit processing without conscious identification is not enough to elicit synesthesia (at least, not sufficient to cause a reliably detectable synesthetic congruency effect). Thus, it seems clear that access to a concept is important, and that the depth of processing needs to be considerable (probably to full conscious identification) before synesthesia arises.

## **IS SYNESTHETIC COLOR EQUIVALENT TO REAL COLOR?**

Another major focus of synesthesia research has been whether induced synesthetic color is equivalent to "real" color. Subjective reports again provide mixed information about this. On the one hand, synesthetes sometimes report that their synesthetic colors can be as conspicuous as real colors (Ramachandran and Hubbard, 2001a,b), and on the other hand, they can clearly distinguish synesthetic colors from the actual colors of triggering graphemes. Despite some influential claims in early studies, objective data are generally consistent in demonstrating a difference between synesthetic and real colors.

In an influential study, Ramachandran and Hubbard (2001a) demonstrated that synesthetic color facilitated visual search – a unique synesthetic color caused an obscure shape to "pop out" from surrounding items of a visual display. It is well-established that actual color causes "pop-out" during visual search: when a target has a unique color, the number of distractors has minimal effect on the reaction time (RT) to detect the target (Treisman and Gelade, 1980). This efficient search usually leads to a shallow or flat search slope of the RT by set-size function, demonstrating that color is registered rapidly into its sensory channels and can be effectively used to guide attention. In their well-known synesthetic "pop-out" study, Ramachandran and Hubbard (2001a) had two synesthetes search through an array of black digits for an embedded contour composed of identical digits. Compared to non-synesthetic controls, the synesthetes were more accurate in identifying the embedded target (controls: 59.40%; synesthetes: 81.25%; chance level was 25%). In a later paper (Ramachandran and Hubbard, 2001b), the authors interpreted this result as synesthetic color "popping out" like real color and facilitating visual search.

A synesthetic pop-out effect would be provocative, as it conflicts with what is known about the role of attention in letter recognition (e.g., Mackeben, 1999), and indeed, with our claim that synesthetic color arises *after* attentional processing of the inducer (e.g., Mattingley et al., 2006; Rich and Mattingley, 2010). Subsequent visual search studies, however, found that although synesthetic color occasionally bestows an advantage during visual search, it does *not* lead to the pop-out effect. Palmeri et al. (2002) found a synesthete outperformed controls during a traditional visual search task with a single target among multiple homogeneous distractors. However, the slope of the RT by set-size function was much steeper than the typical flat slope of the pop-out effect. Moreover, the effect only occurred when the *distractors* elicited synesthesia, making it clear the effect was *not* related to the synesthetic color of the target. Edquist et al. (2006) and Sagiv et al. (2006) also had synesthetes perform visual search tasks. In neither study (sample size: 14 synesthetes by Edquist et al., 2006 and 2 synesthetes by Sagiv et al., 2006) was there any evidence of a synesthetic advantage over non-synesthetic controls. Whereas a

unique chromatic letter was immediately manifest, an achromatic letter could only be found via an effortful search that did not differ from controls, indicating *no* synesthetic pop-out. Laeng et al. (2004) also tested a synesthete using a visual search task. Initially, these authors found that synesthesia could lead to an advantage, but only when the target was within the central focus of attention. A later study by Laeng (2009) suggested that pop-out could occur if the target and distractor letters were carefully chosen to elicit colors far distant in color space. This remains the only case to date that resembles a truly efficient search function.

Returning to the embedded figures task, Ward et al. (2010) tested 36 synesthetes and 36 controls and recorded synesthetes' trial-by-trial descriptions of their synesthetic experiences. They observed a general superiority of synesthetes in accuracy (synesthetes: 41.4%; controls: 31.5%), replicating the original Ramachandran and Hubbard (2001a) case-study results, albeit with a smaller (∼10%) advantage for synesthetes. However, most of their synesthetes reported *no* color experience during search, and synesthetic color only seemed to emerge when a letter was attended. They concluded this was inconsistent with "pop-out," based on the subjective reports and the fact that the accuracy advantage for synesthetes was far less than one would expect from actually colored stimuli. Rich and Karstoft (2013) modified the embedded figure task to manipulate set-size, allowing a direct objective test of the pop-out hypothesis. With eight synesthetes and eight matched controls, they replicated the advantage for synesthetes over controls in accuracy at the largest set-size (64 items; synesthetes: 1.87% errors; controls: 11.56% errors) but found no differences in RTs/search slopes that would be indicative of a synesthetic "pop-out" effect. At this point, despite subtle advantages in some visual search studies for some synesthetes, there is little evidence that synesthetic color pops out like real color.

There are alternative explanations to explain the occasional advantage of synesthetes in visual search-type tasks. For example, Mattingley (2009) raised a possibility that synesthetic color may facilitate the grouping of items and confine search scope (also see Ward et al., 2010; Rich and Karstoft, 2013 for discussion of the grouping account). In addition, a recent study (Rappaport et al., 2013) reported that coloring an object with its familiar color (e.g., yellow corn) led to more efficient visual search (slope: 10 ms/item), compared to it being in an atypical color (22 ms/item). Note that the facilitation by color familiarity/memory is still inferior to popout searchesfor real color, which often gives slopes close to 0 (Wolfe and Horowitz, 2004). Such findings resemble the pattern seen in synesthetic color improving visual search, implying that the facilitation originates from well-learnt conjunctive representations, be it synesthetic or ordinary mnemonic, reducing attentional constraints but causing no pop-out. Furthermore, as we will discuss later, the resemblance raises the hypothesis that the human brain employs the same mechanisms to represent synesthetic color and the knowledge about canonical object color.

Other studies using psychophysical measures of low-level visual properties also suggest that synesthetic color is distinct from real color. For example, synesthetic color does *not* induce chromatic adaptation (Hong and Blake, 2008), the simultaneous color contrast effect (Nijboer et al., 2011a), or the color constancy effect

(Erskine et al., 2012). Although there are some claims that synesthetic color behaves like actual color to affect bistable apparent motion (Kim et al., 2006) and to induce the watercolor illusion (Kim and Blake, 2005), these studies tested only two synesthetes and used designs that are not criterion-free. For instance, Kim et al. (2006) asked the synesthetes to decide whether two alternating visual arrays of letters appeared to move leftward or rightward and found that letters that induced the same color tended to be "grouped" together, which biased motion trajectory of the arrays. However, the need to select between two alternatives could result in an effect that simply reflects the tendency to use color similarity to guide a decision. With this design, there is no way of telling whether synesthetic color truly biases perception or simply the decisional criterion. Overall, there are clear differences between synesthetic color and real color, suggesting there must be crucial differences in the mechanisms that underpin the two forms of color experiences.

# **CONTROVERSIES OVER THE CATEGORIZATION OF SYNESTHETES**

One explanation for the discrepant results regarding how closely synesthetic color resembles real color is that different mechanisms underpin synesthesia for different sub-groups, and, for at least some synesthetes, their evoked colors *are* akin to "real" colors. Dixon et al. (2004) distinguish "projectors," who describe synesthetic color as appearing "out in space," from "associators," who describe the color as "in the mind's eye." They suggest that, compared to associators, the synesthetic color of projectors emerges earlier in the visual system. In another proposal, Ramachandran and Hubbard (2001b) differentiate between"lower synesthetes," for whom synesthesia is induced by sensory stimulation, and "higher synesthetes," for whom it is caused by the meaning of the inducer. In lower synesthetes, the cross-wiring between the grapheme area and the color center (V4) is assumed to generate their synesthetic color whereas, in higher synesthetes, the wiring to the parietal lobe produces the anomalous color. These categorizations were originally proposed to reconcile contradictory results. However, they have now generated more debate.

Dixon et al. (2004) tested whether they could objectively distinguish "projectors" from "associators" using two versions of the synesthetic congruency task – naming the physical color of a letter or naming its synesthetic color. The display color and the synesthetic color on a given trial could either match (congruent) or conflict (incongruent). Their synesthetes were first separated by subjective report (seven associators and five projectors). Results showed that those classified as projectors had a larger congruency effect (incongruent minus congruent RT) when naming displayed color than synesthetic color, whereas associators showed the opposite pattern (see Ward et al., 2007 for similar demonstrations with seven associators and seven projectors). However, other studies have failed to find differences between subjectively classified projectors and associators on visual search tasks. For example, Edquist et al. (2006) tested 14 synesthetes and found that their anomalous colors did not affect performance on a visual search task, regardless of whether the synesthete identified his/herself as an associator or a projector (based on initial responses to just the question of whether the color appeared "out in space" vs. "in my

mind's eye," there were five projectors, seven associators, and two subjects who reported that neither of these fitted their experiences; also see below for the variation of reports across time). Sagiv et al. (2006) also tested two self-reported projectors on a visual search task and found no advantage from synesthetic colors. In a more recent study with a large sample size (27 associators and 9 projectors, classified based on responses to questionnaires), Ward et al. (2010) also failed to find any difference between the two types of synesthetes on a visual search task. As a final example, Nijboer et al. (2011b; also see Nijboer and Van der Stigchel, 2009) asked nine synesthetes (six associators and three projectors) to execute an eye movement to a target digit during visual search. The results showed that a target that usually elicited a synesthetic color did not lead to the pop-out effect (indexed by faster saccadic onset) in either the associators or the projectors. One further issue with this dichotomy is that the subjective reports of "in the mind's eye" vs. "out in space" may be unreliable, switching from one description to the other between two points in time (Edquist et al., 2006). This variability presumably reflects the difficulty in using language to describe the subjective experience. Thus, although it is always possible that for some synesthetic individuals synesthetic colors do behave like real colors, there is little evidence of reliable behavioral differences at this point.

Some neuroimaging studies have claimed to lend support to the "associator vs. projector" distinction. Rouw and Scholte (2010) used voxel-based morphometry (VBM) to compare the volume and density of neuronal gray matter (GM) between 16 projectors and 26 associators, compared with 42 non-synesthetic controls. They also used functional magnetic resonance imaging (fMRI) to look at the functional response to graphemes (contrasting colorinducing graphemes vs. non-inducing symbols), relative to 19 non-synesthetic controls. Synesthetes were classified as projectors and associators based on their scores on a questionnaire probing the subjective locus of their synesthetic colors. In structural measures, projectors had increased GM in the cortical areas for perception and action (i.e., the visual cortex and other sensorimotor areas), whereas associators had more GM in the regions involved in memory encoding and retrieval (i.e., the hippocampus and parahippocampal gyri). Their fMRI data also revealed that, compared to projectors, associators showed greater activity in the bilateral parahippocampal gyri, which mapped to the same regions that showed structural alternation in the VBM analyses. The authors suggest that such neuroanatomical differences drive the individual variability in the subjective locus of synesthetic color (internal "in the mind's eye" vs. external "out in space"). Although this inference is intriguing, others have raised concerns about the stringency of the analyses. Hupé et al. (2012) note that it is not clear whether Rouw and Scholte (2007, 2010) applied correction for multiple comparisons to data over the whole brain and suggest that not including brain size as a covariate can be problematic. In addition to the question about whether the analyses have enough rigor, there is conflicting evidence regarding the "distribution" of the two hypothesized forms of synesthetes: in an earlier study by the same authors, Rouw and Scholte (2007) tested 18 synesthetes and found their scores on the "associator vs. projector" dimension appeared more evenly distributed along the spectrum and formed a positive correlation with the fractional anisotropy (an index of fiber density or directional coherence in white matter tracts) of the right temporal cortex, implying the variability is continuous (also see Eagleman, 2012). In their later study (Rouw and Scholte, 2010), which included these 18 synesthetes in a final sample of 42, the distribution of self-reports looked more bi-modal rather than falling along a continuum. The Rouw and Scholte (2010) paper remains the most comprehensive imaging paper in synesthesia research to date. Other neuroimaging studies using fMRI (van Leeuwen et al., 2010)<sup>1</sup> and electroencephalography (EEG; Gebuis et al., 2009) have failed to find reliable differences between associators and projectors in regional haemodynamic responses and electrophysiological waveforms. Overall, we tend to agree with Eagleman (2012) that it is difficult to know whether we are looking at the extremes of a continuum or a categorical distinction between two groups differing fundamentally in the spatial frame of their synesthetic colors.

Compared to the "projector vs. association" distinction, there is less empirical data on the "lower vs. higher" categorization of synesthetes. Hubbard et al. (2005) emphasized the potential importance of individual differences in their fMRI study. They tested six synesthetes using their embedded figures task (discussed previously) and a crowding task. In the latter task, participants had to identify a target letter flanked by distractors in the visual periphery, with all stimuli being achromatic. Identifying the target was difficult as distractors "crowded" the target, rendering it unrecognizable. In the embedded figures task, five of the six synesthetes performed better than their controls in their accuracy of identifying the embedded shape. For the crowding task, three of the six synesthetes had a slight advantage over their controls in correctly identifying targets, which the authors interpreted as synesthetic color mitigating visual crowding (note that no difference was found at the group level). The authors then examined the response of visual cortical regions in synesthetes and controls presented with achromatic graphemes. They reported significantly higher V4 activation in synesthetes than controls (albeit with a liberal statistical approach, a one-tailed test that assumes *a priori* higher V4 activity in synesthetes than controls; see Hupé et al., 2012). Further, the authors found a significant correlation between performance on the crowding task and the strength of V4 activity such that synesthetes with greater V4 activation tended to perform better on the visual crowding task. The authors propose that individual differences in the ability to perform the crowding task map onto the "lower" (better performance, more V4 activity) vs. "higher" (average performance, less V4 activity) synesthete distinction2. As they note, however, it is not possible with such a small sample size to distinguish sub-groups from polar extremes of a continuum.

There is a general consensus among researchers that the salience or intensity of synesthetic experience varies between individuals. The challenge lies in finding whether this variation truly indicates fundamental differences, perhaps qualitatively involving distinct neural mechanisms, such that we should categorize synesthetes into sub-groups, or whether it reflects individual variability within the same basic mechanism.

### **DEBATES CONCERNING THE NEURAL BASIS OF SYNESTHESIA**

The notion that there must be a difference in neuronal "wiring" between synesthetes and non-synesthetes has been popular among scientists and the general public alike. As discussed earlier, the *disinhibited feedback* theory suggests that synesthesia results from a deficiency of inter-neuronal inhibition. The evidence for this theory (e.g., synesthesia-like experience elicited by drug or hypnosis) is all behavioral and does not provide direct support for the existence of malfunctioning neuronal inhibition. Further, a recent paper systematically compared developmental synesthesia to druginduced similar experiences and found that the two have more differences than commonalities (Sinke et al., 2012a). Neuroimaging studies have provided more direct examination of the predictions of the *cross-activation* theory. In the case of grapheme–color synesthesia, the cross-activation view assumes that synesthetic color is first induced by the grapheme area directly triggering activation in V4, the specialized area for processing color; at a subsequent stage, the attentional mechanisms mediated by the parietal lobe act as "glue" to combine synesthetic color with visual word form. Although the involvement of the parietal lobe in synesthesia is generally agreed upon amongst researchers, there is debate about the first hypothesized stage. There are two principle points of contention.

The first issue with the cross-activation theory is that there is still debate about whether the human brain really hosts a cortical region specialized for processing visual word form. Reading text activates the left lateral occipitotemporal sulcus/the left fusiform gyrus, a site often referred to as the visual word form area (VWFA; see Cohen et al., 2002; McCandliss et al., 2003). This site is reliably activated by graphemic stimuli, regardless of different scripts or languages. Moreover, brain lesions of the VWFA cause pure alexia, a selective deficit in word recognition (Cohen et al., 2000). Such evidence has led to the theory that this left ventral occipitotemporal section of the brain is specialized for conjoining graphemic features into a word shape, prior to the stimulus accessing its phonological and semantic codes (Dehaene and Cohen, 2011). However, the specificity of this area to graphemic processing has been challenged. For instance, Price and Devlin (2003) found that this "visual word form" area could also be activated by images of animals and vehicles. Devlin et al. (2006) also found that this area responded to the semantic relations between words, suggesting that it does not exclusively represent visual word form. Two recent studies explored the functionally connectivity of the VWFA (Vogel et al., 2012a) and contrasted its response to words and line drawings (Vogel et al., 2012b). Results showed that the activity of this region did not functionally co-vary with other reading-related regions and showed no preferential response to words, which the authors interpret as evidence that it is a more general visual processor used in reading but also in other visual tasks. Given the debate,

<sup>1</sup>van Leeuwen et al. (2010) categorized 21 synesthetes into associators or projectors using a questionnaire. In their fMRI experiments, none of their region of interest analyses revealed a reliable difference between the two groups. Although they reported associators showed "marginally" higher (*p* = 0.09) activation in a left parietal lobule cluster than did projectors, these effects appeared rather weak as they were obtained using planned tests (due to null results of their main analyses) with liberal criteria (i.e., α = 0.05, not corrected for multiple comparisons).

<sup>2</sup>Although Hubbard et al. (2005) draw a parallel between their "higher/lower" synesthetes and the "associator/projector" distinction discussed previously, Ward et al. (2007) later found these categorizations to be orthogonal.

it may be premature to conclude the putative contribution of this region to synesthesia is, as the cross-activation view assumes, the visual analysis of graphemic components. It is possible it reflects non-perceptual processing, such as the semantic representation of a stimulus.

The second, perhaps more problematic, issue for the crossactivation account is whether synesthetic color recruits V4. Some studies have observed V4 activation when synesthetes view or hear graphemes (Nunn et al., 2002; Hubbard et al., 2005; Sperling et al., 2006). However, other studies did not find V4 activation during synesthesia (Rich et al., 2006; Hupé et al., 2012; Sinke et al., 2012b). Hupé et al. (2012) conducted a thorough review of the literature to assess the origin of these discrepancies; they claim that all imaging studies in synesthesia that report V4 activation in synesthesia use overly liberal statistical criteria. In the study by Nunn et al. (2002), the areas activated by synesthetic color did *not* actually overlap with those by real color. In the study of Sperling et al. (2006), there was no correction for multiple comparisons for statistical tests; despite this liberal criterion, they found V4 activity in only two out of four synesthetes. Finally, in Hubbard et al. (2005), the criterion was also liberal (one-tailed tests assuming higher V4 activation in synesthetes than in controls). Of course, these statistical issues are not limited to studies reporting V4 activation, but the Hupé et al. (2012) critique emphasizes the potential for such analysis decisions to lead to a misperception of the robustness of such findings. It is therefore by no means unequivocal from the data presented to date that synesthetic color activates V4.

In addition to the issue of overly liberal statistical criterion, Mattingley (2009) speculates that the greater V4 activation in synesthetes in some studies may be confounded by mental imagery. Various lines of inquiry seem to give credence to this suggestion. Barnett and Newell (2008) reported that synesthetes (*n* = 38) scored higher than controls on the Vividness of Visual Imagery Questionnaire (VVIQ), a test probing the vividness of their visual imagery. Although this merely suggests that synesthetes may have more vivid imagery and an inclination to imagine, there are two pieces of evidence linking this behavioral result to neural processing. First, Cui et al. (2007) showed that individuals (*n* = 8) who score higher on VVIQ tend to show higher activity in the early visual cortex (Brodmann's areas 17 and 18, regions of interest (ROIs) defined using a neuroanatomical database) when performing mental imagery. Second, Rich et al. (2006) used a color perception localizer (colored minus gray-scale Mondrians) to define ROIs in the visual cortex responsive to colored stimuli in seven synesthetes and matched controls. Within these colorresponsive ROIs (voxels passing small volume correction), they found that voluntary color imagery resulted in activation for both synesthetes and controls in the fusiform gyrus, a location consistent with the literature definition of V4. These findings together imply that V4 activation in synesthesia studies could potentially reflect synesthetes imagining color, rather than synesthetic color *per se*. A recent study lent further support to this imagery explanation of V4 activity: Sinke et al. (2012b)found that when synesthetes and controls were matched for their scores on VVIQ, there was no difference in V4 activation between the two groups. That is not to say that synesthesia is only voluntary imagery, but just that

in a scanning environment where synesthesia is the focus, participants who are trying hard might utilize color imagery, which consequently activates V4. We need more evidence before concluding that synesthesia originates from the cross-wiring between V4 and grapheme area, and that synesthesia and "real" color share neural bases.

Thus far, the data we have reviewed shows a discrepancy between behavioral data and the assumptions made by the dominant theories about the neural mechanisms of synesthesia. We argue that behavioral evidence suggests that synesthesia is strongly reliant upon and influenced by the *conceptual* representation of inducers and has attributes that are distinct from those of actual color. The neurocognitive theories of synesthesia (especially the cross-activation view) focus on the grapheme area and V4, postulating that synesthesia depends on the feature-level (lines, intersections, etc.) perceptual processing of a grapheme and resembles real color. We need a framework for synesthesia that is better able to integrate a broader gamut of behavioral and neuroimaging findings. In the following section, we expand upon the role conceptual knowledge has in synesthesia, and propose a conceptual-mediation neurocognitive framework.

# **PART 2: THE ROLE OF CONCEPTUAL KNOWLEDGE IN SYNESTHESIA**

As reviewed in Part 1, there is good evidence that substantial processing of an inducing stimulus, including access to meaning/concept, is necessary to elicit synesthesia (e.g., Dixon et al., 2000; Mattingley et al., 2001; Rich et al., 2005; Nijboer and Van der Stigchel, 2009; Rich and Mattingley, 2010; Nijboer et al., 2011b; Simner, 2012). Research into the nature and effects of synesthetic experience has also highlighted distinctions between synesthetic and real colors in terms of their psychophysical characteristics (e.g., Edquist et al., 2006; Sagiv et al., 2006; Hong and Blake, 2008; Arnold et al., 2012; Erskine et al., 2012). The overall pattern of contemporary findings suggests a need for a framework that emphasizes the role of high-level processing in synesthesia, such as conceptual knowledge.

**Figure 1A**shows a schematic illustration of our proposed stages involved in the elicitation of synesthetic color: initially attentive processing of an inducer leads to perceptual analyses of its physical attributes (e.g., lines, curves, conjunctions, etc.). A series of perceptual processes culminates in the identification of that stimulus at a conceptual level, regardless of the modality of presentation. The action of identification may differ between stimulus types (e.g., letter identification is individuating the lexical identity, whereas for a sound there can be the awareness that a sound has occurred, recognition of instrument or speaker, through to individuation of that sound with a label, such as *middle C* or *F-sharp minor*). To this point, processing in synesthetes and nonsynesthetes should be identical. Activation of the stimulus concept spreads through a semantic network: the activation propagates from the core conceptual representation of the inducer to its associated nodes of conceptual attributes, which for synesthetes, includes a link to the concept of a color absent (or weak) in nonsynesthetes. We conceptualize this as a similar linkage to the way in which the concept of a banana includes its canonical color

stream, attention to the inducer letter "A" serves as the gatekeeper for subsequent analyses. The perceptual system then analyses how lines are conjoined, deriving the visual word form of "A." The visual representation of "A" subsequently activates the core lexical concept of

banana and its related conceptual features; **(C)** the salience/vividness of synesthetic color may depend on the strength or directness of the link between the lexical presentation of "A" and its synesthetically red color.

(**Figure 1B**). The end-point of this cascade of cognitive processing is an experience of synesthetic color, with variability in the individual experience from a "*feeling*" of color to a strong "*percept*" of color, perhaps depending on the strength and distance of the connections within the neural network. A strong link bridging the ensemble of neurons coding lexical meaning in the semantic network to those coding color in the perceptual areas might travel through the occipitotemporal regions involved in color-related processes representing *memorized* and/or *perceived* color, resulting in a vivid percept of color. By contrast, a weaker link connecting lexical and color representation within the semantic network, or one that persists only partway down the system toward the perceptual areas, may only give a faint percept or a "feeling" of color without clear visual components (**Figure 1C**).

How can this conceptual-mediation model account for the range of behavioral and neural data? First, the role of attention is clear. Without attention to an inducer, there is no identification, and therefore the process does not kick off. Second, the abstract identity of the inducer is linked to its synesthetic color at a conceptual level and the processing propagates back down the network toward more perceptual representations. This framework encompasses the top-down effects including semantic context, as well as allowing for the individual variability in the "percept-like-ness" of the experience. Finally, as elaborated later, when non-synesthetes are asked to make voluntary associations between (for example) sounds and colors, there are systematic patterns that are strikingly similar to those seen in the inducer-concurrent pairings of synesthetes. There is evidence that these trends of cross-modal mappings observed in non-synesthete are mediated by semantic knowledge, again suggesting it may be a groundwork subserving synesthetic and ordinary phenomena. We propose that synesthesia has cognitive roots in the semantic knowledge base that we all share, rather than requiring a fundamentally different neural network of connections.

One prediction of the conceptual-mediation model is that synesthetic colors should be more similar to high-level color knowledge than to perceiving "real" color, because the critical link occurs between concepts – the link is between, for instance, the *concepts* of lexical entity "A" and perceptual property "red." There are a few recent studies suggesting that this may indeed be the case. Results of recent fMRI and brain stimulation studies also hint that synesthetic color and color knowledge may recruit common neural substrates, beyond the realm of V4. In the following sections, we review this recent evidence and further discuss the proposal that synesthesia relies on the cognitive and neural mechanisms of conceptual knowledge. We also outline how this conceptualization can better explain why non-synesthetes show strikingly similar trends to the unusual experiences of synesthetes when asked to associate graphemes/sounds with visual attributes, and the implicit cross-modal mappings we all share.

# **GRAPHEME–COLOR SYNESTHESIA RESEMBLES COLOR KNOWLEDGE/MEMORY**

The conceptual-mediation model predicts that synesthetic colors should more closely resemble high-level color knowledge, where color is an intrinsic attribute contributing to the concept of an object (e.g., knowing a banana is yellow) than low-level color perception (e.g., perceiving a shade of yellow hue). Nijboer et al. (2011a) tested whether synesthetic color could elicit the simultaneous color contrast effect, a phenomenon reflecting early visual processing. They presented an achromatic grapheme on a colored background and had 12 synesthetes adjust the color of a white hash sign to match their synesthetic color. Twelve controls performed the same task with physically colored graphemes (matched to their synesthetic counterparts). The key manipulation was that the background and grapheme could have colors similar in hue or opponent colors. Controls demonstrated the typical simultaneous color contrast effect (Valberg and Lange-Malecki, 1990): when the grapheme and background had opponent colors (e.g., a green background and a red grapheme), responses tended to "offset" the background color. Specifically, a green background made a red grapheme seem redder than it actually was (color contrast), so controls added more red than what was present in the reference stimulus (i.e., grapheme). Synesthetes showed an inverse pattern to that of controls such that their responses tended to "conform" to the background color. When a grapheme induced a *red* synesthetic color appearing on a *green* background (opponent colors), synesthetes added insufficient red to match their synesthetic color. When it was a *red* synesthetic color on a *red* background (similar colors), they added excessive red.

The absence of the typical color contrast effect from synesthetic colors concurs with earlier studies that early mechanisms do not contribute to synesthesia (e.g., Hong and Blake, 2008). However, it is the reverse pattern of synesthetic colors relative to the typical effect of real color that raises an interesting possibility: synesthetic color may be an anomalous case of *object color knowledge* (e.g., knowing that bananas are typically yellow). There is evidence that color knowledge can affect perceived color appearance (Hansen et al., 2006), and also that the representation of high-level color memory is less precise then real color (Knill and Richards, 1996). Thus, Nijboer et al. (2011a) suggest that, similar to color knowledge, the perceptual representation of synesthetic color may be less solid and tend to fluctuate more easily than real color. This difference in perceptual characteristics relative to real color may lead to synesthetic colors assimilating into the background, resulting in the reverse pattern. To explain the potentially pliable nature of synesthetic color, Nijboer et al. (2011a) drew an analogy to the impact of perceptual noise on shape perception: when the contour of a stimulus is obscured by noise and becomes less concrete, shape representation gradually assimilates into the clearly defined background (Van der Kooij and te Pas, 2009). Although it would be more cogent to have a direct comparison between synesthetic colors and general object knowledge with regard to the effect on perceived color appearance, these data support the notion that high-level color knowledge could form the basis for synesthetic colors.

The idea that synesthetic colors may resemble a higher-level color representation such as color knowledge or color memory has been directly addressed in another recent study. Arnold et al. (2012) tested six synesthetes who reported that seeing a printed grapheme evoked a color experience whereas hearing a spoken grapheme generated no color. The synesthetic color of a heard grapheme could nonetheless be retrieved from memory. Synesthetes adjusted a color patch to match their synesthetic

colors. The graphemes were presented in two different modes (printed/spoken). Thus, measures of precision (distance in color coordinates between matched and synesthetic color) could be derived and compared between when it was *online* (seen concomitantly with a grapheme) and *offline* (recalled from memory when hearing a grapheme – although note this was based on subjective report; it would have been ideal to have an objective measure of the difference in synesthetic colors between the two conditions). They found the degree of precision was comparable between colors elicited by printed and spoken graphemes, suggesting that synesthetic color perceived in real-time is perceptually equivalent to the color recalled from memory. In a control experiment, the precision of actual color seen in real-time greatly exceeded that of color memory. These results indicate that visible real color has a higher perceptual precision than color memory, consistent with Knill and Richards (1996). In contrast, synesthetic color, be it seen online or recalled offline, is perceptually analogous to color memory.

In sum, although synesthetic color may be as vivid as real color at the subjective level, recent psychophysical data suggests that it is akin to reinstatement of perceptual memory. This is a significant step forward in determining the nature of synesthetic experience, and the level at which the critical representation might occur. Such effects are consistent with the proposal that the critical process linking inducers to concurrents is conceptual, rather than entirely perceptual, in nature.

# **SYNESTHETES SHOW SIMILAR TRENDS TO NON-SYNESTHETIC INDIVIDUALS**

The comparison between synesthetes and non-synesthetes in their ways of associating inducing stimuli with perceptual attributes offers further clues regarding the contribution of conceptual processing in mediating synesthesia. In the case of grapheme–color synesthesia, large-scale studies looking for patterns across synesthetes suggest that *lexical/semantic knowledge* has an important role in mediating the coupling between lexical stimuli and colors. Rich et al. (2005) found a striking consistency among 150 synesthetes in their synesthetic pairing between graphemes and colors. For half the letters of the alphabet, certain letter–color pairs were reported significantly more often than would be expected by chance (e.g., "R" elicited red for 36% of the sample, "Y" elicited yellow for 45%, "O" elicited white for 56%, "D" elicited brown for 47%). Interestingly, similar trends were apparent for a group of non-synesthetic controls who were asked to pair colors with letters and numbers (i.e., controls reported most of the same letter–color associations as synesthetes). Simner et al. (2005) also reported a noticeable similarity between synesthetes and controls in their grapheme–color associations (e.g., in both groups, "A" tended to be red, "B" tended to be blue). Both studies suggest that grapheme–color synesthesia, whilst only manifest in a small percentage of the population, stems from mechanisms common to non-synesthetes, such as similar semantic knowledge or learning experiences. In non-synesthetes, retrieving this type of weak conceptual link (e.g., associating "A" with red) is a volitional mental operation. In synesthetes, however, accessing the links from inducers to concurrents is typically involuntary, implying a strong link between the lexical identity and color as a related attribute of the concept.

In addition to the associations between letters and colors, similarities between synesthetes and non-synesthetes have also been found in other types of synesthesia. As with grapheme– color synesthesia, there are hints that the similarities reflect contributions of high-level conceptual knowledge rather than low-level sensory processing. One example is auditory–visual synesthesia, a phenomenon where auditory stimuli elicit visual experiences.

There is good evidence showing systematic trends in the way synesthetic visual experiences vary with acoustic features of sounds (e.g., pitch). These patterns are also present in the implicit mappings seen in non-synesthetes. Ward et al. (2006) had 10 auditory–visual synesthetes select their synesthetic colors in response to instrument sounds. Ten controls were asked to associate colors with the same sounds. Although the color chosen by synesthetes was more consistent than controls across two repetitions, there was a pattern in both groups: as auditory pitch got higher, the selected colors gradually became brighter, akin to the typical pitch–brightness mapping (Marks, 1975). This systematicity is not limited to the color dimension. We tested seven synesthetes for whom sounds induced synesthetic visual experiences and, in addition to asking about synesthetic colors, we documented other non-color features, such as size, shape, and location of the experience (Chiou et al., 2012). Their synesthetic experiences in response to instrument notes were usually colored geometric objects, often with a specific spatial location (in the mind's eye). As illustrated in **Figures 2A–C**, there were systematic patterns such that as the auditory pitch got higher, the appearance of synesthetic objects became brighter in color, smaller in size, and higher in spatial location. This trend closely resembles the correspondences between auditory pitch and visual features documented in non-synesthetes (Spence, 2011), which we suggest implies a common mechanism underlying the two phenomena. We further speculate that such shared mechanisms operate at the stage of conceptual processing: there is accumulating evidence that cross-modal correspondences rely more on cognitive processing at an abstract conceptual level and perhaps less on mechanisms of

the perceptual system. For instance, the mapping between pitch and location has been demonstrated to rely on recognition of a sound's relative pitch height within a given context rather than acoustic feature of absolute pitch height (Chiou and Rich, 2012); the mapping between pitch and size has been found to be semantically mediated (Gallace and Spence, 2006); and the same set of conceptual regularities seem to guide whether various sensory attributes form a harmonious combination or not (Walker, 2012; Walker and Walker, 2012; Walker et al., 2012). Furthermore, the impact of some cross-modal mappings (e.g., pitch-size; loudness– brightness) on perceptual judgment has been shown to reflect biases at a post-perceptual decisional level, rather than perceptuallevel phenomena (Marks et al., 2003; Keetels and Vroomen, 2011). Although whether different types of cross-modal mappings differ in their extent of reliance on "perceptual vs. conceptual" factors awaits more exploration (for discussion, see Spence, 2011), the current pattern of results seems to suggest a key role of concepts in various types of cross-modality correspondences. If synesthesia builds on these cross-modality mechanisms we all share (as argued by different research teams; see Ward et al., 2006; Chiou et al., 2012; Bankieris and Simner, 2013), it should similarly depend on conceptual processing.

There are also commonalities in the synesthetic mapping between brightness and magnitude. Cohen Kadosh et al. (2007) tested individuals with digit-color synesthesia and found the brightness value of synesthetic color was negatively correlated with numerical size. This resembles the patterns seen in nonsynesthetes such that brighter color is associated with smaller numerical size (Kadosh et al., 2005) and physical size (Walker and Walker, 2012). The similarity of size–brightness mapping for both abstract (i.e., numbers) and concrete (i.e., physical objects) size is consistent with these effects arising from a higher-level conceptual stage.

Finally, the relationship between vision and touch also supports a conceptual-level relationship that underlies both synesthetic and non-synesthetic mappings. Two recent studies reported that non-synesthetic participants matched haptic sensations to visual

relationship between pitch and the location of synesthetic objects, indicated by categorical codings of synesthetes' description. Asterisks indicate a statistically significant correlation (p < 0.05). Note that there are 20 data-points (2 repetitions by 10 notes) in each figure, but some data-points are concealed due to auditory pitch (Hz) being compressed in logarithmic scale. Redrawn based on **Figure 4** of Chiou et al. (2012) with permission of the publisher.

features in ways akin to an individual with tactile–visual synesthesia. Specifically Simner and Ludwig (2012; also see Ludwig and Simner, 2012) studied EB, who experiences synesthetic colors from tactile stimuli. They found that, for EB, smooth, soft, and round stimuli tended to induce brighter colors, compared to rough, hard, and spiky stimuli. Similar patterns between touch and vision were also found in non-synesthetic participants, despite them having no conscious color experiences. Moreover, for both EB and non-synesthetes, some tactile stimuli seemed to form a reliable link with certain colors (e.g., soft stimuli were associated with pink). The authors interpreted the resemblance as stemming from shared mechanisms, such as semantic knowledge. For example, the link between softness and pink might originate from the Western convention of pairing pink color with infancy and femininity, two traits indirectly associated with softness. Thus, as with other types of synesthesia, the similarity in underlying patterns between tactile–visual synesthetes and non-synesthetic individuals is consistent with the involvement of high-level cognitive processes at a conceptual stage.

The studies reviewed above suggest that synesthesia relies on mechanisms involving ordinary semantic representations and abstract concepts, echoing the view that synesthesia may be an exaggerated form of "normal" cognitive operations (Martino and Marks, 2001; Ward et al., 2006). In particular, the knowledge regarding lexicon–color and cross-modality association seems to mediate the "content" of synesthetic experience (e.g., its hue or brightness). This suggests that the link between synesthetic inducer and concurrent is not random and arbitrary but guided by conceptual knowledge about lexical/semantic properties and cross-modality relations. In the following section, we review more evidence showing that conceptual mediation seems a general principle for other types of synesthesia.

# **IS CONCEPTUAL MEDIATION A GENERAL MECHANISM FOR SYNESTHESIA?**

As discussed earlier, it has been found that most grapheme–color synesthetes report experiencing the same synesthetic color regardless of the modality (printed or spoken) and format (italic, bold, or different fonts) in which the inducing stimulus is presented (Rich et al., 2005). In addition, it has also been found that an ambiguous graphemic stimulus can induce different synesthetic colors when interpreted differently (e.g., an ambiguous symbol can be interpreted as "S" or "5"; Myles et al., 2003; Dixon et al., 2006). These findings suggest that grapheme–color synesthesia relies crucially on the concept of lexical meaning. There are also some recent reports of novel forms of synesthesia that are completely conceptual in nature. Nikoli´c et al. (2011) documented two individuals who report that different swimming styles (breaststroke, butterfly, backstroke, and front crawl) evoke different colors. This "swimming-style synesthesia" is determined by conceptual rather than perceptual factors: the same color is elicited no matter whether the synesthete is receiving proprioceptive feedback during swimming, performing motor imagery of swimming action, or even just seeing the photos of swimming posture. The determinant of color seems to be the concept of swimming style with little impact of perceptual factors. Such observations led the authors to dub the phenomenon "*ideasthesia*" and propose that it

is the "idea" of the inducer that triggers synesthetic colors (Jürgens and Nikoli´c, 2012; Mroczko-Wasowicz and Werning, 2012).

The claim that conceptual mediation is critical for synesthesia seems incompatible with a recent study showing that letters with similar shapes evoke similar synesthetic colors (Brang et al., 2011). These authors interpreted this finding as neurons coding graphemic features activating those coding chromatic sensations in the contiguous V4, hence synesthetic color being an outcome of low-level neural wiring. Another recent study, however, showed that exactly the same result could be explained by high-level conceptual mechanisms: Jürgens and Nikoli´c (2012) trained 20 German synesthetes to learn artificially created graphemic symbols for 10–15 min and asked them to report whether these symbols evoked colors after training. Results of subjective reports showed that these pseudo graphemes were able to provoke color experience (also see Mroczko et al., 2009 in which the authors demonstrated rapid transfer of synesthetic colors to novel symbols using an objective congruency effect in a color naming task). Critically, these new "graphemes" tended to "inherit" synesthetic colors from German letters that were similar in shape. As it only took minutes to form the novel associations between "graphemes" and colors, the authors suggest that such a timeframe would be too rapid for the formation of new neural cross-wiring to occur. Instead, the result is more likely to be driven by high-level conceptual processing that guides the choices and creation of synesthetic colors. In our view, this implies similar mechanisms may operate during language acquisition in early childhood such that a synesthete may learn a new grapheme based on its visual similarity to a grapheme learnt earlier, which consequently affects its synesthetic color. For example, if "O" is learnt first and associated with lime green, "Q" may be initially learnt as "O with a stroke" and inherit the synesthetic color from "O"; as "Q" gradually develops its graphemic individuality, its lime green color may turn into another color with similar hue, such as olive green. Although speculative, this conceptual account may also explain why similarly shaped letters of the first and second language tend to induce similar colors (Rich et al., 2005; Barnett et al., 2009).

In addition to the types of synesthesia reviewed above, concepts have also been found to mediate "lexical-gustatory synesthesia," in which words are associated with tastes (e.g., the word "castanets" elicits the taste of tuna). In an elegant demonstration, Simner andWard (2006) presented images of objects that are rarely encountered in daily life (e.g., a platypus) and had six lexicalgustatory synesthetes perform picture naming. As the objects were uncommon, sometimes participants experienced the "tip-oftongue" (TOT) phenomenon in which they recognized the object and knew many of its properties but just failed to retrieve the name. When TOT occurred, the experimenter then questioned the participant further about the word's phonology (to ensure that the participant could not retrieve any information about constituent letters or the number of syllables) and asked synesthetes to report the synesthetic taste. After the naming task, the experimenter read aloud the names of objects and asked synesthetes to indicate their synesthetic tastes again. The key result was that in the state of complete TOT (i.e., neither name nor any phonological or lexical piece could be retrieved) synesthetes sometimes still

reported experiencing tastes, and the reported tastes matched with the tastes they had subsequently when hearing the objects' names. Simner and Ward (2006) suggest that thinking of a word's meaning is sufficient to trigger synesthetic tastes, occurring even in the absence of access to any lexical or phonological code. Together with other types of synesthesia, the overall pattern of evidence seems to indicate that the key stage of the information processing stream at which synesthetic percepts arise is the activation of concept. By contrast, stages upstream (e.g., perceptual analysis) or downstream (e.g., lemma or phonology) to the concept/meaning of the inducer do not seem essential for the elicitation of synesthetic experiences.

Having considered the behavioral evidence showing the key role of conceptual knowledge in mediating synesthesia, in the next section we turn to the brain, discussing what neural structures are potential substrates for conceptual mediation. We focus especially on the neural representations of object color knowledge, and the predictions our conceptual-mediation model makes for the regions involved in synesthesia beyond V4.

# **THE ROLE OF THE ANTERIOR TEMPORAL LOBES IN REPRESENTING OBJECT COLOR KNOWLEDGE AND GRAPHEME–COLOR SYNESTHESIA**

There is good evidence that the anterior temporal lobe (ATL) serves as a core substrate in the neural architecture for conceptual/semantic knowledge (Patterson et al., 2007; for review, also see Lambon Ralph and Patterson, 2008). Various lines of inquiry using neuropsychological and neuroimaging approaches have shown that theATL mediates conceptual processingfor a variety of stimuli (words, images, sounds, odors, etc.). This has led to the "hub-andspoke"model of conceptual representation, where the ATL acts as a "hub" that works in tandem with modality-specific areas (spokes) to integrate various perceptually based content into a meaningful conceptual entity (Patterson et al., 2007; Lambon Ralph and Patterson, 2008). As illustrated in **Figure 3**, for example, the concept of a banana as an edible food that is yellow in color and curvy in shape requires the concerted operation of modality-based regions processing color, shape, motor information, and the ATL constructing more abstract concepts about "banana," "edibility," and "food." One fMRI study of synesthesia and a recent neurostimulation study of ordinary color knowledge together give hints that synesthetic color experience may recruit a portion of the ATL that also underpins typical conceptual memory of object–color associations. This preliminary neural evidence, in combination with the previously reviewed behavioral results showing that synesthetic color resembles color memory and relies upon semantic representation, lead us to speculate that there is a role for the ATL in the neural substrate of synesthesia.

Could the ATL be a key neural structure in synesthesia? We propose that, based on the behavioral evidence of conceptual processing being important in synesthesia, and on the role the ATL plays in normal object knowledge, it could be a key component of the neural substrates that give rise to synesthesia. At this point, there are a few hints in the published literature and recent work from our own laboratory supporting such a speculation. First, van Leeuwen et al. (2010) reported an fMRI exploration in which, in one of many analyses, synesthetic color appeared to recruit a portion of the right ATL. In an initial experiment, these researchers

analyzed brain activation in 19 synesthetes and 19 controls within an occipitotemporal ROI to identify areas that were sensitive to real and synesthetic colors. In a second experiment, they then tested the two groups using a repetition suppression experiment to explore where neurons responded to actual color (i.e., a physically colored square; target) as if it was the same as a previously presented synesthetic color (i.e., a color-inducing achromatic grapheme; prime). The authors hypothesized that there would be a drop in neural activation in the color-sensitive areas (V4, delimited based on the localization results from the first experiment) when the prime (synesthetic color) and the following target (real color) were of the same hue relative to different hues. The predicted effect would mean that synesthetic colors are coded by the color center V4, but instead the results showed *no* suppression effect of synesthetic color in any of the pre-defined color-sensitive ROIs of the visual cortex. Subsequent analyses of brain regions outside the color-selective ROIs, however, found repetition suppression in synesthetes in the right ATL and the hippocampus, two areas crucial for memory formation and conceptual knowledge (Tulving and Markowitsch, 1998; Lambon Ralph et al.,2009). The authors concluded that synesthetic color does not fully share a neural basis with real color and is represented by high-level cortical areas beyond the scope of the visual cortex, such as those involved in memory. Note, however, that the neural suppression effect of synesthesia was obtained using planned tests in the absence of a significant interaction involving the group factor (synesthetes vs. controls). Such results must therefore be treated cautiously, but they do provide tantalizing hints that synesthetic color may

be represented in brain areas involved in high-level functions of semantic memory.

The involvement of the ATL in representing conceptual knowledge (Patterson et al., 2007), and the possible repetition suppression effect in this region of synesthetic brains (van Leeuwen et al., 2010), give an (admittedly preliminary) basis for a plausible cognitive-neural model in which synesthesia relies on "normal" mechanisms for conceptual knowledge. A critical step for such a model is first to demonstrate that this region mediates the conceptual associations between objects and canonical colors in non-synesthetes. We recently tested this hypothesis by disrupting the neural processing in the ATL using continuous theta-burst stimulation (cTBS, a high-frequency form of transcranial magnetic stimulation, TMS) to explore whether the ATL is necessary for integrating typical color with object shape and name (Chiou et al., in press). In separate experimental sessions, we disrupted neural processing of either the left ATL or a control site (the occipital pole) in eight non-synesthetic participants. In each session, participants named images of different objects with characteristic colors; the objects were presented in either their typical or an atypical color, which results in a color congruency effect (e.g., faster object recognition/naming of a yellow banana than a purple banana). Participants also performed a control task that required naming the amount of numbers in an array, where the element digit could be congruent or incongruent with the total amount (e.g., 333 or 555). This task produces a congruency effect that does not involve conceptual integration but is driven by conflicts in lexical retrieval. The critical result is that ATL stimulation abolished the otherwise robust color congruency effect but had no impact on the numerical effect, indicating a selective disruption of object color knowledge. Neither color nor numerical congruency effects were affected by stimulation at the control occipital site, ruling out alternative explanations of TMS disrupting any congruency-type effect in a non-specific manner. This result suggests that the ATL plays a key role in mediating the coupling between objects and their canonical colors.

We speculate that the neural computation performed by the ATL to link object concepts to canonical colors may be analogous to that involved in linking inducer concepts to synesthetic colors. When synesthetic color becomes strongly bound to the concept of an inducer (e.g., letter), it becomes a feature or attribute of that inducer, which shares characteristics with typical conceptual knowledge about semantics and cross-modality relations. This would explain why synesthesia is unaffected by early perceptual mechanisms (e.g., brightness contrast, chromatic adaptation, etc.), shares systematic patterns with normal cross-modality correspondences, and is influenced by high-order cognitive factors (e.g., identification of meaning, contextual interpretation, lexical/semantic relationship with other conceptual entities, etc.).

If the ATL is crucial for synesthesia, why have we not seen activation in this region in neuroimaging studies of synesthesia? There are clear methodological reasons why this may be the case. First, a recent meta-analysis has suggested that the absence of the ATL activation may reflect technical limitations of magnetic resonance imaging (Visser et al., 2010b). Specifically, images of the ATL are usually distorted due to field inhomogeneity around

the air-filled cavities near the ATL (Devlin et al., 2000), hence leading to substantial signal dropout. In addition, many neuroimaging studies have limited coverage of the temporal lobe due to a restricted field of view (FOV) during data acquisition. According to Visser et al. (2010b), the inferior section of the ATL is often not included when the FOV is narrower than 15 cm. The second major contributing factor to the lack of ATL activation is the prevalent use of ROI approach: with the aim to test whether synesthesia activates color-sensitive area V4, and to maximize power in the often small samples, it is common for synesthesia studies to confine the analyses to the ventral occipitotemporal areas, ignoring other brain regions. Together, the lack of ATL activation is likely to reflect an absence of evidence rather than an evidence of absence. Some recent papers have shown that with distortion-corrected techniques that prevent signals of the ATL from degradation, ATL activity is reliably observed when participants retrieve conceptual knowledge (Binney et al., 2010;Visser et al., 2010a;Visser and Lambon Ralph, 2011). We therefore may yet see studies demonstrating the involvement of the ATL in synesthesia if the technical issues concerning dropout are taken into account in experimental design and data acquisition.

It is worth noting that the possible repetition suppression reported by van Leeuwen et al. (2010)was in the *right* ATL,whereas Chiou et al. (in press) stimulated the *left* ATL. There is evidence showing that the bilateral ATLs contribute equally to conceptual processing, no matter whether the stimuli are graphemic or pictorial (Lambon Ralph et al.,2009; Pobric et al.,2010; but see Gainotti, 2012). It is possible that TMS to the left ATL can generate some subtle effects on the right ATL, through connections to homologous regions (e.g., Schambra et al., 2003; although note that here the contralateral effect is based on the excitability of the motor cortex, whether this is generalizable to the ATL remains unknown). Alternatively, there may be a hemispherical difference between the left and right ATL in representing synesthetic color and object color knowledge.

It is important to stress that we are *not* suggesting that synesthetic experience is merely conceptual association, mental imagery, or metaphoric thinking. Synesthesia is elicited involuntarily by a recognized inducer and is experienced as a conscious percept, two diagnostic features making it distinct from voluntary imagery and ordinary conceptual knowledge. Nonetheless, the psychophysical similarity between synesthetic color and object color memory, alongside the possible involvement of the conceptrelated ATLs, suggest that the ostensibly perceptual synesthetic experience requires critical non-perceptual mechanisms linking inducers and concurrents. Additional mechanisms, however, may still be necessary to enable a synesthetic link between inducers and concurrents to emerge as a conscious percept in a person's subjective experience. The key mechanisms that make synesthetic experience less intentional than, and thus distinct from, other concept-related phenomena (e.g., cross-modal mappings, typical color knowledge, etc.) may lie in the broad neural structures that subserve attention and feature integration, including the parietal lobe (Taylor, 2001; Laureys, 2005). Exploring the dynamics between the ATLs and parietal regions will be crucial for developing our neurocognitive conceptual-mediation model

of synesthesia. In the following section, we raise some testable predictions.

# **SOME TESTABLE HYPOTHESES REGARDING THE ROLE OF THE ATL IN SYNESTHESIA**

We propose that the ATL is a critical node in the neural network underpinning the conceptual mediation observed in synesthesia (see **Figure 4**), reflecting the accumulating evidence of the importance of identification of meaning and conceptual modulation. The ATL serves as a "hub" that mediates conceptual integration between inducer meaning and its associated concurrent. After the sensory cortices perform the initial analysis of the inducer's perceptual properties (e.g., visual word form or acoustic features of a sound), the processing stream feeds into the ATL, which functions as a repository for a myriad of high-level conceptual attributes. Neural activity in the ATL underpins the conceptual entity of the inducer and allows access to associated conceptual attributes. Analogous to its role in the ordinary conceptual integration between objects and typical colors (Rogers et al., 2007; Hoffman et al., 2012; also see Chiou et al., in press), we propose that the ATL also mediates the connection between synesthetic inducers and concurrents. Note that although the outcome of this connection is anomalous (i.e., a conscious experience not present in non-synesthetes), weak versions of this link presumably also exist in non-synesthetes, leading to implicit associations.

It should be emphasized that the ATL performs the integration at a conceptual (or supramodal) level. Namely, it combines

the meaning of the inducer with an abstract concept of the concurrent that may be devoid of perceptual content (e.g., letter "A" is red in color or middle C is metallic gray – linking perhaps in a propositional/symbolic manner). To gain access to the embodied/phenomenological content of synesthetic experiences, the ATL "hub" needs to work in tandem with cortical regions (spokes) that are adjacent to (but may not overlap with) those subserving perceptual features, hence a distributed coding pattern that involves (at least) dual sites. In the case of synesthetic color, for example, the "spoke" may be the *left medial lingual gyrus*, an area close to the typical anatomical locus of human V4 and has been reported to be activated by retrieval of object color knowledge (Chao and Martin, 1999) and synesthetic color (Rich et al., 2006). Although synesthetic color may recruit "spoke" areas of the broad perceptual cortices (e.g., the lingual gyrus), it does not necessarily require this to be the same core cortical sites as those coding the wavelength of real color (e.g., V4/the posterior fusiform gyrus), which could explain the documented differences between actual and synesthetic colors. That is, the "spoke" representation of synesthetic color may be coded in the *vicinity* of but does *not necessarily overlap* with V4. Furthermore, we suspect that the individual variability in the salience of synesthetic color may lie in the "strength" with which the ATL "hub" connects with the color-related "spoke" (i.e., how far and persistently the neural activation travels). In synesthetes who *see* vivid lexically induced colors, there may be reliable feedback signals of the ATL reaching to and activating the broad color-related visual cortices. By contrast, in synesthetes who *feel* color, the neural activation may not perpetuate through to the perceptual cortices – they may even stay within the scope of the ATL. Note these are just examples; the range of synesthetic experience seems to us to be a continuum, not a dichotomy.

In addition to the "hub-and-spoke" system that amalgamates inducers with concurrents, the elicitation of synesthetic experiences also requires the parietal lobe. Based on the strong involvement of the posterior parietal lobe in attention (Donner et al., 2002), its contribution to synesthesia is typically interpreted as a facilitation of attentional binding between grapheme and color (Esterman et al., 2006; Muggleton et al., 2007). However, there are two possible (not mutually exclusive) roles for the parietal lobe in synesthesia: first, it has a domain-general function in that it works as a part of the fronto-parietal cognitive control system that prioritizes stimuli and directs voluntary attention to an entity pertinent to synesthetic experience, such as a colorinducing digit. Second, it has a domain-specific function that it acts as "glue" that binds inducer and concurrent into a united whole. The second function of the parietal lobe may convert the content of a synesthetic concurrent into a conscious experience, making it more vivid than ordinary conceptual knowledge of color.

The strength of the model outlined above is that it generates specific predictions and is testable (and thus falsifiable). A straightforward prediction is that disrupting the neuronal activity of the ATL will lead to a breakdown of the conceptual coupling between synesthetic inducers and concurrents. This disruptive effect should occur in any type of synesthesia that involves conceptual association. Therefore, although auditory–visual and grapheme–color

synesthesia involve different inducing stimuli and recruit distinct brain regions for initial perceptual processing, TMS to the ATL should disrupt both types of synesthesia if they both rely on the conceptual integration executed by the ATL. It is noteworthy that this predicted TMS effect of the ATL might disrupt distinct processes from the disruption caused by TMS to posterior parietal regions. Whereas TMS to the parietal areas may impair the attentional processes involved in conscious synesthetic percepts, TMS to the ATL should cause disruption at a stage prior to the involvement of parietal mechanisms – the conceptual coding of inducers and their conceptual link to concurrents.

# **CONCLUDING REMARKS**

In this article, we review evidence demonstrating the critical contribution of conceptual representations in synesthesia. There are some claims that synesthesia may involve low-level visual mechanisms and acts like the color experience caused by the wavelength of light. The weight of evidence, however, suggests that synesthesia relies on high-level cognitive processes, such as identification of lexical meaning, and behaves distinctly from real color in terms of its psychophysical properties. Furthermore, neuroimaging investigations have failed to find consistent activation of the core cortical substrate of color vision (V4) when an individual experiences synesthetic color, consistent with the suggestion that cortical areas beyond the scope of the early perceptual system may be crucial. We suggest that the data call for a re-evaluation of the current neurocognitive models of synesthesia, which assume that synesthesia is a perceptual phenomenon. Based on recent behavioral and neural evidence, we speculate that the synesthetic mapping between inducers and concurrents may be analogous to the typical conceptual mapping between objects and their perceptual attributes. This conceptualization of synesthesia is compatible with recent reports that synesthetic color experience is akin to color memory. It also better explains the strikingly similar trends of non-synesthetic individuals to synesthetes. In a broader sense, we believe this framework can potentially inform us how the high-level cognitive system shapes our conscious experience of the world.

## **ACKNOWLEDGMENTS**

The authors are grateful to the three reviewers for their constructive comments. Rocco Chiou is supported by international Macquarie University Research Excellence Scholarships (iMQRES). Anina N. Rich is supported by the Australian Research Council Fellowship scheme (DP0984494).

# **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 29 July 2013; accepted: 26 January 2014; published online: 19 February 2014. Citation: Chiou R and Rich AN (2014) The role of conceptual knowledge in understanding synesthesia: evaluating contemporary findings from a "hub-and-spokes" perspective. Front. Psychol. 5:105. doi: 10.3389/fpsyg.2014.00105*

*This article was submitted to Cognitive Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Chiou and Rich. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# ADVANTAGES OF PUBLISHING IN FRONTIERS

FAST PUBLICATION Average 90 days from submission to publication

COLLABORATIVE PEER-REVIEW

Designed to be rigorous – yet also collaborative, fair and constructive

RESEARCH NETWORK Our network increases readership for your article

# OPEN ACCESS

Articles are free to read, for greatest visibility

# TRANSPARENT

Editors and reviewers acknowledged by name on published articles

GLOBAL SPREAD Six million monthly page views worldwide

# COPYRIGHT TO AUTHORS

No limit to article distribution and re-use

IMPACT METRICS Advanced metrics track your article's impact

SUPPORT By our Swiss-based editorial team

EPFL Innovation Park · Building I · 1015 Lausanne · Switzerland T +41 21 510 17 00 · info@frontiersin.org · frontiersin.org