BEYOND THE SIMPLE CONTRASTIVE ANALYSIS: APPROPRIATE EXPERIMENTAL APPROACHES FOR UNRAVELING THE NEURAL BASIS OF CONSCIOUS EXPERIENCE

EDITED BY: Jaan Aru and Talis Bachmann PUBLISHED IN: Frontiers in Psychology

#### *Frontiers Copyright Statement*

*© Copyright 2007-2015 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

*All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88919-547-3 DOI 10.3389/978-2-88919-547-3

## About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

## Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

## Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

## What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

## **BEYOND THE SIMPLE CONTRASTIVE ANALYSIS: APPROPRIATE EXPERIMENTAL APPROACHES FOR UNRAVELING THE NEURAL BASIS OF CONSCIOUS EXPERIENCE**

Topic Editors: **Jaan Aru,** University of Tartu, Estonia **Talis Bachmann,** University of Tartu, Estonia

A tongue-in-cheek allusion to the state of art in studying neural correlates of consciousness; the Sherlocks may be still immature. Image credit: DARPA (http:// www.darpa.mil), iStockphoto (http://www. istockphoto.com), edited by Kristjan-Julius Laak.

Contrasting conditions with and without conscious experience has served consciousness research well. However, research based on this simple contrast has led to controversies about the neural basis of conscious experience. One key reason for these ongoing debates seems to be that the simple contrast between conditions with and without consciousness is not specific for unraveling the neural basis of conscious experience, but rather also leads to other processes that precede or follow it. Acknowledging this methodological problem implies that some of the previous research findings about the neural underpinnings of conscious experience are actually reflecting the prerequisites and consequences rather than the direct correlates of conscious perception. Thus, it is required to re-evaluate the previous results to find out which of them are telling us anything about the neural basis of consciousness. But first and foremost, to

overcome this methodological problem we need new experimental paradigms that go beyond the simple contrastive analysis or find the ways how some older but well forgotten paradigms may foster a new look at this emerging problem.

Accordingly, this research topic is looking for empirical and theoretical contributions that: 1) envision new and suitable experimental approaches to study consciousness that are

free from the limitations of the simple contrastive analysis; 2) provide empirical data that help to separate the neural correlates of conscious experience from the prerequisites and consequences of it; 3) help to re-assess previous research findings about the neural correlates of conscious perception in the light of the methodological problems with the traditional contrastive analysis.

We hope that the theoretical insights and experimental approaches collected within this Research Topic help us to gain a more refined understanding of the neural basis of conscious experience.

**Citation:** Aru, J., Bachmann, T., eds. (2015). Beyond the Simple Contrastive Analysis: Appropriate Experimental Approaches for Unraveling the Neural Basis of Conscious Experience. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-547-3

# Table of Contents


## Still wanted—the mechanisms of consciousness!

#### *Jaan Aru1 \* and Talis Bachmann2 \**

*<sup>1</sup> Faculty of Mathematics and Computer Science/Faculty of Law, University of Tartu, Tartu, Estonia <sup>2</sup> Faculty of Law, University of Tartu, Tartu, Estonia \*Correspondence: jaan.aru@gmail.com; talis.bachmann@ut.ee*

#### *Edited by:*

*Morten Overgaard, Aarhus University, Denmark*

#### *Reviewed by:*

*Guido Hesselmann, Charité Campus Mitte, Germany*

**Keywords: consciousness, NCC, perception, state of consciousness, TMS, multivariate pattern analysis**

Thirty years ago one of us proposed a theory of perception exemplified by the phenomenon of visual masking, based on neural mechanisms known to be responsible for contents and state of consciousness (Bachmann, 1984). Some of the colleagues labeled it "exotic," perhaps trying to be polite by avoiding certain other adjectives. Be that as it may, within only a few years consciousness as a serious topic in mind sciences became firmly established (e.g., Baars, 1988; Crick and Koch, 1990). However, it can be argued that since then we have not learned much about the neural correlates of consciousness (NCC). Although many influential high-profile studies on consciousness have been published over the last 25 years, it is not clear how much of this research is directly relevant for understanding the neural basis of conscious experience (Aru et al., 2012a; de Graaf et al., 2012; Miller, 2015). The key reason for this pessimistic view is the following: many studies using various experimental paradigms have relied on the contrast between trials with and without conscious perception, but this contrast is not selective for revealing the NCC (Miller, 2007; Aru et al., 2012a; de Graaf et al., 2012). Rather, such contrast can always also lead to processes that in reality precede or follow conscious experience. As the majority of studies have been conducted with some variations of such contrastive analysis, it is hard to estimate how many of these studies are directly informative about NCC. Hence, although there are theories of consciousness that have made it to standard neuroscience textbooks, the puzzle of consciousness persists and the need for focused interdisciplinary attacks on the problem is as timely as ever.

One might think that we are overselling the problem, but contributions to the current research topic support our cautious standpoint. In particular, Pitts et al. (2014a) show that the P300, often declared to be the neural correlate of conscious access, is not observed when the subjects consciously perceive the target but it is task-irrelevant, i.e., when the subjects do not have to report the target (see also Pitts et al., 2014b). Also, Vidal et al. (2014) use an experimental paradigm of perceptual suppression through decreased contrast to demonstrate that local gamma band responses in several brain areas can increase while conscious perception is suppressed. Hence, increases in local gamma band responses do not reflect the NCC (see also Aru et al., 2012b). Finally, Ruhnau et al. (2014) review recent work illustrating that prerequisites of consciousness are not necessarily only local neural processes but rather depend on fluctuations of large-scale networks. Hence long-range integration across brain areas cannot be a specific marker of NCC. Although there will be definitely debates about these particular studies and results, the bottom line here is that the status of certain neural processes once thought to reflect the correlates of consciousness (such as the P300, local gamma band responses and long-range integration) has to be carefully reconsidered. (We kindly remind the readers that discussions about these studies can be carried out in the comments section of the respective articles.)

In addition to these studies showing that old results and theories have to be re-evaluated (Pitts et al., 2014a; Ruhnau et al., 2014; Vidal et al., 2014), the present research topic provided the following potential guidelines for consciousness research:


Bachmann and Francis, 2013, for this purpose). Tapia and Beck (2014) describe the ways how combining and comparing traditional visual modal masking and TMS-masking can inform us about the putative mechanisms of visual consciousness, roles of feedforward and feedback processes, and also the possible meaning of alpha-frequency activity in mediating awareness. In addition to reviewing the pertinent TMS-masking research and commenting on other non-invasive brain stimulation methods for studying NCC, de Graaf and Sack (2014) describe another important use of TMS—creating artificially evoked phenomenology such as phosphenes. The contributions of this Research Topic are not alone in the recent works emphasizing the usefulness of stimulation methods for locating the NCC (Parvizi et al., 2012; Koubeissi et al., 2014).


It might be that for understanding the NCC we need more advanced tools for measuring and controlling neural processes. However, it is important to note that such sophisticated tools will not be sufficient for closing in on the mechanisms of consciousness due to the problem mentioned above: contrasting trials with and without conscious perception cannot reveal the NCC. Thus, given the accelerating pace of technical improvements it is necessary that the cognitive neuroscience community steps up the game too and asks: how can our experimental paradigms specifically target the neural basis of consciousness? The present research topic did not provide definite answers, but several promising directions of research have been envisaged and potential pitfalls pointed at.

## **ACKNOWLEDGMENTS**

We would like to thank the authors as well as the reviewers for contributing to the present research topic.

### **REFERENCES**


the window to consciousness. *Front. Psychol.* 5:1230. doi: 10.3389/fpsyg.2014. 01230


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 23 December 2014; accepted: 04 January 2015; published online: 21 January 2015.*

*Citation: Aru J and Bachmann T (2015) Still wanted—the mechanisms of consciousness! Front. Psychol. 6:5. doi: 10.3389/fpsyg.2015.00005*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2015 Aru and Bachmann. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Using multivariate decoding to go beyond contrastive analyses in consciousness research

## *Kristian Sandberg1,2 \*, Lau M. Andersen3 and Morten Overgaard1,4*

*<sup>1</sup> Cognitive Neuroscience Research Unit, Hammel Neurorehabilitation Centre and University Research Clinic, Aarhus University, Aarhus, Denmark*

*<sup>2</sup> Institute of Cognitive Neuroscience, University College London, London, UK*

*<sup>3</sup> Cognitive Neuroscience Research Unit, Aarhus University, Aarhus, Denmark*

*<sup>4</sup> Center for Cognitive Neuroscience, Department of Communication and Psychology, Aalborg University, Aalborg, Denmark*

#### *Edited by:*

*Jaan Aru, University of Tartu, Estonia*

#### *Reviewed by:*

*Martin Nikolai Hebart, University Medical Center Hamburg-Eppendorf, Germany*

*Nathan Faivre, California Institute of Technology, USA*

#### *\*Correspondence:*

*Kristian Sandberg, Danish Neuroscience Center, Aarhus University Hospital, Nørrebrogade 44, Building 10G, 8000 Aarhus, Denmark e-mail: krissand@rm.dk*

Contrasting conditions with and without awareness has been the preferred method for investigating the neural correlates of consciousness (NCC) for decades, yet recently it has been suggested that further insights can be made by moving beyond this method, specifically by meticulously controlling that potential precursors and consequences of the NCC are not mistaken for an NCC. Here, we briefly review the advantages and potential pitfalls of existing paradigms going beyond the contrastive method, and we propose multivariate decoding of neural activity patterns as a supplement to other methods. Specifically, we emphasize the ability of multivariate decoding to detect which patterns of neural activity are consistently predictive of conscious experiences at the single trial level.This is relevant as the "NCC proper" is expected to be consistently predictive whereas processes that are consequences of consciousness may not occur on every trial (making them less predictive) and prerequisites of consciousness may be present on some trials without conscious experience (making them less predictive).

**Keywords: consciousness, multivariate decoding, multivariate pattern analysis, contrastive analyses, MEG, fMRI**

## **THE EVOLUTION OF CONTRASTIVE ANALYSIS**

In early outlines of contrastive analyses in consciousness research, emphasis was placed on comparing pairs of psychological phenomena of which one was conscious and the other was not (e.g., Baars, 1994). Behavioral characteristics and neural activity could thus be compared between the conscious and unconscious cases. In the case of vision, for instance, neural activity related to masked and unmasked stimulus presentations (Dehaene et al., 2001) or to stimuli presented at various durations (Kjaer et al., 2001) has been investigated. Over the last two decades, methods have evolved so rapidly that it is now difficult to determine what is a natural extension of the contrastive analysis method and what is an alternative method. In this article, we discuss some of the recent developments, and we consider how multivariate decoding, as an extension of or in combination with contrastive analysis, can contribute to identifying neural correlates of consciousness (NCC).

Many recent paradigms were developed in order to avoid confounds present in the original proposals and experiments. For instance, if stimulus duration is varied, the two conditions no longer differ exclusively in terms of the subjective experience of the participant, but also in terms of an important stimulus characteristic, which could be expected to have an impact on conscious as well as unconscious processing (Overgaard, 2004). For this reason, some scientists have preferred paradigms where the physical parameters remain stable, but only the conscious experience varies. This has been done, for instance, using masked stimuli by contrasting trials based on reports of awareness (e.g., Babiloni et al., 2010). Furthermore, in some relatively early studies participants primarily performed objective tasks, and to the extent that awareness reports were used, they were used to confirm that conditions could be treated as subliminal/supraliminal (Dehaene et al., 2001; Kjaer et al., 2001; Silvanto et al., 2005). In contrast, in some later studies, scientists have more often preferred to base analyses on trial-by-trial reports of awareness (or confidence) even when multiple physical stimulus conditions are used (Christensen et al., 2006; Koivisto, Mäntylä et al., 2010). The use of awareness reports can be seen as a necessary consequence of the wish to control for physical parameters. Methodologically speaking, these reports separate conditions when trials no longer differ in terms of objective characteristics. But their use is also partly a consequence of theoretical arguments in favor of the crucial role of awareness ratings as a key measure of validity in consciousness research (Overgaard, 2006, 2010). Some scientists even prefer to keep accuracy stable so that *only* the level of awareness varies between conditions (Lau and Passingham, 2006; Lau, 2008) or to examine the correlates of accuracy and awareness separately while ensuring that mask and stimulus have very different neural signatures (Hesselmann et al., 2011).

Common to most recent studies is that the need to control for potential confounds has resulted in a shift from the examination of complete unawareness versus complete awareness to the examination of smaller differences in graded awareness ratings or changes in the probability of obtaining reports of awareness. As the change between conscious and unconscious perception occurs more suddenly across stimulus intensity for the attentional blink (than for masking), this paradigm has sometimes been preferred (e.g., Sergent et al., 2005) although others are reluctant to use the paradigm as they suspect it reflects failure to attend (possibly conscious) perception (e.g., Lamme, 2006). Bistable perception provides another method for ensuring both conscious and unconscious perception under equal stimulation conditions. Many earlier studies using ambiguous perception examined differences in neural activity related to ambiguity/non-ambiguity (Lumer et al., 1998) or reversals of perception (Kornmeier and Bach, 2004), but some have also compared neural activity related to one perceptual state versus another (Andrews et al., 2002; Sterzer and Rees, 2008; Sandberg et al., 2013).

### **RECENT DEVELOPMENTS**

Recently, it has been argued that it is possible that studies using contrastive analyses cannot distinguish between a NCC and its prerequisites (NCC-pr) and consequences (NCC-co; Aru et al., 2012). An NCC-pr is neural activity associated with task specific initial processing (which predicts later conscious experiences) whereas an NCC-co is neural activity related to a process that occurs for conscious stimuli only, for instance encoding in working memory. Aru et al. (2012) have argued that by manipulating stimulus processing in various ways, NCC-pr and NCC-co should change, but the NCC should remain stable. In one experiment, Melloni et al. (2011) manipulated the stimulus expectation across conditions and found that an early EEG component (around 100 ms) only reflected differences between seen and unseen stimuli when there was no expectation of the stimulus, and similarly a later component (the P300) only correlated with awareness when stimuli had to be encoded in working memory, but not when a representation was already present. In contrast, a component between the two, at around 200–300 ms, correlated with conscious perception independently of condition. This indicated that the first component was an NCC-pr, the middle component at 200–300 ms a likely NCC candidate, and the P300 an NCC-co.

Although this method for moving beyond contrastive analysis is certainly novel and useful, it assumes one can evoke the same experience by means of multiple, very different manipulations. However, there is no guarantee that the experience is identical even if the same proportion of awareness responses is obtained across conditions. Ratings of awareness can be viewed as a decision process in which evidence is gathered for a particular response (e.g., Lau, 2008), for example "seen," but when different manipulations are made, the decision axis is no longer shared, and thus it is unknown if the NCC can be expected to remain unchanged (Jannati and Di Lollo, 2012). A potential solution to this could be the use of more detailed awareness ratings, but it may also be possible to improve the paradigm in general using decoding approaches as we will return to later.

Accordingly, we still have no paradigm to investigate NCCs without potential systematic confounds. Newer paradigms, to some degree, have solved problems in previous paradigms, yet have introduced new ones. For this reason, we argue that converging evidence across multiple paradigms is essential in the search for the "NCC proper" (Overgaard, 2011).

#### **MULTIVARIATE DECODING**

Here, we use the term multivariate decoding [also sometimes referred to as multivariate/multi-voxel pattern analysis, pattern classification, "brain reading," or simply decoding (Haynes and

Rees, 2006; Norman et al., 2006; Haynes, 2009)] as an umbrella term for a group of analysis techniques for which the goal, in this context, is to decode the conscious experience of a participant based on large amounts of brain data. We will exemplify the general logic behind multivariate decoding by example of a within-subject decoding.

Take an MEG dataset (**Figure 1**), for instance, of a subject with *x* epochs of class **A** (e.g., "aware") and *x* epochs of class **B** (e.g., "no awareness"): each data point of each epoch is called a feature. For a given dataset with *n* sensors/sources and *t* time points, one will thus have *n* X *t* features for each epoch. The dataset is then divided into two parts, a training set (often 90% of the data) and a test set (the remaining 10%; **Figure 1A**). A model is fitted to the training set and each feature is assigned a weight. Dependent on the sign of a given weight, it raises the posterior probability of a given epoch to belong to class **A** or **B**, respectively. The fitted training set, with its feature weights, is then used to predict the class of each epoch for the test set (**Figure 1B**). The predicted class label for a given epoch is the class label that has the highest posterior probability assigned to it when the feature weights for that epoch are summed together. One can then obtain a classification score, which is the percentage of correctly classified epochs. **Figure 1C** shows an example of this. To test the generality of the classification score, one can cross-validate the score by dividing the data set into training and test sets in different ways.

We believe that multivariate decoding has a role in neuroscientific consciousness research for several reasons and in the following we will go through these. We will, however, first emphasize that decoding results should be interpreted with care: although a given mental state can be decoded above chance from particular neural activity, this does not in itself imply a causal relationship. In this sense, multivariate decoding shares some of the limitations of correlation studies. Multivariate decoding, nevertheless, opens up new possibilities that have not previously been available.

#### **INCREASED SENSITIVITY OF MULTIVARIATE DECODING**

One main advantage of multivariate decoding is the greater sensitivity than that of traditional mass-univariate approaches typically used in contrastive analyses (i.e., the testing of single variables one at a time; Haynes and Rees, 2006; Norman et al., 2006). Multivariate decoding is more sensitive that univariate testing due to pooling of information and the informativeness of the co-variance of the features (Haynes and Rees, 2006). Furthermore, univariate tests typically test for linear relationships whereas the nature of the relationship does not need to be specified to achieve successful decoding (Haynes, 2009). The advantage of multivariate decoding in consciousness research has been shown for fMRI where Haynes and Rees (2005) showed that decoding based on V1–V3 voxels combined was more predictive of perception during binocular rivalry than decoding based on the combined mean of the same voxels. Similarly, using MEG Sandberg et al. (2013) showed that perception during binocular rivalry can be decoded at an accuracy just a few percent below peak decoding accuracy (around 75%) using just 10 occipital sensors, which were individually at chance (below 51.5%).

At its core, all univariate testing regards data points as independent of one another, which is evidently false for both MEEG and

fMRI data. It is precisely the heavy spatial and temporal correlations of neuroimaging data that make them fit for multivariate analyses. In contrast to univariate tests, multivariate tests can facilitate the information contained in the temporal and spatial dependencies between data points in both sensor and source space (MEEG) and in voxel space (fMRI) in a single test.

## **FINDING CONSISTENT CORRELATES USING MULTIVARIATE DECODING**

Multivariate tests are more sensitive to differences between conditions that are present during *all* epochs, and that they are less sensitive to differences between conditions that are only present during *some* of the epochs. Indeed, Haynes (2009) emphasized that a core NCC (or "NCC proper") should in principle be able to predict a conscious state perfectly. From this it follows that higher decoding accuracy is generally a sign of greater representational accuracy although it must be emphasized that care should be taken when comparing decoding accuracies across different brain areas, and there are several aspects to consider. For instance, Kamitani and Tong (2006) found that perceived motion direction was only decoded as well from MT+ as from earlier visual areas V1–V4 when the same number of voxels was used. Indeed, a later article by Smith et al. (2011) mention that when comparing fMRI decoding accuracies across conditions, participants, or brain regions, it is important that several factors are controlled for including the number of voxels and stimulus repetitions (and we might add that not only the number of spatial, but also the number of temporal, features should be controlled for). Additionally, they specifically emphasize the importance of controlling for or taking into account the mean amplitude of the component of interest as they show that decoding accuracy increases as a function of mean amplitude even if specificity is not increased. The function with which classifier accuracy increases as a function of response amplitude (measured as percent signal change for fMRI) can nevertheless be estimated and compared across areas for a more valid comparison of decoding accuracy. A simpler, but not always feasible solution is to compare components of equal amplitude.

A note of caution is necessary, however: even when mean amplitude is controlled for, the obtainable signal from two components may differ in their signal-to-noise ratios (for instance, if the angle of the neurons prevents a good signal in MEEG). This necessitates that one is cautious when interpreting differences in accuracy between MEEG components unless one has a good way to estimate differences in noise ceilings. Such estimations are possible with encoding models (Kay et al., 2008) or with representational similarity analysis (Nili et al., 2014), but it is presently an unresolved issue for decoding models and further work in this field is important for ensuring the validity of comparisons of decoding accuracies. It should be emphasized that the issue is not likely to be dramatic and presently a rough estimate of noise ceiling may be achieved by prior knowledge of decoding accuracies across different tasks for various brain regions/components.

Univariate tests are of course sensitive to differences that are present on all epochs, but crucially they can, in addition, be sensitive to differences that appear only on some epochs, but show some average difference between conditions (e.g., aware/unaware). This has important implications for the attempt to separate NCC-pr, NCC, and NCC-co. In **Figure 2**, we show simulated data with three components for which there are average differences between trials reported as "aware" and "unaware" by a participant. We would expect the actual NCC to vary consistently with the conscious experience – whenever the participant has an experience of the stimulus, the relevant component should reflect this. The NCC-pr, however, might be present without the NCC on some trials (i.e., one particular prerequisite of conscious experience was present on a trial, but perhaps some others were not, and the participant thus had no experience) in which case the component becomes an unreliable predictor and should not be assigned high weights by the classifier when all data are taken into account, and it should

**FIGURE 2 | Consistency of the neural correlates of consciousness (NCC).** Three simulated, hypothetical signals of differing consistency and strength are plotted. All could be candidate NCC, thus reflecting differences between trials classified as "aware" and "unaware" by a participant. For the first component, there is a small average difference, but the component is not consistently larger for "aware" trials, making it unlikely that the component reflects awareness. The component could reflect a prerequisite for consciousness (NCC-pr) as it has to be present for awareness, but it does not guarantee awareness. For the second component, there is a medium average difference, and the component is consistently larger for "aware" trials. On the single trial level, the component thus reflects awareness and it may thus be an actual NCC. Finally, for the third component, there is a large average difference, but the component is only found on a subset of "aware" trials, and it does thus not consistently reflect awareness. The component could thus reflect processes that are consequences of awareness (NCC-co), which occur exclusively for "aware" trials, but may not occur on every single aware trial. Note that traditional univariate statistics based averaged participant-specific averages would erroneously find more evidence for the last component being the NCC proper in this example.

produce suboptimal decoding accuracy when used to train/test the classifier alone. This corresponds to the first component in **Figure 2**. The NCC-co, on the other hand, might not occur after each single NCC component (even if it occurs after some NCC components), and it should never occur without an NCC component. It is thus expected to be similarly suboptimal for decoding even if it produces very large responses on some trials and a large average difference. This corresponds to the third component in **Figure 2**. The actual NCC is thus expected to be consistently the most predictive at the single trial level even if it does not produce the largest average difference. This corresponds to the second component in **Figure 2**. As mentioned above, multivariate decoding approaches are able to identify the most consistent correlates, but traditional univariate analyses typically base statistics on participant-specific means and would in our example find significant evidence in favor of the third component even though it only occurs on some trials. Importantly, if the aim is to compare components, as in our example (**Figure 2**), univariate tests are not readily interpretable. There is no straightforward interpretation of what a difference in amplitude between components means (Luck, 2014). In comparison, the interpretation of differences in decoding accuracy is straightforward – it simply means that the pattern

holds more information about the label of the state, say "aware" or "unaware."

In cases where the confounding processes occur on every single trial with an awareness response, multivariate decoding on its own will not be able to distinguish between NCC and NCCpr/NCC-co as all responses could be equally predictive. For this reason, we believe that the optimal paradigm is a combination of decoding and the methods suggested by Melloni et al. (2011) and Aru et al. (2012). One way to combine methods would be to use cross-task decoding – i.e., using several tasks resulting in similar conscious experiences and training/testing on different tasks using a leave-one-out procedure. In this case, decoding performance should be best for components that generalize across experimental contexts.

Using multivariate decoding onMEG data, a study by our group have found that conscious experience during binocular rivalry was predicted relatively accurately by activity around 130–320 ms after stimulus onset and that an earlier and a later component was not consistently predictive (Sandberg et al., 2013). In an additional (ongoing) MEG study, multivariate decoding furthermore showed that activity around this time was the most predictive of small, graded differences in the clarity of conscious experience on the single trial level (Andersen et al., in preparation). Similarly, decoding can be used on different brain areas in turn in order to compare how consistently predictive these are separately (and/or combined; Norman et al., 2006). For binocular rivalry, this was done for V1–V3 by Haynes and Rees (2005) and across the cortex by Sandberg et al. (2013). Lastly, it should be acknowledged that when doing multivariate analyses, "decoding" is not strictly necessary. There are ways of doing "encoding" as well, where one can extract parameters from the model, as in classical univariate models. Encoding applications are at the moment, however, less available than decoding applications, both theoretically and practically, but see Allefeld and Haynes (2014)for a novel approach.

### **OTHER POSSIBILITIES USING MULTIVARIATE DECODING**

The use of multivariate decoding opens up for potential research, which would otherwise be difficult or even impossible to conduct. For MEG, conscious experience can be decoded using only a few milliseconds of data gathered within the first 200 ms after stimulus presentations (Sandberg et al., 2013, 2014). Particularly, if nearperfect, near real-time decoding can be achieved, it may be possible to exploit such speed in the control of brain-computer interfaces. At present, one study was able to achieve above 85% decoding accuracy for three of eight participants (and around 95% for one; Sandberg et al., 2013). In comparison, univariate decoding (i.e., using the single best sensor at the single best time point) resulted in lower accuracies (around 10% lower), and would furthermore require both time point and sensor to be specified in advance. Additionally, other studies have shown cases in which multivariate decoding is above chance in the absence of an average activity difference (Sterzer et al., 2008).

Because decoding can be accomplished prior to report, it raises the possibility that an MEG based brain-computer interface could be used to generate changes in the environment even before they are produced by the motor behavior of the individual, which could

be of key importance in the study of overt behavior and sense of agency. Furthermore, neural correlates can be analyzed before and after the preparation to report in the attempt to filter out correlates of introspection, metacognition, and motor preparation. And finally, fast and accurate decoding allows for manipulations of stimuli or brain activity (using TMS, for instance) around the time where an event is experienced, but before it is reported, and it may allow for the study of awareness without report.

Haynes and Rees (2006) emphasized the importance of the then unresolved issue of how well activity generalizes over time, across situations (paradigms) and even across participants. This can be examined by conventional methods using correlations, but decoding provides a method of examining whether minor changes are critical or whether the overall patterns are generally maintained. Haynes and Rees (2005) used fMRI to examine drops in decoding accuracy across days, but the first long-term study was conducted by Sandberg et al. (2014), who found that the decrease in decoding accuracy within participants across 2.5 years was only around 1%, which was comparable to the drop across a few days. This study also found that the drop when attempting to generalize across participants (even at the source level) was much greater (around 10%). Further studies examining whether minor details in patterns of activity predict related changes in perceptual experience can be used to address theoretical questions about multiple realization in the brain.

It has also been established that it is possible to decode the conscious experience of one individual using a classifier trained on a different individual although the accuracy is lower than for within-individual decoding (Poldrack et al., 2009; Haxby et al., 2011; Sandberg et al., 2013, 2014). This opens up possibilities that so far have been outside the reach of cognitive neuroscience methods. One might apply multivariate decoding to investigate whether neural correlates generated in experiments using one paradigm can be used to train a classifier to decode the experience in other paradigms as we discuss above. Furthermore, between-participant decoding opens possibilities of decoding across groups for which it is uncertain whether one has conscious experiences, such as vegetative or minimally conscious patients. When consciousness has been examined in non-human animals, methods such as flash suppression have been used to ensure the validity of report as the stimuli are bistable but conscious perception can be manipulated by the experimental setup (Sheinberg and Logothetis, 1997). Such or similar methods could in principle also be used with patients, and it could be possible to decode both within individuals but also to examine how well classifiers generalize from healthy individuals to reduced consciousness patients. Here again, the improved accuracy of multivariate decoding provides an advantage compared to univariate approaches.

### **ACKNOWLEDGMENTS**

This work was supported by the European Research Council (Kristian Sandberg and Morten Overgaard).

## **REFERENCES**

Allefeld, C., and Haynes, J.-D. (2014). Searchlight-based multi-voxel pattern analysis of fMRI by cross-validated MANOVA. *Neuroimage* 89, 345–357. doi: 10.1016/j.neuroimage.2013.11.043


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 28 May 2014; accepted: 14 October 2014; published online: 30 October 2014. Citation: Sandberg K, Andersen LM and Overgaard M (2014) Using multivariate decoding to go beyond contrastive analyses in consciousness research. Front. Psychol. 5:1250. doi: 10.3389/fpsyg.2014.01250*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Sandberg, Andersen and Overgaard. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited andthatthe original publication inthis journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Isolating neural correlates of conscious perception from neural correlates of reporting one's perception

#### *Michael A. Pitts <sup>1</sup> \*, Stephen Metzler <sup>1</sup> and Steven A. Hillyard2*

*<sup>1</sup> Department of Psychology, Reed College, Portland, OR, USA*

*<sup>2</sup> Department of Neurosciences, University of California, San Diego, La Jolla, CA, USA*

#### *Edited by:*

*Talis Bachmann, University of Tartu, Estonia*

#### *Reviewed by:*

*Giorgio Marchetti, Jaan Aru, University of Tartu, Estonia Mind, Consciousness, and Language Research Net, Italy*

#### *\*Correspondence:*

*Michael A. Pitts, Department of Psychology, Reed College, 3203 SE Woodstock Blvd Portland, OR 97202, USA e-mail: mpitts@reed.edu*

To isolate neural correlates of conscious perception (NCCs), a standard approach has been to contrast neural activity elicited by identical stimuli of which subjects are aware vs. unaware. Because conscious experience is private, determining whether a stimulus was consciously perceived requires subjective report: e.g., button-presses indicating detection, visibility ratings, verbal reports, etc. This reporting requirement introduces a methodological confound when attempting to isolate NCCs: The neural processes responsible for accessing and reporting one's percept are difficult to distinguish from those underlying the conscious percept itself. Here, we review recent attempts to circumvent this issue via a modified inattentional blindness paradigm (Pitts et al., 2012) and present new data from a backward masking experiment in which task-relevance and visual awareness were manipulated in a 2 × 2 crossed design. In agreement with our previous inattentional blindness results, stimuli that were consciously perceived yet not immediately accessed for report (aware, task-irrelevant condition) elicited a mid-latency posterior ERP negativity (∼200–240 ms), while stimuli that were accessed for report (aware, task-relevant condition) elicited additional components including a robust P3b (∼380–480 ms) subsequent to the mid-latency negativity. Overall, these results suggest that some of the NCCs identified in previous studies may be more closely linked with accessing and maintaining perceptual information for reporting purposes than with encoding the conscious percept itself. An open question is whether the remaining NCC candidate (the ERP negativity at 200–240 ms) reflects visual awareness or object-based attention.

#### **Keywords: attention, awareness, masking, task-relevance, VAN, P3b**

## **INTRODUCTION**

Determining the neural basis of consciousness is one of the most challenging problems in modern cognitive neuroscience. Much progress has been made over the past 25 years by simplifying the problem and focusing first on identifying neural correlates of conscious perception or "NCCs" (Baars, 1989; Logothetis and Schall, 1989; Crick and Koch, 1990, 2003). The primary strategy has been to compare brain activity elicited by physically identical stimuli of which subjects are aware vs. unaware. While appealing in its simplicity, this approach has recently been criticized for being too inclusive in what counts as an NCC (Aru et al., 2012; de Graaf et al., 2012). Depending on how awareness is manipulated, neural mechanisms that are necessary-but-not-sufficient or sufficient-but-not-necessary for conscious perception have often been misinterpreted as true-NCCs. Aru et al. (2012) refer to such neural processes as "pre-requisites" and "consequences" of conscious perception, respectively, and have encouraged researchers to develop new paradigms to help distinguish the "NCC-proper" from these related, yet functionally distinct, processes.

### **TERMINOLOGY AND THEORETICAL CONSIDERATIONS**

While the central tenets of Aru et al.'s (2012) proposal are well-justified, we prefer to use the terms "pre-conscious" and "post-perceptual" instead of "pre-requisite" and "consequence." In our view, the term pre-requisite is too general and the term consequence too restrictive. For example, retinal processing could be considered a pre-requisite of visual awareness (with the exception of TMS-induced phosphenes); however, no one has proposed retinal activity as a potential NCC. The term *pre-conscious*, instead, is used here to refer only to cortical and cortico-thalamic activity following the initial feedforward activation of primary sensory cortex. Pre-conscious processing immediately precedes (both temporally and functionally) conscious processing and is capable of establishing elaborate perceptual representations that may become conscious (Dehaene and Naccache, 2001; Dehaene et al., 2006). The term consequence, on the other hand, appears to be used by Aru and colleagues to refer to certain neural events that *necessarily* follow the conscious awareness of a stimulus. In contrast, the term *post-perceptual* leaves open the possibility that conscious perception may occur with or without these subsequent processing events. As outlined below, simple manipulations of the task can eliminate post-perceptual processing while leaving conscious perception intact.

It should also be noted that the term "perceptual awareness" (often abbreviated as "awareness") will be used here to describe situations in which perceptual content is reportable, although actual reporting may not take place, e.g., because the subject has been instructed to only report awareness of certain types of stimuli. This reportability requirement is commonly used to operationally define conscious perception but differs from certain theoretical frameworks that consider some types of perceptual processing as "phenomenally conscious" even if the subject is unable to report anything about these percepts when specifically instructed to do so (Lamme, 2006; Vandenbroucke et al., 2014). It also differs from notions of phenomenal consciousness in which perceptual content is accessible, yet not necessarily accessed (Block, 2007, 2011). In both cases, we consider such processing as "pre-conscious" rather than a special type of unreportable consciousness, and our main goal is to isolate neural correlates of "access consciousness" from correlates of pre-conscious and post-perceptual processing (Dehaene and Changeux, 2004, 2011; Block, 2005; Dehaene et al., 2006). Because the term "access" can have different meanings in different contexts, we will qualify its usage for clarity, e.g., "conscious access" typically refers to global availability of perceptual information for flexible use by a variety of cognitive systems, while "access of perceptual information for report" refers to a much narrower set of post-perceptual operations involved in task-related memory and decision-making processes.

While it is important to separate both pre-conscious and postperceptual activity from correlates of conscious perception, the current study focuses mainly on post-perceptual processing. To illustrate the problem of conflating neural correlates of postperceptual processing with correlates of conscious perception, it is useful to consider "the refrigerator door problem" (a neural counterpart to the "refrigerator light illusion" described by Block, 2001). Imagine you have no access to the internal machinery of a refrigerator and your goal is to determine under what conditions the light inside the refrigerator turns on. You might start by opening the door to check and proceed to try just opening it a crack or just for a brief moment. However, every time you open the door the light is always on and you can't be sure the light would have been on if you had kept the door closed. In consciousness research, one of the goals is to determine whether unique brain signal X correlates with conscious perception (the light turning on), but to do so requires a perceptual report from the subject (opening the door). The problem is that it is often difficult to determine whether the same brain signal X would have occurred if the subject had not accessed this information for report (kept the door closed).

#### **POST-PERCEPTUAL PROCESSING IN PARADIGMS THAT MANIPULATE VISUAL AWARENESS**

To manipulate visual awareness in the lab, the most commonly used paradigms include backward masking, the attentional blink, change blindness, binocular rivalry, and signal detection at threshold (e.g., Pins and ffytche, 2003; Fernandez-Duque et al., 2003; Sergent et al., 2005; Koivisto et al., 2006; Pitts et al., 2010). In all cases, subjects are asked to provide some type of perceptual report after each trial. These reports range from identification of target stimuli to detailed visibility ratings such as using a perceptual awareness scale (Ramsøy and Overgaard, 2004). At first glance, this method of requiring a subjective report after each trial seems unproblematic. Regardless of whether the stimulus was consciously perceived, the subject always has to report something even if the report is "I saw nothing." Indeed, in many paradigms, stimuli are visible on roughly half of the trials, and there would be no means of sorting trials into aware vs. unaware conditions without trial-by-trial reports. However, upon closer scrutiny, aware vs. unaware contrasts in paradigms such as these are likely to expose not only differences in brain activity related to conscious perception but also differences in post-perceptual processing. On aware trials, subjective reports rely on the maintenance of perceptual information in working memory and access of this information by higher-level cognitive systems that enable decision-making and response planning/execution. On unaware trials, there is no conscious perceptual information to maintain or access even though a decision must be made and a negative response must be planned and executed. Thus, in addition to differences in conscious perception, these two types of trials differ in terms of post-perceptual processing such as maintenance in working memory and access of perceptual information for decision-making.

**Figure 1** provides a simplified schematic outline of some of the stages of processing likely to be involved in aware vs. unaware trials for a typical backward masking experiment in which the stimulus (e.g., an outline square) is perceived on roughly half of the trials. Note that in addition to differences in visual awareness, the two trial types also differ in post-perceptual processing, thus any differences in neural activity between aware and unaware trials may reflect post-perceptual maintenance and access for report instead of awareness *per se*. Importantly, the attentional blink, change blindness, signal detection, and a number of other paradigms are vulnerable to this same confound.

#### **THE SUSTAINED INATTENTIONAL BLINDNESS PARADIGM**

As an alternative to these more commonly employed paradigms to manipulate awareness, we recently adapted the inattentional blindness paradigm, originally developed and extensively tested in behavioral studies by Mack and Rock (1998), for use in conjunction with electrophysiological recordings of brain activity (Pitts et al., 2012). This paradigm includes three experimental phases. In the first phase, subjects perform a distracter task while a critical (unexpected) stimulus is presented directly in the center of their view. After 200 or more presentations of the critical stimulus, the subjects are queried regarding their awareness of this stimulus. Typically, about half of all subjects report a complete lack of awareness of the critical stimulus and are thus deemed inattentionally blind in this first phase. Importantly, after being asked about the critical stimulus, subjects are instructed to "continue performing the same task as before" in the second phase of the experiment. Due to the intervening questions, all subjects become aware of the critical stimulus during phase 2, but because they are performing the same distracter task they do not need to access information about the critical stimulus for immediate perceptual report. Finally, in the third phase of the experiment, subjects are instructed to forego the distracter task and perform a discrimination task in which the critical stimulus becomes task-relevant.

This modified inattentional blindness paradigm allows comparisons of brain activity across three conditions in which

awareness and post-perceptual processing are manipulated in a step-wise fashion: phase 1 = unaware; phase 2 = aware without post-perceptual processing (critical stimulus is task-irrelevant); phase 3 = aware with post-perceptual processing (critical stimulus is task-relevant). **Figure 2** sketches a simplified overview of the stages of processing involved in the three phases of this inattentional blindness paradigm.

The contrasts of interest in this three-phase experiment are between brain activity elicited by the critical stimulus in phase 2 vs. phase 1 (i.e., awareness vs. unawareness of task-irrelevant stimuli) and phase 3 vs. phase 2 (i.e., task-relevant vs. taskirrelevant stimuli). The first contrast (phase 2 vs. 1) should reveal neural correlates of conscious perception *per se*, while the second contrast (phase 3 vs. 2) isolates correlates of post-perceptual processing (maintenance in working memory and access for report). It should be noted that direct comparisons of brain activity across phases would be of questionable validity because the ordering of the three phases cannot be counterbalanced and neural activity may differ due to repeated exposure to the stimuli, neural fatigue, sensory adaptation, etc. To circumvent this issue, a control stimulus (e.g., a randomized background having no shapes or features) can be randomly intermixed with the critical stimuli in each phase, and neural responses to the critical stimulus can first be contrasted with neural responses to the control stimuli *within* each phase prior to making across-phase comparisons. Also, in order to prevent contamination by pre-motor and motorrelated activity, the critical stimuli are never targets that require manual responses, even in the third phase of the experiment. Instead, an infrequent target stimulus is also intermixed in all phases of the experiment, and during the third phase subjects must discriminate between the critical (non-target) stimulus and this target stimulus in order to respond appropriately.

## **PRE-CONSCIOUS, CONSCIOUS, AND POST-PERCEPTUAL EEG SIGNATURES IN THE SUSTAINED INATTENTIONAL BLINDNESS PARADIGM**

Using this modified inattentional blindness paradigm, a potential NCC-proper was isolated from pre-conscious and postperceptual activity (Pitts et al., 2012, 2014; Pitts and Martinez, 2014). Event-related potentials (ERPs) elicited by shape stimuli (formed by oriented line elements) were compared to randomly oriented control stimuli across the three phases of the experiment. During all three phases (including inattentional blindness), an initial occipital negativity was evident from ∼160–220 ms. We interpreted this activity as pre-conscious because it was evident even when subjects did not report any conscious awareness of the shape stimuli due to inattentional blindness induced by performing the distracter task. We initially referred to this component as "Nd1" for "negative difference 1" (Pitts et al., 2012), but have subsequently labeled it "CIN" for "contour integration negativity" (Pitts and Martinez, 2014), on the assumption that it reflects pre-conscious neural activity associated with extracting contour information from the stimulus array. In the second and third phases (aware conditions), the CIN was followed by a bilateral occipital-parietal negativity from ∼200–300 ms (initially labeled "Nd2"), which was similar in timing and scalp topography to the previously reported "visual awareness negativity" or "VAN" component (Koivisto and Revonsuo, 2010; Railo et al., 2011). Importantly, this negativity was also elicited during phase 1 in the group of subjects who later reported having spontaneously noticed the (irrelevant) shape stimulus. When the shape stimuli were task-irrelevant, no ERP differences were evident subsequent to the VAN, as subjects were not required to further process this information. In the third phase, however, the VAN was followed by a selection negativity (SN) and two late positive components

(a late occipital positivity "LOP," and the centro-parietal P3b) as well as by induced gamma oscillations (Pitts et al., 2012, 2014). We interpreted these latter effects as post-perceptual because they were absent in the second phase (aware, task-irrelevant) and present in the third phase (aware, task-relevant).

In a subsequent experiment (Shafto and Pitts, 2013), we tested a different critical stimulus, line drawings of faces, in a similar inattentional blindness paradigm. Here, we found that the facespecific N170 component as well as the VAN were absent during inattentional blindness (phase 1) but were present in both aware conditions (phase 2 and 3). Consistent with the results from the shape experiment, the SN, LOP, P3b, and induced gamma oscillations were only evident in the third phase in which the face stimuli were task-relevant. Importantly, in both experiments we were able to compare brain activity in aware vs. unaware conditions (phase 2 vs. phase 1-unaware subjects) without requiring immediate access for perceptual report. This was accomplished by presenting the stimuli well above threshold (300 ms duration, easily visible if expected) and by delaying the perceptual report until after the entire block of trials (which lasted ∼10 min and delivered 200+ stimuli). In both the shape and face experiments, subjects who were initially inattentionally blind often expressed genuine surprise at having failed to notice such salient stimuli for 10 min, and many subjects reported "not being able to avoid noticing the stimuli" once they knew the stimuli were being presented (in phase 2). These design features are critical in paradigms in which trial-by-trial perceptual reporting is intentionally avoided; i.e., one must be certain that the critical stimuli are never seen during the unaware condition and that these same stimuli are obvious and readily visible during the aware condition, even when subjects are performing a separate task.

This modified inattentional blindness paradigm shows promise in helping to distinguish pre-conscious and postperceptual activity from the NCC-proper. Future fMRI and intracranial EEG studies should consider adopting similar methods. However, one drawback with the inattention paradigm is that it only includes 3 of the possible 4 combinations of visual awareness and task-relevance; i.e., the unaware, task-relevant condition, which is common in other awareness paradigms, is absent in the inattentional blindness paradigm. This missing condition, in which a stimulus is attended and relevant to the task but not consciously perceived, may be important for separating neural correlates of awareness from neural correlates of attention. It is also advantageous to cross-validate results by employing more than one type of experimental paradigm, and the above mentioned results have only been obtained so far using the inattentional blindness approach. The current study was designed to address these outstanding issues.

#### **A 2 × 2 MANIPULATION OF VISUAL AWARENESS AND TASK-RELEVANCE IN A MASKING PARADIGM**

One option for measuring NCCs using the masking paradigm is to present stimuli for a brief duration followed by a mask such that the stimulus is consciously perceived on approximately 50% of trials (Koivisto et al., 2008). This approach allows comparisons of brain activity on aware vs. unaware trials while the physical stimuli (stimulus + mask) remain identical. Of course, with this approach all stimuli must be task-relevant because the experimenter has no other means of sorting individual trials into aware and unaware conditions. A different approach is to employ two different masking latencies (i.e., stimulus durations), one that is very short leading to 0% awareness and another that is considerably longer resulting in 100% awareness (Koivisto and Revonsuo, 2008). In this variation of the masking paradigm, task-relevance can be manipulated, and trial-by-trial reporting is unnecessary. However, the use of two different mask-onset latencies introduces another problem, namely that brain activity is likely to differ due to physical stimulus differences as opposed to a difference in awareness (Bachmann, 2009). For example, ERPs elicited by a mask presented 16 ms after a stimulus will be superimposed with ERPs elicited by the stimulus, whereas a mask presented 300 ms after a stimulus will only affect the stimulus-elicited ERPs at latencies beyond ∼350 ms. To control for this confound, a control stimulus can be presented and masked at each of the two latencies, and the ERPs elicited by the control stimuli can be subtracted from the ERPs elicited by the stimuli of interest prior to making any aware vs. unaware contrasts. The subtraction essentially removes the mask-elicited ERP that is superimposed with the stimulus-elicited ERP. This was the approach used in the current study, which compared stimuli of 16 ms (unaware) vs. 300 ms (aware) duration, each of which was either task-relevant or taskirrelevant on separate blocks of trials. The overall design was similar to a previous study that compared ERPs elicited by masked vs. unmasked letters while spatial and non-spatial attention were manipulated (Koivisto and Revonsuo, 2007).

Here, the stimuli were contour shapes, colored lines, and control stimuli consisting of a random array of lines (**Figure 3A**). We first performed a behavioral detection study in which these stimuli were masked at 5 different latencies (i.e., stimulus durations of 16, 33, 50, 67, and 300 ms). Based on the results from this experiment, two masking latencies (16 and 300 ms) were selected for the EEG experiment because these mask-onsets showed close to 0 and 100% detection rates, respectively, for both shape and color stimuli. In the EEG experiment, on separate blocks of trials, shapes or colored-lines were deemed task-relevant, and the shape, color, and control stimuli were masked at each of the two latencies. This design allowed comparisons of ERP difference waves (shape minus control, color minus control) across 4 types of trials: aware, task-relevant; aware, task-irrelevant; unaware, task-relevant; unaware, task-irrelevant.

#### **METHODS**

#### **PARTICIPANTS**

Twenty-six healthy adults participated in the EEG experiment. All were recruited as volunteers and gave informed consent prior to the beginning of the experiment. Data from eight participants were later excluded; three due to excessive EEG artifact, and five due to awareness of some of the 16 ms color stimuli, as assessed by a behavioral post-test (see below). The final group consisted of 18 participants (mean age: 21 years old; 13 female). An initial behavioral experiment was conducted to determine the masking SOAs used in the EEG experiment. Twelve (different) subjects participated in this behavioral study; data from one subject was excluded from analyses due to failure to follow task instructions. All experimental procedures were approved by the Reed College institutional review board in compliance with the Declaration of Helsinki.

#### **STIMULI AND PROCEDURE**

Stimuli consisted of 20 × 20 arrays (visual angle = 6.2 × 6.2◦) of oriented white line segments (0.34◦), identical to those used in Pitts et al. (2012, 2014). In the inter-stimulus intervals (ISIs) the lines were visible and in randomized orientations. Upon each stimulus presentation all of the lines rotated to new positions for the specified duration. For control stimuli, the line elements shifted to new random orientations. Two types of shape stimuli were created by arranging the new orientations of subsets of line segments to form a 10 × 10 square or a 7 × 7 diamond. Two types of color stimuli were created by changing either 3 or 4 of the randomly oriented line segments from white to red (RGB: 255,0,0). The spatial locations of the red lines were randomized but constrained to overlap with the locations of the shape contours to control for spatial attention across the shape and color tasks (see procedure below). Across trials, the positions of the red lines varied but were restricted to occur in upper, lower, left, and right regions with no two red lines being located in the same region. Example stimuli are shown in **Figure 3A**. For both the behavioral experiment and the EEG experiment, the line element arrays were always visible; no blank inter-stimulus intervals (ISIs) were used. Each stimulus presentation was followed by a return to a new pattern of randomly oriented line segments, which served as the masking/ISI stimuli; each ISI lasted a random interval between 1000 and 1200 ms. A central fixation cross (0.5◦) was present at all times. All stimuli were presented on a dark background (0.07 cd/m2) on an LCD monitor (refresh rate = 60 Hz). Stimuli were created and displayed using Presentation software (Neurobehavioral Systems, Albany, CA).

In the behavioral experiment (**Figure 3B**), three different stimuli were presented: square shapes, 4-red-lines, and random (control) arrays. The diamond and 3-red-line stimuli were not presented because these stimuli would eventually serve as targets in the EEG experiment, while the ERPs of interest would be for the non-target stimuli (squares, 4-red-lines). Each stimulus was followed by a masking (ISI) stimulus at 5 equiprobable stimulus onset asynchronies (SOA): 16.67, 33.34, 50, 66.67, and 300 ms. Thus, there were 15 combinations of stimulus-type and mask SOA, and each combination was presented 20 times. The subject's task was to press a response button with their right index finger whenever they perceived either color or shape. Subjects were encouraged to adopt a liberal response criterion, responding even if they just caught a glimpse of color or shape, while avoiding guessing. The goal of this experiment was to determine a masking SOA that would render both color and shape invisible for all subjects on all trials. It was also important to verify that the 300 ms stimuli could be detected on 100% of trials.

Based on the results from the behavioral experiment, two different masking SOAs were employed in the EEG experiment (16.67 and 300 ms), resulting in 10 different stimulus-mask

combinations: (2 types of shape stimuli + 2 types of color stimuli + 1 random control stimulus) × (2 mask SOAs) = 10 combinations. During each block of trials, these 10 trial types were intermixed and presented in random order, each at 10% probability. On separate blocks of trials, subjects performed either a color or a shape task. In the color task, the target stimuli were 3-redlines and in the shape task, the diamond stimuli served as targets. Subjects pressed a response button with their right index finger upon target detection. In order to control for between-condition ERP differences associated with motor preparation and execution, all target trials (and any trials in which subjects responded) were excluded from ERP analyses. Each task was performed for 900 trials (∼15 min) before switching to the other task for 900 trials, and this sequence was then repeated (task-order was counterbalanced across subjects). For example, half of the subjects performed the color task for 15 min, switched to the shape task for 15 min, switched back to the color task for 15 min, and finished with 15 min of the shape task. Thus, 1800 total trials were completed for each task, 180 of each stimulus type. **Figure 3C** shows a summary of the EEG experiment design, and a video example of the stimulus sequence is provided in **Movie 1**.

Because the behavioral experiment revealed detection rates greater than zero for the 16.67 ms color stimuli in some subjects, an additional behavioral post-test was conducted after each EEG session. In this post-test, color stimuli (4-red-lines), shape stimuli (squares), and control stimuli (random arrays) were presented and masked at 16.67 or 300 ms. A total of 180 stimuli were presented (across three 1 min blocks), 30 trials of each stimulusmask combination. As in the behavioral study, subjects were instructed to press a button whenever they detected either color or shape (employing a liberal response bias). Five of the 26 participants detected 16.67 ms color stimuli on at least one trial during this post-test and were excluded from ERP analyses. No subjects detected any of the 16.67 ms shape stimuli, nor were there any "false alarm" responses to the randomized control stimuli. Active experiment time for the EEG study was ∼60 min, short rest breaks were provided after every 60 trials (∼1 min), and longer breaks were given after every 300 trials (∼5 min). Each experimental session lasted 3.5–4.5 h including EEG cap preparation, practice trials, rest breaks, and the behavioral post-test.

#### **EEG RECORDING AND ERP PRE-PROCESSING**

EEG was noninvasively recorded from the scalp via Ag/AgCl electrodes sewn into customized caps with 96 electrode placements (EASYCAP, Herrsching, Germany). Electrode locations were modified from the standard 10–20 system to allow equidistant spacing (electrode positions reported here refer to the nearest channels of the international 10–20 system). Electrode impedances were kept below 5 k-. Signals were digitized (at 500 Hz) and amplified by three 32 channel amplifiers (Brain Amp Standard, Brain Products, Gilching, Germany). Eye movements and blinks were monitored by left and right horizontal EOG channels and a vertical EOG channel under the left eye, respectively. An electrode positioned at CPz served as the reference during recording.

ERPs were time-locked to the line segment orientation changes, low-pass filtered at 25 Hz (24dB/Oct), re-referenced to the average of the left and right mastoids, and baseline corrected from −100 to 0 ms. The left and right horizontal EOG channels were re-referenced as a bipolar pair. Trials were discarded if they contained eye movements, blinks, or other muscle artifacts in a −600 to +600 ms interval surrounding stimulus-onset. Artifact detection was accomplished semi-automatically via per-subject adjustment of the following peak-to-peak thresholds: eye movements (50µV, 50 ms steps, in bipolar HEOG), blinks (100µV, 200 ms steps, in VEOG and FP1), and muscle noise (150µV, 200 ms steps, all remaining channels). On average, 16% of trials were rejected due to a combination of these artifacts. Individual electrodes showing extended periods of noise in the raw EEG were removed and replaced by interpolated signals from surrounding channels using topographic spherical splines (channels included in ANOVAs were not interpolated).

#### **ERP ANALYSES**

Our strategy for ERP analyses was to first identify time windows and electrodes of interest in the grand-averaged difference waves of all conditions averaged together, using our previous results as a guide (Pitts et al., 2012). Peak latencies were identified and mean amplitudes were assessed in ± 20 ms time windows around these peaks (± 50 ms for the broad P3b). The electrode showing the maximal signal, along with 5–8 adjacent sites (according to each component's scalp topography), were selected for analysis. For the shape stimulus difference waves (shape minus random), four distinct components were evident, each of which closely matched our previous results in terms of timing and scalp distribution: contour integration negativity, "CIN" (160– 200 ms), visual awareness negativity, "VAN" (200–240 ms), late occipital positivity, "LOP" (310–350 ms), and the "P3b" (380– 480 ms). For the color stimulus difference waves (color minus random), four components were also evident, some of which showed reduced latencies compared to the corresponding shapeelicited components. The first color-elicited component showed a very similar time course and scalp topography to a previously reported color vs. non-color ERP difference (Schoenfeld et al., 2003). This component, which consisted of a midline ventralposterior negativity accompanied by a vertex positivity from 130 to 170 ms, has previously been referred to as the "sensory effect of color" which we abbreviate here as "SEC" (Zinni et al., 2014). The SEC (130–170 ms) was followed by the VAN (200–240 ms), LOP (290–330 ms), and P3b (380–480 ms) components, each of which showed similar scalp topographies to their shape-elicited counterparts.

Statistical analyses began with 2 × 2 ANOVAs with the factors awareness (16 or 300 ms durations) and task-relevance (attended or unattended) for each difference wave component using the time windows specified above. Difference wave amplitudes were averaged across electrode clusters according to the scalp topographies of each component as follows: CIN [*PO4, P6, O2, PO8, O10, PO10*]; SEC [*FC1, FCZ, FC2, C1, CZ, C2, CPZ*]; VAN [*PO4, P6, O2, PO8, O10, PO10*]; LOP [*P1, PZ, P2, PO3, POZ, PO4, O1, OZ, O2*]; P3b [*CP1, CPZ, CP2, P1, PZ, P2, POZ*].

All main effects and interactions were then further explored by conducting cluster mass permutation tests on the difference amplitudes (shape minus random; color minus random) for each of the 4 conditions separately (*aware, task-relevant*; *aware, taskirrelevant*; *unaware, task-relevant*; *unaware, task-irrelevant*). To increase statistical power, separate tests were carried out for the early (100–300 ms) and late (300–600 ms) time windows and permutation analyses were restricted to 63 of the 96 electrodes (covering central, parietal, temporal, and occipital regions) based on a priori hypotheses regarding scalp topographies of each component. Thus, 6363 comparisons were made for the early time window and 9450 for the late time window. In all cases, twotailed cluster mass permutation tests (Bullmore et al., 1999), with a family-wise alpha level of 0.05, were conducted using the original data and 2500 random within-subject permutations of the data. Electrodes within approximately 3.02 cm of one another were considered spatial neighbors. All cluster mass permutation analyses were carried out using the mass univariate ERP Toolbox (Groppe et al., 2011).

#### **RESULTS**

#### **BEHAVIORAL EXPERIMENT**

Mean shape and color detection rates at each of the five masking SOAs tested are provided in **Figure 4**. Overall, the shape stimulus was more readily masked than the color stimulus. Eight of the 11 subjects never responded to shapes presented for 16 ms, two subjects responded once (5%) and one subject twice (10%);

note that some of these responses may have reflected accidental button presses rather than transient perceptual capacities. For the 16 ms color stimulus, five subjects never responded, and detection rates for the remaining six subjects were as follows: 10, 15, 25, 30, 30, 40%. The average false alarm rate (responses to random stimuli at any of the five stimulus durations) was 0.5%. Three subjects had 2% false alarm rates while all other subjects had zero false alarms. All three subjects showing false alarm rates > 0 also showed detection rates > 0 for the 16 ms color stimulus.

Because our goal was to identify a single masking SOA that could render both shape and color stimuli invisible for a large majority of subjects/trials, we chose the 16 ms SOA for use in the EEG experiment. Importantly, because roughly half of the subjects in this behavioral experiment showed non-zero detection rates for the 16 ms color stimulus, we also administered a behavioral post-test after each EEG session (see methods above). We excluded from EEG analyses any subject who responded to one or more 16 ms stimuli in this post-test (five out of 26 subjects were excluded for this reason). For the remaining subjects, detection rates in this behavioral post-test were 0% for both 16 ms stimuli and 97.73% (s.e.m. = 0.18%) and 98.19% (s.e.m. = 0.10%) for the 300 ms shape and color stimuli, respectively.

#### **EEG EXPERIMENT**

Behavioral results from the EEG experiment indicated that the shape task was slightly more difficult than the color task, although performance on both tasks was strong. For the shape task, subjects detected the 300 ms diamond-shaped targets on 93.55% of trials (s.e.m. = 1.3%), d = 4.61 (s.e.m. = 0.11), RT = 563 ms (s.e.m. = 10 ms). For the color task, subjects detected the 300 ms 3-red-line targets on 98.24% of trials (s.e.m. = 0.5%), d = 4.68 (s.e.m. = 0.09), RT = 537 ms (s.e.m. = 10 ms). While d did not differ statistically across the two tasks, RTs were significantly shorter for the color task, *t*(18) = 4.56, *p* = 0.0003. Response rates to the 16 ms color and 16 ms shape targets were both 0%.

Grand-averaged ERPs elicited by the non-target shape and color stimuli are compared to ERPs elicited by the random (control) stimuli in **Figures 5**, **6**, respectively. In both figures, electrodes representative of the scalp locations of the main components of interest are shown. Note that the ERPs elicited by the control stimuli differ according to stimulus duration and that within-block comparisons were made in all cases. For example, the shape-elicited ERPs in the aware, task-relevant condition were compared to the random-stimulus ERPs during the shape-task blocks, whereas the shape-elicited ERPs in the aware, task-irrelevant condition were compared to the random-stimulus ERPs during the color-task blocks. Difference waves formed by subtracting the ERPs to the appropriate control stimuli from the ERPs to the shape and color stimuli are shown in **Figure 7**. In these difference waves the components CIN (shape) SEC (color), VAN, LOP, and P3b can be visualized. Results from all statistical analyses are provided below for shape and color stimuli, for each of the components of interest, organized according to their temporal sequence (see Methods Section for descriptions of each component's time course and scalp distribution).

#### *Early sensory effects*

ANOVA for the CIN component elicited by the outline square showed a main effect of awareness, *F*(17) = 12.42 (*p* = 0.0026), a main effect of task-relevance, *F*(17) = 11.74 (*p* = 0.0032), and no interaction. Cluster mass permutation analyses for each condition separately confirmed that these main effects were due to significant amplitude differences in the *aware, task-relevant* (−1.58µV, *SD* = 1.38) and *aware, task-irrelevant* (−0.91µV, *SD* = 1.41) conditions (Figure S1). Difference amplitudes in the *unaware, task-relevant* and *unware, task-irrelevant* conditions did not significantly differ from zero; although a trend toward a negative difference over the posterior scalp was observed for the *unaware, task-relevant* conditions (see **Figure 8**).

ANOVA for the SEC component elicited by the colored lines revealed a main of effect of awareness, *F*(17) = 15.24 (*p* = 0.0011), with no main effect of task-relevance, nor an interaction between the two. Permutation analyses confirmed significant amplitude differences in the *aware, task-relevant* (1.87µV, *SD* = 2.16) and *aware, task-irrelevant* (0.83µV, *SD* = 1.24) conditions, while amplitude differences in both unaware conditions were not significant (Figure S2).

#### *Visual awareness negativity (VAN)*

For the shape-elicited VAN component, ANOVA resulted in a main effect of awareness, *F*(17) = 11.83 (*p* = 0.0031), a main effect of task-relevance, *F*(17) = 17.83 (*p* = 0.00057), and a significant interaction, *F*(17) = 8.42 (*p* = 0.0099). Cluster mass permutation tests revealed a significant VAN component in the *aware, task-relevant* and *aware, task-irrelevant* conditions and an absence of this component in both unaware conditions. The interaction between awareness and task-relevance was explained by a substantial amplitude increase in the *aware, task-relevant* (−3.19µV, *SD* = 3.09) compared to the *aware, task-irrelevant* condition (−1.58µV, *SD* = 2.02).

ANOVA for the color-elicited VAN showed a main effect of awareness, *F*(17) = 40.91 (*p* = 0.000007), a main effect of task-relevance, *F*(17) = 15.05 (*p* = 0.0012), and an interaction between the two *F*(17) = 14.21 (*p* = 0.0015). Similar to the

shape-elicited VAN, cluster permutation tests showed significant amplitude differences in both aware conditions and the interaction was explained by relatively larger amplitudes in the *aware, task-relevant* (−5.77µV, *SD* = 3.91) vs. the *aware, task-irrelevant* condition (−3.56µV, *SD* = 2.54) along with no significant differences in either of the unaware conditions.

#### *Late positivites*

ANOVA for the shape-elicited LOP component showed a main effect of awareness, *F*(17) = 46.79 (*p* = 0.000003), a main effect of task-relevance, *F*(17) = 11.03 (*p* = 0.0040), and a significant interaction, *F*(17) = 29.60 (*p* = 0.000044). Permutation tests suggested that these effects were due to a large amplitude difference in the *aware, task-relevant* condition (4.48µV, *SD* = 2.70), a smaller difference in the *aware, task-irrelevant* condition (1.48µV, *SD* = 2.02), along with no significant amplitude differences in either unaware condition.

For the color-elicited LOP, ANOVA indicated a main effect of awareness, *F*(17) = 15.67 (*p* = 0.0010), a main effect of task-relevance, *F*(17) = 35.41 (*p* = 0.000016), and a significant interaction, *F*(17) = 25.05 (*p* = 0.00011). However, unlike the shape-elicited LOP, cluster mass permutation tests showed that these effects were driven by a large amplitude difference in the *aware, task-relevant* condition (4.59µV, *SD* = 3.48), along with a smaller effect in the *unaware, task-relevant* condition (0.67µV, *SD* = 1.09). No amplitude differences during the LOP time window were evident in the *aware, task-irrelevant* or *unaware, task-irrelevant* conditions.

Finally, for the P3b analyses, ANOVA for the shape stimuli resulted in a main effect of awareness, *F*(17) = 26.67 (*p* = 0.000078), a main effect of task-relevance, *F*(17) = 18.51 (*p* = 0.00048), and a significant interaction, *F*(17) = 36.85 (*p* = 0.000012). All of these effects were driven by a large amplitude difference in the *aware, task-relevant* condition (7.72µV, *SD* = 5.85), along with no significant amplitude effects in any of the other conditions. Although cluster permutation analyses showed positive amplitude differences in the *aware, task-irrelevant* condition from ∼300–400 ms (Figure S1), these effects correspond to the small LOP described above and were absent for electrode sites and time windows corresponding to the P3b.

ANOVA for the P3b elicited by the color stimuli also showed a main effect of awareness, *F*(17) = 33.32 (*p* = 0.000023), a main effect of task-relevance, *F*(17) = 34.71 (*p* = 0.000018), and a significant interaction, *F*(17) = 22.63 (*p* = 0.00018). Cluster mass permutation tests for each condition revealed a different pattern of effects from that of the shape-elicited P3b. In this case, color-elicited P3b amplitudes were significant for all conditions except the *unaware, task-irrelevant* condition. The P3b in the *unaware, task-relevant* condition (1.31µV, *SD* = 1.43) was similar in magnitude to the P3b in the *aware, task-irrelevant* condition (1.73µV, *SD* = 1.76) and occurred slightly earlier in time (see

Figure S2), while the P3b in the *aware, task-relevant* condition was clearly the largest (8.28µV, *SD* = 5.50).

#### *Results summary*

**Figures 8**, **9** provide summaries of the results for shape and color stimuli, respectively. In these figures, scalp topographies for each of the main components (defined by amplitude differences between shape and random ERPs, and color and random ERPs, respectively) are provided across all four conditions resulting from the 2 × 2 manipulation of awareness and task-relevance. Asterisks under the scalp maps indicate that a significant amplitude difference was found during the component's time-window as assessed by cluster mass permutation tests.

The overall goal of this experiment was to identify components that are present in both aware conditions (regardless of taskrelevance) and absent in both unaware conditions. Across the shape and color stimuli, only the early sensory components (CIN, SEC) and the visual awareness negativity (VAN) showed this pattern. For both shape and color, these components were evident in aware conditions and absent in unaware conditions, regardless of task-relevance. Whereas the LOP also showed this pattern for the shape stimuli (**Figure 8**), it was absent for aware, taskirrelevant color stimuli (**Figure 9**). The P3b did not consistently correlate with awareness in shape or color trials (**Figures 8**, **9**), being absent for aware, task-irrelevant shapes, and present for unaware, task-relevant color.

## **DISCUSSION**

In agreement with previous results using the inattentional blindness paradigm (Pitts et al., 2012, 2014), the present study used a 2 × 2 manipulation of awareness and task-relevance in a backward masking task and found a consistent ERP correlate of conscious visual perception, the visual awareness negativity (VAN) component. The VAN is a mid-latency occipital-parietal negativity (measured here from 200 to 240 ms) that has been targeted by a number of previous studies using various awareness manipulations (Koivisto and Revonsuo, 2003, 2008, 2010; Ojanen et al., 2003; Wilenius-Emet et al., 2004; Koivisto et al., 2005; Wilenius and Revonsuo, 2007; Railo et al., 2011; Sandberg et al., 2013). ERP components preceding the VAN are likely to reflect preconscious processing, considering that a previous study using identical stimuli (Pitts et al., 2012) found evidence for the elicitation of such components in unaware (and unmasked) conditions. In the current study, components prior to the VAN were absent in the unaware conditions, suggesting interference of pre-conscious processing by the masking stimulus (presented at 16 ms SOA). Components subsequent to the VAN have often been proposed as potential NCCs, including most notably the P3b component (Sergent et al., 2005; Babiloni et al., 2006; Del Cul et al., 2007; Lamy et al., 2009; Dehaene and Changeux, 2011; Batterink et al., 2012; Kouider et al., 2013). The current study, however, provides evidence against this view, as the P3b along with another component subsequent to the VAN (the LOP) did not consistently

correlate with awareness. Instead, these later components are likely to reflect post-perceptual or attention-based processes necessary for completing the task, as their amplitudes were by far the largest in the aware/task-relevant condition. Moreover the P3b was present in some of the unaware, task-relevant conditions, and absent in some of the aware, task-irrelevant conditions.

## **CORRELATES OF VISUAL AWARENESS vs. POST-PERCEPTUAL PROCESSING**

It is difficult to create controlled conditions in which subjects are aware vs. unaware of physically identical stimuli without requiring an immediate trial-by-trial perceptual report. Nevertheless, it is important to devise new methods for allowing such contrasts because the neural correlates of conscious perception are easily confusable with the neural correlates of post-perceptual maintenance and access for immediate subjective report. In both task-relevant and task-irrelevant situations (in which the stimuli are readily visible), subjects are considered to be conscious of the perceptual information because whenever they are asked to report what they see, they are able to easily do so. It follows that brain activity associated with conscious perception should be present in both cases. However, when perceptual information is accessed for immediate report, a variety of additional neural processes are likely to ensue, each of which is related to "doing something extra" with the perceptual information the subject is already conscious of. Importantly, our manipulation of task-relevance was designed to test a particular post-perceptual process: *access of perceptual information for report*. This should not be confused with the more general concept of *conscious access* (Dehaene and Changeux, 2004; Block, 2005), which we assume was present in our aware (300 ms stimulus) task-*irrelevant* conditions. In other words, we expect that on any given trial, if the subjects had been asked, they would have easily been able to report seeing the 300 ms task-irrelevant stimuli (this was indirectly verified by our behavioral experiment and behavioral post-test), thus perceptual information was globally available but the subjects did not have to use this information to complete the primary task-at-hand.

In our previous inattentional blindness experiments (Pitts et al., 2012, 2014; Shafto and Pitts, 2013), we attempted to deal with this issue by rendering a stimulus easily visible (if expected) but irrelevant to the task-at-hand and delaying the perceptual report until after a full block of trials. In this situation, in which subjects were fully aware of a stimulus but did not have to do anything with the perceptual information because it was irrelevant to the task, we found that only early sensory (CIN) and mid-latency (VAN) ERP components were elicited. Subsequent components such as the P3b were only evident when the stimuli became relevant to the task. In the current experiment, instead of diverting attention to create an unaware condition, we severely masked some stimuli (16 ms masking SOA) and compared ERPs to conditions in which the same stimuli were clearly visible (300 ms masking SOA). Importantly, in order to assess post-perceptual processing we manipulated the task such that stimuli were relevant to the task during some blocks of trials and irrelevant during other blocks. We found a similar pattern of results as in the inattentional blindness studies, in that the VAN was consistently associated with perceptual awareness, while subsequent components were either absent when the stimuli were irrelevant to the task or present when subjects were unaware of task-relevant stimuli.

Despite the main pattern of results, it remains possible that a small P3b was present (but below the statistical threshold) during aware, task-irrelevant shape trials (both in the current study as well as in Pitts et al., 2012 and Pitts et al., 2014). In this case, one

might argue that the P3b is a correlate of conscious perception but was very weak during the task-irrelevant conditions because subjects were so focused on the relevant stimuli that they only caught a glimpse of the irrelevant stimuli on a subset of trials. Future studies might address this by incorporating infrequent, unexpected, probe trials in which the experiment is interrupted after a task-irrelevant stimulus and subjects are asked to "report what you just saw." If subjects can always report awareness of task-irrelevant stimuli, but the P3b is still absent, a stronger case could be made for a lack of a relationship between the P3b and awareness; however, one must be careful to avoid inadvertently rendering the task-irrelevant stimuli relevant by including too many "surprise" report trials.

#### **ATTENTION IN THE ABSENCE OF AWARENESS?**

An unexpected finding in the current study was evidence for a P3b elicited in the absence of reported awareness. This effect appeared in the unaware, task-relevant condition for color stimuli. It was absent for the equivalent condition for shape stimuli and there was no evidence for a P3b in any of the unaware, taskirrelevant conditions. While very few studies have reported a P3b in the absence of conscious perception, it is generally accepted

that attention and awareness are independent (yet often interact), and there is growing evidence for attentional modulations in the absence of awareness (Bernat et al., 2001; Koch and Tsuchiya, 2007; Wyart and Tallon-Baudry, 2008; Tsuchiya and Koch, 2009; Bachmann, 2011; Tallon-Baudry, 2011; Marchetti, 2012; Aru and Bachmann, 2013). We previously posited that the P3b is postperceptual and argued that while the P3b does not appear to be necessary for conscious perception, it may be a sufficient marker; i.e., whenever a P3b is observed one can be confident that the subject was aware of the stimulus (Pitts et al., 2012, 2014). The current result, however, suggests that the P3b might not be necessary *or* sufficient, as it was elicited by color stimuli of which the subject was presumably unaware. Interestingly, the P3b was completely absent for task-irrelevant color stimuli of which the subject was also unaware, thus suggesting a task-based attentional modulation of perceptually undetected stimuli. This result is consistent with a recent study that showed an N2pc component for color singletons that captured attention even when the subject was unaware of the stimuli, but only when color was relevant to the search task (Ansorge et al., 2010), as well as an earlier study that found evidence of attention capture (N2pc) in the absence of awareness due to object substitution masking (Woodman and Luck, 2003). Similarly, recent studies have provided evidence for working memory related processing in the absence of awareness (Hassin et al., 2009; Gilchrist and Cowan, 2010; Soto et al., 2011).

An alternative explanation for this finding is that some subjects on some trials may have been partially aware of the 16 ms color stimuli even though their reports indicated lack of awareness. In these partial awareness situations, subjects may have attempted to perform the discrimination task (to determine if the stimulus contained 3 or 4 red lines) but could only detect the presence of color without being able to discriminate between target and non-target color stimuli (see Windey et al., 2014 for a recent review of graded vs. dichotomous awareness). While we conducted an initial behavioral study to determine an appropriate masking SOA and administered a behavioral post-test to each of the EEG subjects in which a simple detection task replaced the discrimination task, it is still possible that some residual awareness for a number of color stimuli occurred during the EEG portion of the study. Indeed, we excluded five out of the original 26 participants because they were able to detect at least one of the 16 ms color stimuli during the behavioral post-test. Because one of the goals of the current design was to avoid trial-bytrial reports, we intentionally did not acquire the data necessary to fully evaluate this alternative interpretation. Future studies should consider following up on this preliminary result in order to determine whether a P3b can be elicited during attentive, but unaware conditions, perhaps by employing a detection task with low response criterion during EEG recording (Squires et al., 1973, 1975).

#### **PRE-CONSCIOUS PROCESSING**

In contrast to our previous results using the inattentional blindness paradigm (Pitts et al., 2012), the current study provided no evidence for an early pre-conscious ERP difference between shape and random stimuli in the unaware conditions. One of the major findings in our previous study was that a component we labeled as Nd1 (for negative difference 1, here referred to as the CIN), distributed over the occipital midline from ∼160–200 ms, was elicited in all conditions, even when subjects were unaware of (inattentionally blind to) the shape patterns. Why was this component absent from the unaware conditions in the current study? The most likely possibility is due to the differences in bottom-up stimulus strength between the two studies, a factor that is known to influence whether a stimulus will be processed non-consciously, pre-consciously, or consciously (Dehaene et al., 2006). In our previous inattentional blindness experiment, the shape stimuli were always presented for 300 ms in duration (including the unaware condition), whereas the current study employed heavily masked stimuli (16 ms durations) to create the unaware conditions. In line with the interactions between top-down attention and bottom-up stimulus strength described by Dehaene et al. (2006), the current results showed a small (just below statistical threshold) CIN in the unaware, task-relevant condition. In other words, with such severe masking and the resulting reduction of bottom-up stimulus strength, an unseen stimulus might only be processed preconsciously if attended. In contrast, in our previous study topdown attention was not required because sufficient bottom-up stimulus strength allowed this same stimulus to be processed pre-consciously.

One of the advantages of the current experimental design was a degree of internal replication made possible by the 2 × 2 crossing of the color and shape stimuli/tasks. This allows one to ask whether the same pattern of results seen for the shape stimuli was also observed for the color stimuli. An early sensory effect of color (SEC) was not observed for either of the unaware conditions, although a trend in this direction was evident (see **Figure 9**, left column, third row). To further explore how pre-conscious processing is influenced by top-down attention and bottom-up stimulus strength, future studies could present extended duration color stimuli during inattentional blindness or present backwardmasked color stimuli at various masking SOAs while manipulating task-relevance.

#### **VAN: ATTENTION OR AWARENESS?**

A main focus of recent consciousness research has been the relationship between attention and awareness (Tsuchiya and van Boxtel, 2013). Historically, attention and consciousness were often treated as similar if not identical concepts; nowadays, however, many researchers are proposing that each refers to a separate category of neural and psychological processes. Although attention and awareness may be functionally distinct, the question of whether each can exist independently of the other remains a topic of debate (Cohen et al., 2011, 2012; Tsuchiya et al., 2012; Aru and Bachmann, 2013). One view posits that attention can operate in the absence of awareness, and awareness can occur in the absence of attention, whereas an opposing view argues that while attention can influence processing of stimuli of which the subject is unaware, there is no such thing as awareness in the absence of attention; i.e., attention is necessary for conscious perception. Results from experiments employing the inattentional blindness paradigm offer strong support for the latter view (Cohen et al., 2011; Mack and Clarke, 2012; Pitts et al., 2012).

The present results, along with our previous inattentional blindness results, suggest that the most viable candidate for an ERP correlate of awareness is the VAN (Nd2) component. In our previous study (Pitts et al., 2012), a well-known attention-related component, the SN (Harter and Aine, 1984; Hillyard and Anllo-Vento, 1998), was evident immediately after the VAN, but was only observed in conditions where the stimulus was task-relevant. Distinguishing between the VAN and the SN in a data-driven manner is not easy, because both components consist of a midlatency (∼200–300 ms), posterior, bi-lateral negativity. To further complicate matters, both the VAN and the SN show variable (and overlapping) latencies depending on the stimuli and task. The only way to isolate the VAN from the SN component may be to manipulate task-relevance; in our previous study both components were present in task-relevant situations, whereas only the VAN was present in aware but task-irrelevant conditions (Pitts et al., 2012). In the present study, however, no clear SN was evident in the aware, task-relevant condition, perhaps because the stimuli used here were so easily discriminable that the processing resources indexed by the SN did not have to be engaged. Alternatively, it is possible that the VAN measured in the current study overlapped the SN in the task-relevant condition such that the two components were indistinguishable. It will be important for future studies to distinguish between the VAN and the SN, especially in situations in which the stimuli are task-relevant.

In the current study as well as in Pitts et al. (2012), the amplitude of the VAN was larger in the task-relevant compared to the task-irrelevant condition. If the VAN indeed reflects perceptual awareness, which is often assumed to be an all-or-none phenomenon (you see the stimulus or you don't), why might its amplitude vary according to the task? One possibility might be that subjects were not aware of the task-irrelevant 300 ms stimuli on every single trial because their attention was focused on a separate task (i.e., partial inattentional blindness). This seems unlikely, especially for the color stimuli (see **Movie 1**); in a separate experiment (Pitts et al., 2014) we found a 0% inattentional blindness rate for similar 300 ms color stimuli. Another possibility is that the timing of perceptual awareness is more consistent across trials when the stimulus is relevant to the task; whereas, on task-irrelevant trials subjects may notice the irrelevant stimuli on every trial but at slightly different times on different trials because their primary task is to determine if a potential target is present. If this is the case, one would expect larger amplitudes with briefer time-courses for awareness-related ERPs in task-relevant conditions and smaller amplitudes with extended durations for the same ERP components in task-irrelevant conditions, given that the ERPs are derived by averaging across many trials. Data from Pitts et al. (2012) follow this pattern very closely, and although the current data show amplitude differences for the VAN without obvious corresponding differences in component duration, the shape of the VAN is slightly skewed with a longer right-tail in the task-irrelevant condition (see **Figure 7**), consistent with the trial-by-trial latency jitter account. A third alternative explanation is that the VAN and SN components may have been temporally superimposed such that only a portion of this negativity reflects perceptual awareness. The amplitude increase during task-relevant situations might be due to an increase in top-down attention reflected by the SN component, rather than a larger number of aware trials or greater trial-to-trial consistency in the timing of awareness.

While there is growing evidence that the VAN tracks closely with awareness while the SN varies as a function of task demands and attention, it is important to acknowledge the existence of a variety of attentional processes. In addition to exogenous (bottom-up) and endogenous (top-down) attention, visual attention can be selectively allocated to spatial locations, features, or entire objects. An alternative interpretation of the VAN is that it reflects some form of object-based attention that is necessary for conscious perception (perhaps a specific interaction between attention and high-level perceptual representations). Because many manipulations of awareness also involve manipulations of attention, this possibility cannot be easily discounted. To give some examples, inattentional blindness involves altering attention to alter awareness; backward masking at short vs. long latencies alters bottom-up attention to influence awareness; backward masking and signal detection at threshold capitalizes on stochastic trial-by-trial fluctuations of attention; the attentional blink involves differences in attention between seen and unseen stimuli; and change blindness differs from change detection based on the allocation of spatial attention. Because of the common co-manipulation of attention and awareness across a variety of paradigms, we previously argued that naming an ERP component the "visual awareness negativity" (VAN) might be pre-mature (Pitts et al., 2012), although we adopt this nomenclature here for consistency with the literature. If the VAN turns out to reflect a type of object-based attention instead of awareness *per se*, this might suggest that an obvious ERP correlate of conscious perception has yet to be discovered (Verleger, 2010). This would explain the amplitude increase for the VAN in task-relevant vs. task-irrelevant conditions, i.e., task-relevance enhances objectbased attention. Importantly, this would not mean that a neural correlate of awareness does not exist, but rather that ERPs can only measure a limited set of neuronal events. In any case, studies of ERPs can still be useful in helping to narrow down the timewindow for which potential NCCs could be found, while more sensitive techniques such as intra-cranial recordings in human epileptic patients may be necessary to identify NCCs.

A final possibility worth consideration is whether a particular type of interaction between attention and perceptual representation *is* the underlying neural mechanism of conscious awareness, and the VAN is a marker of this type of interaction. To explore this idea, future consciousness research might focus efforts on understanding object-based attention, perceptual encoding, and the interaction between the two, rather than searching for neurons or neural networks specifically dedicated to consciousness *per se* (Cohen and Dennett, 2011). Currently, there is not strong evidence for or against the view that the VAN component reflects object-based attention instead of visual awareness. In addition to developing experimental paradigms which can better isolate NCC-proper from pre-conscious and post-perceptual activity (the focus of the current special issue), it is imperative that researchers craft experimental designs that improve our chances of distinguishing between neural correlates of object-based attention and neural correlates of awareness.

#### **ACKNOWLEDGMENTS**

The authors would like to thank the reviewers for their insightful comments, especially regarding alternative approaches to data analysis. We would also like to thank Michael Cohen for constructive comments on earlier drafts of this manuscript, and Enriqueta Canseco-Gonzalez for fruitful discussions regarding statistical analyses.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fpsyg. 2014.01078/abstract

**Movie 1 | Video depicting an example sequence of trials from the EEG experiment (instructions refer to a block of trials in which shapes were task-relevant).**

#### **REFERENCES**


Pitts, M. A., and Martinez, A. (2014). "Contour integration: sensory, perceptual, and attention-based ERP components," in *Cognitive Electrophysiology of Attention: Signals of the Mind*, ed G. R. Mangun (San Diego, CA: Academic Press), 178–189. doi: 10.1016/B978-0-12-398451-7.00014-2


in the absence of report. *J. Cogn. Neurosci.* 26, 955–969. doi: 10.1162/ jocn\_a\_00530


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 14 June 2014; accepted: 08 September 2014; published online: 08 October 2014.*

*Citation: Pitts MA, Metzler S and Hillyard SA (2014) Isolating neural correlates of conscious perception from neural correlates of reporting one's perception. Front. Psychol. 5:1078. doi: 10.3389/fpsyg.2014.01078*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Pitts, Metzler and Hillyard. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## A cross-modal investigation of the neural substrates for ongoing cognition

## *Megan Wang† and Biyu J. He\**

*National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA*

#### *Edited by:*

*Jaan Aru, University of Tartu, Estonia*

#### *Reviewed by:*

*Karim Jerbi, Institut National de la Santé et de la Recherche Médicale, France Mariano Sigman, Universidad Torcuato di Tella, Argentina*

#### *\*Correspondence:*

*CA, USA*

*Biyu J. He, National Institute of Neurological Disorders and Stroke, National Institutes of Health, 10 Center Drive, Building 10, Room B1D728, Bethesda, MD 20892, USA e-mail: biyu.he@nih.gov*

*†Present address: Megan Wang, Ph.D. Program in Neuroscience, Stanford University* What neural mechanisms underlie the seamless flow of our waking consciousness? A necessary albeit insufficient condition for such neural mechanisms is that they should be consistently modulated across time were a segment of the conscious stream to be repeated twice. In this study, we experimentally manipulated the content of a story followed by subjects during functional magnetic resonance imaging (fMRI) independently from the modality of sensory input (as visual text or auditory speech) as well as attentional focus. We then extracted brain activity patterns consistently modulated across subjects by the evolving content of the story regardless of whether it was presented visually or auditorily. Specifically, in one experiment we presented the same story to different subjects via either auditory or visual modality. In a second experiment, we presented two different stories simultaneously, one auditorily, one visually, and manipulated the subjects' attentional focus. This experimental design allowed us to dissociate brain activities underlying modality-specific sensory processing from modality-independent story processing. We uncovered a network of brain regions consistently modulated by the evolving content of a story regardless of the sensory modality used for stimulus input, including the superior temporal sulcus/gyrus (STS/STG), the inferior frontal gyrus (IFG), the posterior cingulate cortex (PCC), the medial frontal cortex (MFC), the temporal pole (TP), and the temporoparietal junction (TPJ). Many of these regions have previously been implicated in semantic processing. Interestingly, different stories elicited similar brain activity patterns, but with subtle differences potentially attributable to varying degrees of emotional valence and self-relevance.

**Keywords: ongoing cognition, semantic processing, linguistic processing, cross-modal, default-mode network, fMRI, conscious flow**

## **INTRODUCTION**

Imagine you are at a New Year's party. A friend is recounting her recent trip to New Zealand, meanwhile the television in front of you is playing a tennis match. You find yourself staring at the TV without following the game, and fully absorbed in your friend's exciting story. Alternatively, if you are a tennis fan, you might find yourself following the game and missing part of your friend's story, recognizing the need to correct your attentional focus. In both cases, the sensory inputs to your brain are identical while the stream of your conscious content is rather different. What brain mechanisms might contribute to the ongoing flow of your conscious mind beyond sensory inputs (James, 1890; Dehaene and Sigman, 2012)?

One initial approach to investigating this question is to identify brain areas reliably modulated by similar content of conscious flow in the face of changing sensory inputs. Since it cannot be excluded that certain unconscious processes are also consistently modulated across time in such a paradigm, consistent modulation in the face of similar conscious stream constitutes a necessary albeit insufficient condition for identifying brain activities potentially underlying internal conscious flow. Alternatively, reliable modulation by the same sensory inputs in the presence of different conscious content, as in the example above, would reveal lower-level processing of sensory stimulus. Because both the sensory stimulus and the conscious flow in this example are time-varying features, "reliable modulation" means a similar temporal response profile across time. The approach of using reliable modulation by naturalistic stimuli to probe a brain region's involvement in stimulus processing has been successfully applied to both within- and across- subject analyses (Hasson et al., 2010). We extended this approach to studying brain areas consistently modulated by the evolving content of a story independent of the modality of sensory input, by presenting subjects with auditory speech or visual text. Because listening to or reading a story would not only trigger linguistic processing, but also post-linguistic processes such as imagery, theory-of-mind, episodic and emotional processing, we hereafter refer to the totality of these processes "*ongoing cognition.*" Importantly, these processes may include brain activities directly underlying the conscious flow as well as their prerequisites and consequences (Bachmann, 2009; Aru et al., 2012; de Graaf et al., 2012; Li et al., 2014).

Many previous studies have studied the convergence of neuroanatomy for the processing of spoken and written languages (Chee et al., 1999; Calvert, 2001; Spitsyna et al., 2006; Jobard et al., 2007; Lindenberg and Scheef, 2007). These previous studies have generally assessed the activation magnitude of a brain region in response to spoken or written language, upon which convergence (i.e., activation in both tasks) was determined. Alternatively, an interaction effect is sometimes determined in multisensory integration studies (Raij et al., 2000; Calvert, 2001; van Atteveldt et al., 2004), which quantifies the activation magnitude to the simultaneous presentation of both modalities beyond the sum of activation magnitudes to each modality presented alone. However, the measure of activation magnitude provides only a crude estimation of a brain region's involvement in the task. For example, a brain region can be activated in both the auditory and visual tasks but with distinct temporal modulation profiles, which would indicate different kinds of processing in the two tasks. By contrast, reliable cross-modal modulation of the temporal response profile of a brain region during an evolving story presented as visual text or auditory speech would constitute stronger evidence for its involvement in the ongoing cognition elicited by the story.

We conducted two experiments. In the first experiment, we presented a story ("Cage") to different subjects via either visual or auditory presentation. Specifically, the visual and auditory presentations were controlled to advance at roughly the same speed. We identified the brain regions exhibiting similar time courses across these two subject groups. Because the sensory inputs are presented through different modalities but the story content is the same, these brain regions are expected to underlie modality-invariant linguistic and post-linguistic processes. In the second experiment, we presented two different stories ("Fish" and "King") simultaneously to the subjects, one auditorily, one visually. In different functional magnetic resonance imaging (fMRI) runs, the subject was cued to pay attention to one modality vs. the other, thus following different stories. Correlating brain activities during an identical task condition across subjects allowed us to identify brain regions consistently modulated by this task condition. By contrast, correlating brain activities between task conditions that had identical physical stimuli but different attended sensory modalities (and thus different stories) allowed us to extract brain activities modulated purely by the sensory inputs.

## **MATERIALS AND METHODS SUBJECTS**

Twenty-seven healthy right-handed English-fluent subjects between 19 and 38 years of age (8 males) with normal or corrected-to-normal vision participated in the study. All subjects provided written informed consent. The experiment was approved by the Institutional Review Board of the National Institute of Neurological Disorders and Stroke. Seven subjects were excluded due to excess movement in the scanner, and two additional subjects were excluded due to failed registration to the atlas. Thus, eighteen subjects (6 males) were included in the analysis.

#### **STIMULI AND TASK DESIGN**

Three short narratives, referred to as the "Cage," "King" and "Fish" stories, were presented visually and auditorily. "Cage" was compiled from the Wikipedia entry on John Cage's composition *4*- *33*-and was used in Experiment 1. "King" is the short story "The Three Questions" by Leo Tolstoy and "Fish" is a short story called "Fred's Fish"; they were used in Experiment 2. Complete transcripts for the three stories and the experimental stimuli used in all conditions can be found in Supplementary Materials. These stories were chosen as materials that most subjects would not be familiar with, to ensure that subjects would have to attend carefully to comprehend the stories. For the auditory version of these stories, a female native English speaker recorded each story with a Logitech H530 headset and edited the recording using Audacity 1.3.13-beta (e.g., by removing breathing artifacts). The visual version was presented in subtitles format using Aegisub 2.1.8, such that each phrase was on screen for the same duration as it was spoken in the auditory version. During the pauses between sentences, a cross-hair was presented at the center of the screen such that the screen was never blank. In Experiment 2, the visual and auditory stories, which had the same duration, were combined in MeGUI to create a stimulus consisting of "Fish" story subtitles simultaneously presented with the "King" story audio recording (FishV+KingA), and a second stimulus consisting of "King" story subtitles simultaneously presented with the "Fish" story audio recording (KingV+FishA).

In Experiment 1, half of the subjects (*N* = 9) followed the "Cage" story presented visually while the other half (*N* = 9) followed the "Cage" story presented auditorily during fMRI scan (**Figure 1A**). In the visual condition, subjects read the story presented phrase-by-phrase at the center of the screen. In the auditory condition, subjects listened to the story presented via headphones (Avotec Inc., FL) while looking at a blank screen (visual fixation was not required). The detailed structure of Experiment 1 is as follows: 10 s of blank, 390 s of stimulus, 5 s of blank, 5 multiple-choice questions probing the comprehension of the story presented for 10 s each, and finally 5 s of blank (**Figure 1A**). Subjects were asked to answer each question during the 10-s interval using one of four buttons. Occasionally subjects pressed outside the allotted time interval or pressed answers twice; in those cases, answers were confirmed verbally immediately after the run ended.

In Experiment 2, subjects were divided into two stimulus groups, and instructed to attend to one modality in one fMRI run and the other modality in the second fMRI run, in a counterbalanced manner. Half of the subjects (*N* = 9) were presented the FishV+KingA stimulus, with five of them instructed to attend to the visual story ("Fish") in the first run and then to the auditory story ("King") in the second run, and four subjects instructed with the opposite order (**Figure 1B**, Group 1). The remaining nine subjects were presented with the KingV+FishA stimulus, again with the attending sequence counter-balanced across subjects (**Figure 1B**, Group 2). In the "Attend to Visual" (AV) condition, subjects were told to ignore the auditory input and focus on reading the story presented at the center of the screen. In the "Attend to Auditory" (AA) condition, subjects were instructed to focus on listening to the story presented auditorily, and to maintain fixation at the center of the screen but ignore the story presented visually.

Each of the two runs in Experiment 2 contained 10 s of blank, 340 s of stimulus, 5 s of blank, 1 question on intrusion presented for 10 s, 5 multiple-choice comprehension questions presented

in the second run; the order of attended modality was counterbalanced within each group. **(C)** Schematic of analysis approach. In Experiment 1, correlations were computed between the two groups of subjects for each ellipses at the bottom illustrate the analysis depicted in **Figure 4**, with each ellipse indicating the significant brain areas from an analysis. For detailed methods see Materials and Methods.

for 10 s each, and finally 5 s of blank (**Figure 1B**). The intrusion question asked the subject to rate the level of intrusion by the unattended story, from 1 (no intrusion) to 4 (full intrusion, or followed both stories equally). Subjects answered the question using one of four buttons. Comprehension questions tested the comprehension of the attended story only, to ensure that the subject was following instructions by attending to the correct sensory stream.

## **EYE TRACKING**

Subjects' eyes were monitored throughout the entire experiment using an MR-compatible eye tracker (NordicNeuroLab Inc., WI) through which the visual stimulus was presented. The eye tracker was calibrated at the beginning and end of the experiment, and more frequently as needed if there was excess head movement. We monitored the subjects' eyes closely to ensure that they were following instructions. That is, the subjects' gaze fixated on the center of the screen when instructed to attend to the auditory stream, and they were seen to make quick saccades when instructed to attend to the visual stream.

## **fMRI DATA ACQUISITION**

Functional and anatomical MRI was conducted on a General Electric 3T scanner with an 8-channel head coil. Anatomical images were obtained using a sagittal magnetization-prepared rapid-acquisition gradient echo (MP-RAGE) sequence with a resolution of 1 <sup>×</sup> <sup>1</sup> <sup>×</sup> <sup>1</sup> *mm*3. An axial T2-weighted structural scan was acquired with *TR* = 4200 ms, *TE* = 120 ms and a resolution of <sup>3</sup> <sup>×</sup> <sup>3</sup> <sup>×</sup> <sup>3</sup> *mm*3. BOLD-contrast functional images were obtained using a single-shot gradient echo sequence with 39 contiguous transverse slices covering the whole brain (slice thickness = 3 mm, in-plane resolution: 3 <sup>×</sup> <sup>3</sup> *mm*2, *TR* <sup>=</sup> 2000 ms, *TE* <sup>=</sup> 27 ms, flip angle = 90◦).

#### **fMRI DATA PREPROCESSING**

fMRI data were preprocessed as follows: (1) compensation of systematic, slice-dependent time shifts; (2) elimination of systematic odd-even slice intensity difference due to interleaved acquisition; (3) rigid body correction for inter-frame head motion within and across runs; and (4) intensity scaling to yield a whole-brain mode value of 1000 (with a single scaling factor for all voxels). Atlas registration was achieved by computing affine transforms connecting the fMRI run first frame (averaged over all runs after cross-run realignment) with the T2- and T1-weighted structural images. Our atlas representative template included MP-RAGE data from 12 normal individuals and was made to conform to the 1988 Talairach atlas (Talairach and Tournoux, 1988). Data were resampled to 3 <sup>×</sup> <sup>3</sup> <sup>×</sup> <sup>3</sup> *mm*<sup>3</sup> voxels after atlas registration.

fMRI signals from each run were detrended and the effect of head motion and its temporal derivative were removed by linear regression. We further removed the effect of low-level physical attributes of the stimuli to avoid inter-subject correlations driven by low-level transients in the visual and auditory stimuli, similar to the method used in Honey et al. (2012a). To this end, we determined the transients in the visual stimuli and the sound envelope of the auditory stimuli. For visual transients, the phrase-to-fixation transitions, fixation-to-phrase transitions and phrase-to-phrase transitions were each modeled as a series of delta functions. For the sound envelope, the audio signal was bandpassed between 4 and 4000 Hz and the envelope was extracted using a Hilbert transform. These four regressors (three types of visual transients and the sound envelope) were each convolved with the hemodynamic response function, downsampled to the sampling rate of the fMRI signal (*TR* = 2 s), and removed from the fMRI data via linear regression. Finally, data from each subject were spatially smoothed with a Gaussian kernel (FWHM = 6 mm).

#### **WITHIN- AND ACROSS- MODALITY RESPONSE RELIABILITY**

Similar to previous studies (Hasson et al., 2004; Honey et al., 2012a), we assessed the correlations of the fMRI signals during stimulus presentation (length: 390 s in Experiment 1; 340 s in Experiment 2) across subjects at each voxel. The first 5 frames of each fMRI run corresponded to the blank period and were not included in the correlation. Thus, scanner magnetic stabilization was already reached by the beginning of stimulus presentation. In Experiment 1, for within-condition reliability assessment, Groups 1 and 2 (see **Figure 1A**) were analyzed separately. fMRI signal time course from each subject was correlated with the mean time course across the remaining subjects in the same group. The Pearson correlation r values were transformed into Fisher *z*-values, which are approximately normally distributed. The Fisher z maps were then averaged across all subjects in each group (*N* = 9) to yield the population average. For across-condition reliability assessment, fMRI signal correlations were evaluated between Groups 1 and 2 that followed the same story presented via visual and auditory modality, respectively (**Figure 1A**). Each subject's fMRI signal time course was correlated with the average time course from the other group. The Pearson correlation r values were transformed into Fisher *z*-values, which were then averaged across all subjects in both groups (*N* = 18) to yield the population average.

In Experiment 2, the correlations were carried out across subjects presented with identical physical stimuli (i.e., between Groups 1A and 1B, and between Groups 2A and 2B, see **Figure 1B**). Two analyses were carried out. In the first, we correlated fMRI runs in which one subgroup attended to the visual stream and the other subgroup attended to the auditory stream (e.g., Run 2 of both Groups 1A and 1B). To avoid repetition suppression effect (Grill-Spector and Malach, 2001), only the fMRI runs in which the stimulus was presented for the first time were used (Run 2 in **Figure 1B**). Because the correlation was carried out between subjects presented with identical physical stimulus but attending to different sensory modalities and thus different stories, the brain regions showing reliable responses should be those involved in low-level sensory processing. In the second analysis, we correlated fMRI runs in which the two subgroups of subjects were presented with identical stimulus and attended to the same sensory stream (e.g., Run 2 from Group 1A and Run 3 from Group 1B), which assessed which brain regions were consistently modulated by each task condition. In both analyses, the fMRI signal time course from each subject was correlated with the average time course from the other subgroup. The Fisher-ztransformed correlation maps were averaged across all subjects in each group (*N* = 9, since Groups 1 and 2 were analyzed separately).

#### **BOOTSTRAPPING BY PHASE-RANDOMIZATION TO ASSESS SIGNIFICANCE**

We assessed statistical significance using a bootstrapping procedure based on phase-randomization. For each voxel, we applied Fourier transform on the time series, randomized the phase component, and inverted the Fourier transform to obtain the shuffled time series. For Experiment 1, each subject's time series was phase-shuffled and correlated with the original average of the other group; this was done 50 times per subject to create a distribution of 900 bootstrapped correlations. For Experiment 2, bootstrapping was performed 100 times per subject, again to yield a distribution of 900 bootstrapped correlations for each analysis (Groups 1 and 2 were analyzed separately). All of the bootstrap correlations were transformed into Fisher *z*-values. We then calculated the mean and standard deviation (SD) across the distribution of 900 bootstrap iterations. Because in the original analysis the correlations were averaged across subjects, the SD of the bootstrap distribution was corrected by a factor of <sup>√</sup>*N*, where *<sup>N</sup>* <sup>=</sup> 18 in Experiment 1 and *<sup>N</sup>* <sup>=</sup> 9 in Experiment 2. The mean of the bootstrap distribution and the corrected SD were then used to convert the original population-average Fisher-z maps into Z-scores, from which statistical significance was determined. To correct for multiple comparisons, we adopted the Monte Carlo method for family-wise error (FWE) correction (McAvoy et al., 2001) and applied a threshold of Z score >3 and cluster size >17 voxels, yielding clusters that survived *p* < 0.05.

## **RESULTS**

To investigate ongoing cognition using controlled semantic content, we presented stories as auditory speech and/or visual text to subjects and correlated the fMRI time series across subjects to map brain areas that responded reliably to a task condition or across different conditions. We applied this correlational approach to identify brain regions underlying sensory processing vis-à-vis ongoing cognition.

#### **BEHAVIORAL RESULTS**

For the "Cage," "King," and "Fish" stories, subjects correctly answered an average of 4.4 ± 0.17 (mean ± s.e.m. across 18 subjects), 4.2 ± 0.23 and 4.3 ± 0.16 comprehension questions, respectively, and there was no significant difference between stories (*p* = 0.73, Kruskal-Wallis test). In Experiment 1, the level of comprehension was not significantly different between subjects who heard the story and those who read the story (*p* = 0.09, Wilcoxon rank-sum test). In Experiment 2, there was no significant effect of the attended modality (*p* = 0.26, Wilcoxon signedrank test) or run order (*p* = 0.97) on the level of comprehension of the attended story.

In Experiment 2, we asked an additional question concerning the level of intrusion by the unattended story. The intrusion level averaged across all runs from all subjects was 2.1 ± 0.13. It was not significantly different between the AV and AA conditions (*p* = 0.49, Wilcoxon signed-rank test), or between the first and second fMRI runs (*p* = 0.38, Wilcoxon signed-rank test). Interestingly, when attending to the "King" and "Fish" stories, subjects reported an average intrusion level of 2.5 ± 0.15 and 1.8 ± 0.19, respectively (*p* = 0.01, Wilcoxon signed-rank test), suggesting that the "Fish" story was more intrusive. This is likely due to the fact that the "Fish" story was told in first-person perspective and had more emotional and personal content while the "King" story was a fable told in third-person perspective.

### **EXPERIMENT 1—"CAGE" STORY PRESENTED ALONE VIA VISUAL OR AUDITORY MODALITY**

In Experiment 1, one group of subjects (*N* = 9) listened to the "Cage" story through headphones, and another group of subjects (*N* = 9) read the "Cage" story presented visually at the center of the screen. The auditory and visual versions of the story were presented at roughly the same speed. First, we identified brain regions that were reliably modulated across subjects within each condition alone. To this end, we computed inter-subject correlations for each voxel within the first group of subjects that read the "Cage" story (**Figure 2A**, top row), and within the second group of subjects that listened to the "Cage" story (**Figure 2A**, bottom row). Unsurprisingly, in the reading ("V") condition, the occipital visual cortices, as well as the intraparietal sulci (IPS) involved in visuospatial attention, are reliably modulated. By contrast, in the listening ("A") condition, there was extensive reliable modulation of the early and higher-order auditory cortices along the superior temporal gyrus (STG). Both the reading and listening conditions consistently modulated the inferior frontal gyrus (IFG), temporal pole (TP), the superior temporal sulcus (STS), anterior cingulate cortex (ACC), and the thalami. Interestingly, the posterior cingulate cortex (PCC) was substantially more involved in the listening condition than the reading condition.

The above results reveal a common set of brain regions that are reliably modulated in both reading and listening conditions. Nonetheless, it remains unknown whether their temporal response profiles are similar across these two conditions. To address this question, we correlated the fMRI time series across these two groups of subjects to extract brain regions reliably modulated by the "Cage" story regardless of the sensory modality used for stimulus input. The results are shown in **Figure 2B** (*p* < 0.05,

**FIGURE 2 | Results of Experiment 1. (A)** Within-condition reliability map. fMRI time series were correlated between subjects who read the "Cage" story (top row) and between those who listened to the "Cage" story (bottom row). The Pearson correlation coefficients were z-transformed and averaged across all subjects in each group (*N* = 9). **(B)** Across-condition reliability map. fMRI time series were correlated between subjects who listened to the "Cage" story and those who read the same story (V × A, see **Figure 1C**, Experiment 1). The population-average z-transformed correlation values were compared to phase-shuffled data to determine the *Z*-score (thresholded at *Z* > 3, cluster size >17 voxels, corresponding to *p* < 0.05 after correction for multiple comparisons).

FWE corrected), which included bilateral STS/STG, TP, and IFG, the left temporal parietal junction (TPJ), the dorsal medial prefrontal cortex (dmPFC) and the thalamus. There was a slight left asymmetry in the response pattern.

#### **EXPERIMENT 2—"FISH" AND "KING" STORIES PRESENTED SIMULTANEOUSLY VIA AUDITORY AND VISUAL MODALITIES**

In Experiment 2, one group of subjects (*N* = 9) were simultaneously presented with the "Fish" story as visual text and the "King" story as auditory speech (FishV+KingA), and instructed to attend to the two sensory streams in alternate runs in a counterbalanced manner (Group 1 in **Figure 1B**). A second group of subjects (*N* = 9) were presented with the "King" story as visual text and the "Fish" story as auditory speech (KingV+FishA) and also instructed to attend to different sensory streams in alternate runs (Group 2 in **Figure 1B**).

We first examined which brain regions were consistently modulated by this task. To this end, we correlated fMRI time series across subjects under an identical task condition. Given two stimulus conditions ("FishV+KingA" and "KingV+FishA") and two attentional states (AV and AA), there were four task conditions in total. Thus, four correlational analyses were carried out between subjects exposed to identical task stimuli and instructions (e.g., between Run 2 from Group 1A and Run 3 from Group 1B). The results from this analysis are shown in **Figure 3** (top and

in the top row, AV × AV; attending the auditory stream in the middle row, AA × AA). In the bottom row, fMRI time series were correlated across subjects

*Z*-scores (*p* < 0.05, corrected for multiple comparisons).

middle rows). Widespread brain regions were consistently modulated by this task, with the strongest activities residing in the lateral occipital cortex (LOC), STG, PCC, TP, IFG, and the TPJ.

To locate brain areas involved in the processing of sensory stimulus regardless of attentional focus and the story followed, we correlated fMRI time series between subjects presented with identical physical stimulus but instructed to attend to opposite sensory streams, such that the attended sensory modality (visual vs. auditory) and the story followed ("Fish" vs. "King") differed between the correlated runs (i.e., Run 2 was correlated between Groups 1A and 1B, and between Groups 2A and 2B). The results of this analysis are shown in the bottom row of **Figure 3**. Unsurprisingly, auditory cortex along the STG showed reliable responses. In addition, ventral visual areas, thalamus, dmPFC and part of the angular gyrus (AG) were involved. Interestingly, there was limited recruitment of the primary visual cortex (V1), likely because the eye movement pattern differed between the AV and AA conditions, under which the subjects performed active reading and passive fixation, respectively.

The above two analyses respectively extracted brain regions reliably modulated by performing this task (i.e., being exposed to simultaneous auditory and visual streams and attending to one of them) and those reliably modulated by the physical sensory stimuli regardless of the required attentional focus and the story content followed. Hence, contrasting them should reveal brain areas involved in "ongoing cognition" beyond sensory inputs—that is, from attentional fluctuations (if they were similar across subjects) and the understanding of the evolving story content to post-semantic processes such as imagery and emotional response. In **Figure 4**, we overlaid the results from the first analysis showing regions consistently modulated by performing this task (shown in yellow, from the top two rows in **Figure 3**) and those from the second analysis showing regions consistently modulated by the physical stimulus alone (shown in blue, from the bottom row in **Figure 3**), with their overlaps shown in orange. For example, in the top-left panels, yellow/orange regions are those reliably modulated when subjects were presented with the KingV+FishA stimulus and attended to the visual stream; the blue/orange regions were those consistently modulated between the AV and AA conditions under the KingV+FishA stimulus. Thus, regions in yellow represent those contributing to ongoing cognition beyond the processing of physical sensory inputs. Next, we extracted these regions and investigated whether, and if so how, their patterns depended on the attended sensory modality.

To compare the putative brain areas involved in ongoing cognition between the AV and AA conditions, we combined the yellow regions in **Figure 4** across the "King" and "Fish" stories for the AV (**Figure 4**, left column) and AA (right column) conditions, respectively. The results are shown in **Figure 5**. As expected, low-level visual areas (Brodmann areas 17 and 18) were more reliably modulated in the AV condition, and the primary auditory cortex (Brodmann areas 41 and 42) was more reliably modulated in the AA condition. These results suggest that primary sensory areas may also contribute to ongoing cognition, depending on the attentional focus.

#### **DISCUSSION**

In summary, we used an audiovisual cross-modal approach to map brain activity patterns underlying ongoing cognition elicited by hearing or reading a story, and to dissociate these activity patterns from modality-specific sensory processing. The brain areas we identified bear significant similarities to previously described semantic network (Martin, 2007; Patterson et al., 2007; Binder et al., 2009; Schwartz et al., 2011; Honey et al., 2012a; Wei et al., 2012; Fairhall and Caramazza, 2013) and the default-mode network (DMN) (Raichle et al., 2001; Buckner et al., 2008), adding further support for the anatomical convergence of these different brain functions.

In the first experiment, we presented a factual story about the composition of *4*- *33*- by John Cage to subjects via either the auditory or visual modality. The visual presentation of the story was synchronized phrase by phrase with the auditory version, which comprised of natural reading by a native English speaker. By correlating brain activity fluctuations between subjects who listened to the story and those who read the story, we were able to extract brain regions consistently modulated by this story regardless of the sensory modality used for stimulus delivery. The most consistently modulated brain areas included the STS/STG, the IFG, and the TP (**Figure 2**), all of which are well-known nodes within the semantic network (Patterson et al., 2007; Binder et al., 2009). Our results from Experiment 1 are largely similar to a previous study that used very similar experimental design (Regev et al., 2013). The main difference between our results and theirs is that we did not find robust cross-modality reliable modulation in PCC (**Figure 2B**), which is consistent with its lack of reliable modulation within the visual condition alone (**Figure 2A**, top). This difference might result from the fact that Regev et al. used a real-life story with autobiographical content, which is known to activate the PCC (Buckner et al., 2008), while the "Cage" story we used in Experiment 1 is an encyclopedic entry about a piece of controversial musical history.

In the second experiment, by presenting two stories simultaneously to the subjects via visual and auditory modalities, and

cueing them to attend to one or the other in different runs, we were able to manipulate the content of ongoing cognition under identical physical stimulus. This attentional modulation was successful, as demonstrated by the low level of intrusion from the unattended story (rated ∼2 on a scale from 1 to 4) and the satisfactory comprehension of the attended story (on average, subjects answered 4.2 and 4.3 out of 5 questions correctly for "King" and "Fish," respectively). We extracted brain regions consistently modulated by performing this task (by correlating fMRI signals across subjects under an identical task condition), and those modulated by the physical stimuli alone (by correlating fMRI signals between task conditions with identical stimulus but opposite attentional focus). Contrasting these results, we found an extensive brain network consistently modulated by ongoing cognition beyond physical sensory inputs, which had very similar spatial patterns for the two stories (**Figure 4**). The main regions involved included the TPJ, IFG, TP, PCC, and the posterior middle temporal gyrus (pMTG), consistent with previous studies on semantic processing (Binder et al., 2009; Honey et al., 2012a; Wei et al., 2012; Fairhall and Caramazza, 2013). Nonetheless, there were some subtle differences between the activation patterns in response to the two stories. First, the "Fish" story elicited stronger response in the retrosplenial cortex, potentially due to the strong autobiographical nature of this story. Second, in both hemispheres, the activation pattern around the TPJ was continuous for the "King" story, while it comprised of three disjoint regions in the SMG, AG, and pMTG in response to the "Fish" story. At present the origin of this difference is unclear. One potential contributor is the different levels of theory-of-mind processing involved (Buckner et al., 2008; Corbetta et al., 2008; Carter and

Huettel, 2013): In particular, the "King" story, which involves a substantial amount of conjecturing of the character's mental state, evoked an activation pattern around the TPJ that is very similar to previous findings on theory-of-mind (see Figure 12C in Buckner et al., 2008).

Interestingly, the regions identified in Experiment 2 were more extensive than those found in Experiment 1. Two factors may have contributed to this difference: First, the comparative analysis illustrated in **Figure 4** may not have completely removed brain regions involved in modality-specific processing. Specifically, since we used conservative whole-brain multiple comparisons correction (*p* < 0.05, FEW corrected) to identify brain areas modulated by the physical stimuli (**Figure 3** bottom row, show as blue/orange in **Figure 4**), there may well be brain areas that we did not have power to detect with this statistical threshold. Second, the difference in story content may have contributed to the more extensive response patterns in Experiment 2: While "Cage" is an encyclopedic entry, "King" and "Fish" are vivid fable and personal stories that are likely to elicit stronger imagery, theory of mind, episodic, and emotional processing.

Many of the brain areas identified in this study as underlying ongoing cognition are well-known components of the DMN, including the PCC, dmPFC, the inferior parietal lobule, and the MTG (Raichle et al., 2001; Fox et al., 2005; Buckner et al., 2008). Previous studies have established that the DMN is activated during "task-independent thought," suggesting that it may be involved in spontaneous cognition (McGuire et al., 1996; Mason et al., 2007; Christoff et al., 2009). Our results extend these previous findings by showing that the DMN time courses are similarly modulated over time across different individuals by the ongoing cognition evoked by a story regardless of the sensory modality used for story presentation, supporting the idea that the *continuous* activity fluctuations in the DMN are reliably modulated by the evolving flow of conscious content (He and Raichle, 2009).

The similarity between the "semantic network" and the DMN has long been noted (Binder et al., 1999, 2009; Wei et al., 2012; Fairhall and Caramazza, 2013). To account for this observation, Binder et al. (1999, 2009) proposed that semantic processing constitutes a large component of spontaneous thoughts under the resting state, during which the DMN is typically more active. The present study identified brain areas consistently modulated by the evolving content of a story, which relies on semantic processing. Thus, the similarity between the present results and the semantic network as well as the DMN is not surprising. Nonetheless, we believe that potential differences in the spatial patterns among the DMN, the semantic network and the brain areas involved in ongoing cognition should be an interesting and important topic for future research. For example, previous studies found that in addition to the DMN, the executive network including the dorsolateral prefrontal cortex (DLPFC) and the dorsal anterior cingulate cortex (dACC) were also involved in task-independent thoughts (Christoff et al., 2009; Spreng et al., 2010). This is similar to our results (**Figures 2**, **4**). Future studies employing detailed dissection of how the specific content of ongoing cognition or semantic processing relates to specific brain activity patterns would be needed to shed light on the intricate functional/anatomical brain architecture supporting these functions, and how these structures intertwine with the subsystems of the DMN (Andrews-Hanna et al., 2010). A promising approach to this end is demonstrated by a recent study using a data-driven generative model applied to fMRI data obtained under natural movie viewing, which revealed a continuous semantic space across the cortical surface (Huth et al., 2012). As the authors noted in that paper, because only visual stimuli were used in their study, visual and conceptual features were likely mixed in their results. Combining the generative model used therein and the present cross-modal approach in the context of natural stimuli should allow future studies to map the hetero-modal semantic space in the human brain.

As mentioned in Introduction, consistent temporal modulation in the presence of similar conscious stream constitutes a necessary albeit insufficient condition for identifying brain activities underlying the conscious flow. An important future direction is to dissociate brain activities directly contributing to the conscious flow and those consistently modulated by the unconscious processes related to the conscious flow, in line with the recently proposed framework of the tripartite process including prerequisites for the neural correlate of consciousness (NCC-pr), neural correlate of consciousness proper (NCC), and consequences of the neural correlate of consciousness (NCC-co) (Bachmann, 2009; Aru et al., 2012; de Graaf et al., 2012; Li et al., 2014). We anticipate that novel cognitive paradigms and/or analytical approaches will need to be developed in order to separate these processes in the context of ongoing cognition. Lastly, the use of electrophysiological recordings (e.g., Honey et al., 2012b) in the context of these paradigms should shed additional light on the underlying neural mechanisms.

## **ACKNOWLEDGMENTS**

This research was supported by the Intramural Research Program of the NIH/NINDS. We are grateful to Dr. Alex Martin for helpful discussions and feedback on a previous draft, as well as Dr. John Ingeholm for assistance with the eye-tracking system.

## **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fpsyg.2014. 00945/abstract

## **REFERENCES**


He, B. J., and Raichle, M. E. (2009). The fMRI signal, slow cortical potential and consciousness. *Trends Cogn. Sci.* 13, 302–309. doi: 10.1016/j.tics.2009.04.004


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 08 June 2014; accepted: 07 August 2014; published online: 26 August 2014. Citation: Wang M and He BJ (2014) A cross-modal investigation of the neural substrates for ongoing cognition. Front. Psychol. 5:945. doi: 10.3389/fpsyg.2014.00945 This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Wang and He. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Perceptual and contextual awareness: methodological considerations in the search for the neural correlates of consciousness

## *Joaquin Navajas, Hernan G. Rey and Rodrigo Quian Quiroga\**

*Centre for Systems Neuroscience, University of Leicester, Leicester, UK*

#### *Edited by:*

*Jaan Aru, University of Tartu, Estonia*

#### *Reviewed by:*

*Ned Block, New York University, USA Jacobo Diego Sitt, Commissariat à l'énergie atomique et aux énergies alternatives, France*

#### *\*Correspondence:*

*Rodrigo Quian Quiroga, Centre for Systems Neuroscience, 9 Salisbury Road, Leicester, LE1 7QR, UK e-mail: rqqg1@le.ac.uk*

In the last decades, the neural correlates of consciousness (NCCs) have been explored using both invasive and non-invasive recordings by comparing the brain activity elicited by seen versus unseen visual stimuli (i.e., the contrastive analysis). Here, we review a selection of these studies and discuss a set of considerations to improve the search for the NCCs using the contrastive analysis. In particular, we first argue in favor of implementing paradigms where different perceptual outputs are obtained using identical visual inputs. Second, we propose that the large disagreement in the field -in terms of the dissimilar neural patterns proposed as NCCs- is partially explained by the fact that different studies report the neural correlates of different conscious processes in the brain. More specifically, we distinguish between the *perceptual awareness* of a visual stimulus, associated to a boost in object-selective neural assemblies, and a more elaborate process (*contextual awareness*) that we argue is reflected in the firing of concept neurons in the medial temporal lobe, triggering a rich representation of the context, associations, and memories linked to the specific stimulus.

**Keywords: consciousness, NCCs, contrastive analyses, perceptual awareness, contextual awareness, physical confounds**

#### **INTRODUCTION**

When we see a picture of a person, our retinal cells transduce light into electrical signals propagated through the brain, triggering a cascade of neural processes that leads to the conscious percept of the specific person we are looking at. The minimal neuronal mechanisms that are jointly sufficient to elicit a specific conscious percept are known in the literature as the neural correlates of consciousness (NCCs; Crick and Koch, 1990).

In order to empirically manipulate awareness, different methods were developed in the past to render a stimulus invisible despite retinal stimulation (Kim and Blake, 2005). For example, a brief stimulus that is normally visible can become invisible if it is preceded or followed by a second one; a phenomenon called "visual masking" (Enns and Di Lollo, 2000). With "Attentional Blink" (AB), the perception of a salient target presented in rapid visual serial presentation (RSVP) is impaired by the detection of a previous stimulus (Raymond et al., 1992). Similarly, when two clearly different images are sequentially shown separated by a brief blank interval, observers typically fail to detect the change in the images, leading to a manipulation called "Change Blindness" (CB; Simons and Rensink, 2005). These experimental manipulations have in common that the stimulus is transient, i.e., it is presented for a short period of time. However, other techniques allow inducing lack of awareness even with prolonged retinal stimulation. For example, during "Binocular Rivalry" (BR) two disparate images are presented to each eye, causing a sequence of subjective perceptual switches experienced by the observer, suppressing one or the other image despite constant visual stimulation (Blake and Logothetis, 2002). The main downside of BR is that the number of

subjective alternations, along with their duration and latencies, are not under experimental control. This issue is absent in a similar technique called "Flash Suppression" (FS) in which one image is presented to one eye, and then is removed from visual awareness by suddenly presenting another image to the other eye (Lansing, 1964; Wolfe, 1984). In the same line, Tsuchiya and Koch (2005) introduced another manipulation, called "Continuous Flash Suppression" (CFS), in which robust and prolonged interocular suppression is achieved by presenting flickering patterns to one eye.

Many previous studies have aimed at finding the NCCs by combining these experimental procedures with different measures of neural activity such as scalp magneto/electro-encephalography (M/EEG; Dehaene et al., 2001; Sergent et al., 2005), functional magnetic resonance imaging (fMRI; Lumer et al., 1998; Portas et al., 2000), intracranial EEG (Fisch et al., 2009; Gaillard et al., 2009), and single-cell recordings in human (Kreiman et al., 2002; Quian Quiroga et al., 2008) and non-human primates (Logothetis, 1998; Macknik and Livingstone, 1998). In general, the methodology undertaken for this line of research is the contrastive analysis, i.e., comparing the neural activity elicited by"seen"versus"unseen" stimuli (Baars, 1993). However, possible drawbacks associated to this empirical approach have been raised in the latest years (e.g., Overgaard, 2004; Aru et al., 2012a). Complementary to these observations, here we discuss a set of considerations to improve the search for the NCCs using the contrastive analysis.

#### **MANIPULATION OF THE PHYSICAL STIMULI**

The first step toward finding the NCCs is to select an experimental method (e.g., backward masking, CFS, etc.) to induce lack of awareness. A simple approach would be to implement this manipulation only for a set of "unseen" trials and contrast the results with a set of "seen" trials, where the manipulation is not used. For example, in the case of CFS, this would imply comparing the neural activity elicited during interocular suppression with a dioptic control in which flickering patterns are absent (Sterzer et al., 2009; Kang et al., 2011; Axelrod et al., 2014). However, the limitation of this approach is that the neural activity induced by the manipulation (e.g., flickering masks in the example of CFS) is absent in the "seen" condition. Therefore, the contrast between "seen" and "unseen" trials could be partially reflecting the processing of different physical stimuli.

An alternative is to use a milder version of the manipulation for the "seen" trials. This can be achieved, for example, by adding different amounts of noise to the stimulus (Jemel et al., 2003) or, in the case of CFS, by changing the contrast of the flickering masks (Kaunitz et al., 2011). But still, differences in low-level features such as luminance, contrast, or spatial frequency can largely modulate brain activity (Scholte et al., 2009). In fact, it has been argued that the ultra-fast detection of faces in natural scenes is partly explained by such features (Honey et al., 2008). One way to reduce these effects is by changing the perceptual ambiguity but controlling for a certain number of low-level variables (Portilla and Simoncelli, 2000; Willenbockel et al., 2010). This strategy ensures that this particular set of variables (e.g., contrast, luminance, and spatial frequency) do not explain differences in the neural activation between "seen" and "unseen" trials. However, the possibility of a hidden low-level variable explaining the differences observed in the neural activations cannot be ruled out.

In order to get rid of possible confounds introduced by the physical stimuli, several studies have proposed to compare different perceptual outputs using identical visual stimuli (Sergent et al., 2005; Quian Quiroga et al., 2008; Lamy et al., 2009; Aru et al., 2012b; Navajas et al., 2013). The underlying idea is to implement a manipulation that leads to ∼50% recognition performance, and then to contrast the activity elicited by these two sets of trials. The challenge, of course, is to find an experimental manipulation to be at the threshold of perception so that a certain stimulus is equally likely to be recognized or not. For example, Sergent et al. (2005) used the AB paradigm and compared "seen" and "unseen" trials with other set of trials in which the stimulus was absent. Several studies were successful in implementing this approach to uncover the neural basis of visibility (Marois et al., 2004; Sergent et al., 2005). But a challenge when using the AB paradigm is the extremely large variability across individuals (Martens et al., 2006; Willems et al., 2013). In general, due to interindividual differences (Kanai and Rees, 2011), previous works have proposed to adjust the stimuli ambiguity on a subject-by-subject basis (Fisch et al., 2009; Aru et al., 2012b; Navajas et al., 2013). For example, Navajas et al. (2013) used a modified backward-masking paradigm in which different degrees of zero-mean Gaussian noise were added to the stimuli. Critically, the variance of the noise was tuned across trials following a double-staircase procedure (Cornsweet, 1962; **Figure 1A**). In this way, "seen" and "unseen" trials were extracted from same noise levels, enabling a comparison across different perceptual states but keeping constant the physical stimulation.

Using the contrastive analysis with identical visual stimulation allows ruling out physical effects that can otherwise contaminate the comparison between conditions. Nonetheless, this approach does not exempt the contrastive analysis from other possible confounds (e.g., Aru et al., 2012a). For example, previous efforts have sought to dissociate the NCCs from the effects of attention (Tse et al., 2005; Bahrami et al., 2007), confidence (Sergent et al., 2005; Li et al., 2014), unconscious processing (Lamy et al., 2009; Salti et al., 2012), and introspection (Pitts et al., 2012; Fraessle et al., 2014). Whether the neural correlates of these processes can be entirely disentangled from the NCCs is still matter of extended debate (Lamme, 2003; Block, 2005; Dehaene et al., 2006; Koch and Tsuchiya, 2007; Kouider et al., 2010).

## **DIFFERENT CONSCIOUS PROCESSES IN THE BRAIN**

**DO DIFFERENT NCCS NECESSARILY CONTRADICT EACH OTHER?** In the last decades, vast empirical and theoretical efforts have been yielded to unravel the NCCs. However, to date, there is little agreement about the areas, timing, and mechanisms involved in eliciting a conscious percept. As an example, our own work has recently provided dissimilar evidence in terms of modulations by awareness occurring at different times and in different areas: (1) An evoked potential measured from the scalp in the OTC peaking at ∼170 ms that predicts conscious face perception (Navajas et al., 2013; **Figure 1B**); (2) Single-cell firing at ∼300–400 ms of highly selective neurons in the medial temporal lobe (MTL) appearing only upon conscious recognition (Quian Quiroga et al., 2008; **Figure 2**); and (3) A deflection in the local-field potential (LFP) preceding the firing of MTL neurons that is present only in recognized trials (Rey et al., 2014; **Figure 2**). To put together these results into a coherent framework, we propose to distinguish two different neural processes associated with conscious perception, namely, perceptual and full awareness.

#### **PERCEPTUAL AWARENESS IN OBJECT-SELECTIVE CORTICAL AREAS**

The search for the NCCs has posed the problem of finding which of the neural activations along the visual system correlate with conscious perception (Crick and Koch, 1990). In this line, evidence from psychophysical (He and MacLeod,2001), physiological (Gawne and Martin, 2000), and neuroimaging (Haynes and Rees, 2005) studies supports the notion that the primary visual cortex (V1) is not directly involved in eliciting conscious percepts (Crick and Koch, 1995; Rees et al., 2002). However, alternative views about the role of V1 in visual awareness were also proposed (Tong, 2003). In turn, object-selective responses in the inferotemporal cortex (ITC) have been consistently shown to be modulated by conscious perception (for a review see Logothetis, 1998). Similarly, an evoked potential in the 5 to 15 Hz frequency band at ∼170 ms was repeatedly shown to be modulated by conscious perception using the contrastive analysis with identical visual stimulation (Fisch et al.,2009; Navajas et al.,2013; Sandberg et al.,2013; **Figure 1B**). Furthermore, conscious reports about face perception can be decoded at the single-trial level based on the peak amplitude of these evoked responses (**Figure 1C**).

In principle, this set of results showing correlations between awareness and brain activity at 100–200 ms after stimulus onset could be in conflict with theoretical proposals arguing that the

NCCs are reflected by "late" (>300 ms) activations (Dehaene and Changeux, 2011). Alternatively, these neural modulations can be regarded as pre-requisites of consciousness (Aru et al., 2012a) occurring after stimulus onset (VanRullen, 2011), thus reflecting a preconscious state (Dehaene et al., 2006). We believe that this evidence supports the existence of a conscious process for visual recognition, namely perceptual awareness, which is linked to a boost in the activity of object-selective neural assemblies in highlevel visual areas. However, we do not claim that this is the only conscious process in the brain. Instead, we propose that the role of perceptual awareness is to rapidly recognize visual stimuli and to

feed this information to other neural circuits involved in different cognitive functions. Among the subset of processes, we will focus on one occurring in the MTL that is directly involved in memory processes.

#### **CONTEXTUAL AWARENESS IN THE MEDIAL TEMPORAL LOBE**

Patients with pharmacologically intractable epilepsy, implanted with intracranial electrodes for clinical reasons, provide the unique opportunity to record, with the proper setup, single-cell activity from the conscious human brain (for a review see Engel et al., 2005). With these recordings, it was found that neurons in the

MTL respond to different images in a remarkably selective and invariant manner. For instance, one neuron fired to seven different pictures of the actress Jennifer Aniston and not to other 80 pictures of other celebrities, animals and places (Quian Quiroga et al., 2005). Moreover, these responses could be triggered not only by pictures but also by the written name of the person and by the name pronounced by a synthetized voice (Quian Quiroga et al., 2009). Altogether, these results support the idea that the function of these neurons is to provide an explicit and abstract representation of the meaning of stimuli, thus being called *concept cells* (Quian Quiroga, 2012).

The latency of the firing of concept neurons is about 300 ms, although parahippocampal cells fire 50–100 ms before (Mormann et al., 2008). This timing is consistent with a set of "late" (>300 ms) activations that were shown to correlate with access to consciousness (Del Cul et al., 2007; Dehaene and Changeux, 2011). In this line, previous research has shown that neurons in the MTL modulate their firing activity with conscious perception (Kreiman et al., 2000, 2002; Reddy et al., 2006; Quian Quiroga et al., 2008). For example, when two incongruent pictures are presented to each eye, the firing of these neurons follows subjective perception (Kreiman et al., 2002). Similarly, in a CB paradigm, Reddy et al. (2006) showed that concept cells do not follow retinal input, as they were only active upon the perception of the changes. Using a backward-masking paradigm, previous research has also shown that concept cells fire only when the subject recognized the stimulus (Quian Quiroga et al., 2008). Remarkably, the responses appear in an all-or-none fashion, even if the visual stimuli were identical – i.e., the same picture at the same duration (Quian Quiroga et al., 2008).

These studies provide critical evidence for asserting that conscious perception is accompanied by these neural responses in the MTL. However, it was argued that this stage of processing might reflect the consequences of conscious recognition (NCCco), rather than recognition *per se* (Aru et al., 2012a; Quian Quiroga, 2012). This claim is supported by the fact that damage to MTL structures does not impair conscious perception (Kensinger

and Corkin, 2000; Postle, 2009). Likewise, here we propose that perceptual awareness precedes the firing of concept cells and is correlated with neuronal firing at 100–200 ms after stimulus onset, probably in the ITC (Logothetis, 1998). Indeed, we believe that a different and more sophisticated conscious process is triggered when this information is propagated to the MTL, activating these highly selective and sparsely firing neurons that represent the meaning of the stimulus for declarative, and particularly episodic, memory functions (Quian Quiroga, 2012). In particular, we support the idea that full awareness of the stimulus is elicited by this sensory-independent conceptual representation.

#### **FROM RECOGNITION TO CONTEXT: A LINKING MECHANISM?**

One of the most intriguing aspects of concept cells is the fact that their mean firing onset is too late (∼300 ms) to be explained by direct projections from high-level visual areas (ITC). In this line, it was argued that this delay might be crucial to enable the integration of information from different cortical areas, giving rise to a unified concept (Quian Quiroga, 2012). A recent study has shown a global LFP deflection in the theta-band (4–8 Hz) that precedes the response onset of concept cells (**Figure 2**) and is present only when the stimulus is consciously recognized (Rey et al., 2014). Moreover, the precise onset of concept cell responses is characterized by an increase in phase locking between the spikes and the LFPs in the theta band.

Even though the neural origin of this LFP deflection remains unclear, we believe that it is not originated from within the MTL. This is partially accounted by the fact that the human hippocampus is not thought to produce substantial contributions to the low-frequency LFP signals due to its structure (Buzsaki et al., 2012). More importantly, since the theta activation was seen globally in the MTL, if it were generated within the MTL, single cell activity responsible for this should have been observed prior to the change in the LFP. However, this situation was not observed (Rey et al., 2014). In turn, we hypothesize that this LFP response may reflect an activation generated by reverberating activity in the ITC crossing a certain threshold and triggering perception. This

LFP would provide a temporal window so that perceptual information can reach the MTL for further processes, such as memory functions.

## **CONCLUDING REMARKS**

This review discussed two methodological considerations in the study of the NCCs. In particular, we first argued for the implementation of paradigms where "seen" and "unseen" trials are obtained through the use of identical stimuli. Using contrastive analysis with identical visual stimulation allows ruling out physical effects that can otherwise contaminate the comparison between conditions. In the second part we discussed a selection of studies in which different NCCs were found at different timings and different brain areas (Quian Quiroga et al., 2008; Navajas et al., 2013; Rey et al., 2014). These seemingly contradictory results can be put together into a coherent framework by discriminating two different neural processes associated with conscious perception (i.e., perceptual and contextual awareness).

Interestingly, other distinctions between different conscious processes in the brain were previously proposed (Block, 1995; Kouider et al., 2010). For example, Block introduced the dichotomy between"phenomenal"and"access" consciousness (for a review, see Block, 2005), which is mainly centered on the question of whether we can have cognitive access to all our perceptual experiences, and thus whether we can see more than we can report (Block, 2011, 2012; Kouider et al., 2012). In this review, we proposed to discriminate between two different types of consciousness for already perceived stimuli (that can be reported) – i.e., processes beyond the distinction of phenomenal and access consciousness. In particular, we distinguish between a type of consciousness that relies on the firing of concept cells (*contextual awareness*) and the one that can be experienced even in the absence of MTL structures (*perceptual awareness*). The most remarkable examples of *perceptual* without *contextual* awareness are provided by patients with bilateral MTL resection or damage, such as patients H.M. (Scoville and Milner, 1957), R.B. (Zola-Morgan et al., 1986), and K.C. (Steinvorth et al., 2005). This condition led to a severe impairment in recollecting autobiographical events with no temporal gradient (Steinvorth et al., 2005), deficits in imagining new experiences (Hassabis et al., 2007), as well as in retaining and retrieving any type of episodic memory (Moscovitch et al., 2005). However, many other cognitive functions remained unaltered in these patients, such as the recognition of faces encoded before the surgery/accident (Kensinger and Corkin, 2000; O'Kane et al., 2004). Altogether, this evidence indicates that the MTL is not involved in the recognition of semantic entities, a process that we propose to be triggered by object-selective cortical areas (*perceptual awareness*). But bilateral damage or resection of MTL structures prevents subjects from having an enriched representation of the context, associations, and episodic memories linked to the specific stimulus, which we argue is elicited by the firing of concept cells (*contextual awareness*).

## **ACKNOWLEDGMENTS**

This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/H051651/1] and the Medical Research Council [grant number G1002100].

#### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 12 June 2014; accepted: 12 August 2014; published online: 29 August 2014. Citation: Navajas J, Rey HG and Quian Quiroga R (2014) Perceptual and contextual awareness: methodological considerations in the search for the neural correlates of consciousness. Front. Psychol. 5:959. doi: 10.3389/fpsyg.2014.00959*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Navajas, Rey and Quian Quiroga. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited andthatthe original publication inthis journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Investigating ongoing brain oscillations and their influence on conscious perception – network states and the window to consciousness

## *Philipp Ruhnau\*, Anne Hauswald and Nathan Weisz*

*Center for Mind/Brain Sciences, University of Trento, Trento, Italy*

#### *Edited by:*

*Jaan Aru, University of Tartu, Estonia*

#### *Reviewed by:*

*Rufin VanRullen, Centre de Recherche Cerveau et Cognition, France Niko Busch, Charité – Universitätsmedizin Berlin, Germany*

#### *\*Correspondence:*

*Philipp Ruhnau, Center for Mind/Brain Sciences, University of Trento, Via delle Regole 101, 38060 Mattarello, Trento, Italy e-mail: philipp.ruhnau@unitn.it*

In cognitive neuroscience, *prerequisites* of consciousness are of high interest. Within recent years it has become more commonly understood that ongoing brain activity, mainly measured with electrophysiology, can predict whether an upcoming stimulus is consciously perceived. One approach to investigate the relationship between ongoing brain activity and conscious perception is to conduct near-threshold (NT) experiments and focus on the pre-stimulus period. The current review will, in the first part, summarize main findings of pre-stimulus research from NT experiments, mainly focusing on the alpha band (8–14 Hz). It is probable that the most prominent finding is that local (mostly sensory) areas show enhanced excitatory states prior to detection of upcoming NT stimuli, as putatively reflected by decreased alpha band power. However, the view of a solely local excitability change seems to be too narrow. In a recent paper, using a somatosensory NT task,Weisz et al. (2014) replicated the common alpha finding and, furthermore, conceptually embedded this finding into a more global framework called "Windows to Consciousness" (Win2Con). In this review, we want to further elaborate on the crucial assumption of "open windows" to conscious perception, determined by pre-established pathways connecting sensory and higher order areas. Methodologically, connectivity and graph theoretical analyses are applied to source-imaging magnetoencephalographic data to uncover brain regions with strong network integration as well as their connection patterns. Sensory regions with stronger network integration will more likely distribute information when confronted with weak NT stimuli, favoring its subsequent conscious perception. First experimental evidence confirms our aforementioned "open window" hypothesis. We therefore emphasize that future research on *prerequisites* of consciousness needs to move on from investigating solely local excitability to a more global view of network connectivity.

**Keywords: alpha oscillations, consciousness, near-threshold perception, network connectivity,Win2Con**

### **INTRODUCTION**

Consciousness is one of the main enigmas of philosophy and has always been of great interest to experimental psychology. In recent years, understanding consciousness has also been a focus of cognitive neuroscience and can be described as one of the most important challenges in unraveling the neural basis of human cognition (Zeki, 2003; Dehaene et al., 2006; Lamme, 2006; Cohen and Dennett, 2011). Since advancing a mechanistic understanding of how phenomenological states emerge from neural activity patterns appears not currently feasible (hard problem, Chalmers, 1995), a popular strategy in cognitive neuroscience has been to first identify the so-called neural correlates of consciousness (NCC), defined as "the minimal set of neural events that give rise to a specific aspect of a conscious percept" (Crick and Koch, 1998).

Neural correlates of consciousness have been investigated with many different neuroimaging methods and electrophysiology. A strong focus initially lay on the differential response evoked by similar stimuli that were either reportable (i.e., consciously accessible) or not. To achieve a similar signal to noise ratio (i.e., equal amount perceived and non-perceived) these stimuli are oftentimes presented at the individual perception threshold. Therefore, a typically applied paradigm to investigate NCCs is the near-threshold (NT) paradigm. The design idea is simple; participants encounter a stimulus at a certain intensity such that it is equally likely to be perceived or not perceived, i.e., at perception threshold. The advantage of this design is that it is not the stimulus that changes from trial to trial but the quality of its perception (perceived; nonperceived). By investigating neural responses to stimuli that have been perceived and comparing them with those that have not been perceived, researchers have the unique opportunity to investigate neural processes based on (absent/present) consciousness.

A recent review has made a distinction between different NCCs (Aru et al., 2012). In order of appearance they are: Neural correlates of prerequisite processes (NCC-pr) – which are necessary for conscious access, yet they do not carry conscious content. Secondly, the NCC directly related to conscious experience – a neural correlate that, if present, would result in a specific conscious experience. Finally, consequences of conscious experience (NCC-co) represent after-effects of conscious experience (Aru et al., 2012). In typical event-related studies focusing on periods following the stimulus, a distinction between the "true" NCC and NCCco is challenging due to strong overlaps; furthermore, even in the period following the stimulus onset there may be an additional impact of NCC-pr (Aru et al., 2012). In the current review we will focus on the prerequisite of consciousness (NCC-pr). We will emphasize the importance of ongoing oscillatory brain activity and suggest a network perspective when investigating prerequisites of consciousness. The importance of distributed network activity in supporting conscious perception has already been proposed by others (e.g., in the global workspace theory, Dehaene et al., 1998; Baars, 2005; and similarly, for a unitary percept within the micro-consciousness framework, Zeki and Bartels, 1999), however, the idea was confined to processing and propagation of already presented stimuli and has not been extended to the period preceding upcoming inflow.

The claim that pre-activated networks (or network connections) are required for conscious perception of NT stimuli has been made in a recent work by Weisz et al. (2014) and first evidence for this hypothesis has been provided. The "Windows to Consciousness" (Win2Con) framework proposes pre-established pathways, which enable the propagation of stimulus representations from sensory areas to higher order cognitive areas (such as prefrontal cortex) and thereby predisposing a conscious representation. We will describe the study by Weisz et al. (2014) in more detail in later parts of this review (see Windows to Conscious Perception) and we will also focus on the implications of the results for neuroscientific research of consciousness (see also other articles in this research topic).

In the first part of this review we will, however, summarize main findings related to the pre-stimulus period, that is, the time window preceding a stimulus. A main focus will be on the alpha band (8–14 Hz), which in terms of significance to, and attention of the neuroscience community has experienced a roller-coaster ride. Being the oldest rhythm described in the history of human electrophysiology, it has long been labeled as the brain's "idling" response, since it is dominant during restful wakefulness and decreases during stimulation (Pfurtscheller and Silva, 1999). Current frameworks of alpha oscillations stress their functional significance as an inhibitory rhythm under top–down control (Klimesch et al., 2007). Alpha oscillations seem to serve as a mechanism to gate sensory information by inhibiting irrelevant or interfering regions and releasing inhibition from necessary areas (Jensen and Mazaheri, 2010). Furthermore, electrophysiological research found ongoing brain activity in the alpha range preceding upcoming stimuli that can predict whether the stimulus can be reported (i.e., consciously accessed) or not (e.g., Hanslmayr et al., 2007; van Dijk et al., 2008). These findings were remarkable and they have provoked quite a few following studies, summarized in the next paragraph.

## **PRE-STIMULUS OSCILLATORY ACTIVITY PREDICTS CONSCIOUS PERCEPTION**

There are a wide variety of studies investigating conscious access to low contrast stimuli. Since the late 1990s, many researchers have identified event-related components of conscious processing of similar stimuli (for a review see Dehaene et al., 2006). This approach, at least implicitly, holds that neural processes relevant to conscious processing are time-locked to the presentation of a stimulus. A few years ago, several studies challenged this notion finding that alpha band power preceding a low-contrast or NT stimulus was significantly lower when the stimulus would be reported as perceived as compared to when it was not perceived (Hanslmayr et al., 2007; Romei et al., 2008a; van Dijk et al., 2008). Importantly, alpha effects were largely constrained to posterior regions involved in visual processing, demonstrating that pre-stimulus effects can be regionally circumscribed and arguing against interpretations that overall arousal/wakefulness levels explain the differences. This general finding intrigued also, because it suggested that conscious perception can be determined by ongoing brain oscillations, which have conventionally been regarded as functionally irrelevant noise. The study and understanding of pre-stimulus oscillatory effects can now be considered as a central aspect in the investigation of NCC prerequisites.

A strong alpha rhythm in posterior regions predicts that an upcoming visual NT stimulus will not be perceived (Ergenoglu et al., 2004; Hanslmayr et al., 2007; Romei et al., 2008a; van Dijk et al., 2008; Busch et al., 2009; Wyart and Tallon-Baudry, 2009). Typically, these studies presented NT targets for a brief time (Busch et al., 2009; Busch and VanRullen, 2010; Lange et al., 2013) and masked them after offset to avoid sensory memory effects (van Dijk et al., 2008; Hanslmayr et al., 2009). Sustained power differences between subsequent perceived and non-perceived trials have been found in the second before the presentation of the NT stimulus. These pre-stimulus alpha decreases in perceived, as compared to non-perceived, trials showed a parietal to occipital topographical distribution, which was localized to visual cortex but also to parietal areas (Romei et al., 2008a,b; van Dijk et al., 2008; Lange et al., 2013). These localizations and the behavioral effects of alpha power increases (i.e., missing NT stimuli), as well as top–down modulated increases of alpha in task-irrelevant areas (Jensen et al., 2002; Thut et al., 2006; Jokisch and Jensen, 2007; Ciesielski et al., 2010), are supportive of the functional inhibition framework (Klimesch et al., 2007; Jensen and Mazaheri, 2010) wherein alpha oscillations are modulating the excitatory state of cortical regions. First invasive evidence for this hypothesis exists. A study on monkeys showed a correlation between alpha oscillations and neural firing, such that low alpha power goes together with an increase in neural firing (carrying the sensory representation, Haegens et al., 2011). Thus, alpha activity does directly affect excitability in local cortical areas. In the case of the visual NT design one might phrase as follows: a low contrast (NT) stimulus that enters the visual cortex during high alpha power does not (or only weakly) activate sensory neurons and – due to this failure – does not influence downstream areas essential for conscious perception.

Near-threshold experiments similar to the ones described above have also been done in the somatosensory modality. Alpha modulations in posterior regions and over somatosensory cortices have been found to modulate detection performance (Linkenkaer-Hansen et al., 2004; Sauseng et al., 2009; Schubert et al., 2009; Weisz et al., 2014). The relationship between alpha power (or in case of the somatosensory system, also called mu) and conscious perception, however, seems not to be as straightforward as in vision. Especially for primary somatosensory areas, alpha power and perception are more related in an inverted U-shape manner (lowest detection rates for low and high alpha power, highest detection rate with medium alpha power, Linkenkaer-Hansen et al., 2004). On the other hand, secondary somatosensory cortices as well as parietal areas show a very similar linear relationship as in the visual domain, increasing detection rates of NT stimuli with decreasing alpha power (Schubert et al., 2009; Weisz et al., 2014). Furthermore, evidence from research on a monkey showed a similar relationship between alpha in the somatosensory system and perception (Haegens et al., 2011). This study, however, applied a (difficult) tactile discrimination (perceptual decisionmaking) task. A recent study (Haegens et al., 2014) investigated thalamic and primary somatosensory cortex (S1) activity in an NT paradigm in two monkeys. This study showed a relation of alpha power in S1 to response bias, such that increased alpha power led to more overall reports of perceived stimuli. To conclude, even though not always completely consistent in terms of the pattern, alpha oscillations also seem to play a crucial role in the perception of tactile stimuli by modulating local excitability of somatosensory cortex.

Surprisingly, in the auditory domain a study well matched to the NT studies described above is still missing (for fMRI see Sadaghiani et al., 2009). This might be because measuring auditory alpha with non-invasive electroencephalography/magnetoencephalography (EEG/MEG) is challenging and therefore evidence on alpha in the auditory system is in general sparse (but see Frey et al., 2014; Strauß et al., 2014; Wilsch et al., 2014). However, several studies on auditory phantom percepts (Leske et al., 2013; Müller et al., 2013) – illusions without actual sensory stimulation, therefore a conceptual antagonist to NT stimulation – showed similar oscillatory modulations as in vision (for a recent comprehensive review on phantom percepts and oscillations see Lange et al., 2014). For instance, Leske et al. (2013) used the Zwicker tone, a brief auditory "afterimage" following the presentation of notch-filtered noise, to investigate the role of auditory alpha in illusory perception. Applying different types of experimental and analytic modulations they found that, generally speaking, lower oscillatory power in the alpha and beta range (10–20 Hz) coincided with a stronger perception of the illusion. Power modulations were most pronounced in Heschl's gyrus, incorporating primary auditory cortex. Similar findings were provided for an illusion of music within presented noise (Müller et al., 2013), and an imbalance of cortical alpha is correlated with a pathological illusion: in tinnitus patients alpha power in auditory cortex regions is lower than in controls (Weisz et al., 2005; Hartmann et al., 2014). Thus, alpha oscillations play a crucial role in the generation of auditory illusions, resembling a similar inhibitory function as in other modalities.

It is worth noting that modulations of alpha power are not merely correlational. Studies using transcranial magnetic stimulation (TMS) showed a direct relationship between the (induced) alpha power and behavior (Romei et al., 2010). They presented lateralized visual NT stimuli and, using TMS, rhythmically stimulated the contra- or ipsilateral occipital cortex at 10 Hz. Participants detected substantially more stimuli when their ipsilateral occipital cortex was stimulated as compared to when the stimulation was contralateral. Furthermore, Romei et al. (2010) could show that the effect was specific to stimulation frequencies in the alpha range (10 Hz) but not to other frequencies (5 and 20 Hz).

As mentioned above, many studies on neural oscillatory activity showed an inhibitory function of the alpha band (Foxe and Snyder, 2011; Haegens et al., 2011; Mathewson et al., 2011; Spaak et al., 2012). This inhibitory function is not restricted to NT paradigms; in fact it has been shown that inhibition caused by alpha oscillation can even increase with stimulus intensity (Chaumon and Busch, 2014). Furthermore, alpha activity seems to be under top–down control (Jensen et al., 2002; Thut et al., 2006; Jokisch and Jensen, 2007; Ciesielski et al., 2010), such that taskirrelevant regions are inhibited by an increase of alpha power. It seems plausible that the neural system is inhibiting undesired routes, thereby actively opening specific gates to guide information through a distributed network architecture (Jensen and Mazaheri, 2010). To perform optimally, our neural system needs a strong inhibitor, which alpha activity seems to provide.

Oscillations are not only characterized by amplitude but also by phase. The idea (and in fact the scientific proof) that the phase of brain activity in relevant areas is directly correlated to how well that area can process information inflow is not new (Lindsey, 1952). In recent years, however, there has been an enhanced focus on this aspect. For instance, there is some evidence that visual perception samples the information inflow at a sampling rate of around 10 Hz (VanRullen and Macdonald, 2012). Thus, it is not surprising that, besides oscillatory power, the phase (or the phase distribution) before an actual stimulus can also be predictive of its perceptibility (Busch et al., 2009; Mathewson et al., 2009; Thorne et al., 2011). Effects of phase are not limited to vision but have also been shown in the somatosensory (Ai and Ro, 2014) and the auditory modality (Henry and Obleser, 2012; Neuling et al., 2012). Recent approaches using entrainment in the alpha range go so far as to ascribe phase the dominant role, causally related to behavior (Spaak et al., 2014). Crucially, however, phase effects are more closely aligned to the upcoming stimulus, and are typically found around stimulus onset (Mathewson et al., 2009; Thorne et al., 2011) or slightly before (Busch et al., 2009). While power modulations are more representative of a sustained excitability change in a task relevant region, phase seems to reflect increased excitability at certain points in time. Thus, it is not surprising that phase modulations are observed for expectation (Stefanics et al., 2010) and attention (Lakatos et al., 2008) when the timing of upcoming stimuli is known. A recent paper by Zoefel and Heil (2013) reminds us, however, that among all the recent phase-craze some caution may be warranted, by showing that contaminations by the evoked response can cause phase effects akin to those reported in the literature.

In summary, evidence from NT paradigms in different modalities showed a prerequisite NCC reflected in a modulation of cortical alpha power in task relevant, mostly local sensory areas. Low alpha power seems to be related to high excitability in sensory areas. Thus, in the case of reduced alpha power, an upcoming low contrast (NT) stimulus leads to an ignition of relevant neural assemblies in sensory areas and thereby a perceptual report is more likely. Local excitability, however, is not the only prerequisite for conscious perception as the Win2Con framework predicts (see Windows to Conscious Perception).

## **ESSENTIAL NODES AND NETWORK PROPAGATION ARE NECESSARY FOR CONSCIOUSNESS**

There are many findings showing specialized areas in the human brain representing the conscious percept and not just mere sensory stimulus input. For instance, the fusiform face area (FFA) and the parahippocampal place area (PPA) alternate in activity depending on whether the conscious percept is a face (FFA active) or a house (PPA active) with the sensory stimulation being identical (Tong et al., 1998). These areas, necessary for a specific content of consciousness, are sometimes called essential nodes (Zeki and Bartels, 1999). Many studies based on the GlobalWorkspace theory (Baars, 1989) provided evidence that essential nodes are not responsible for a conscious percept by themselves, but that a propagation of the stimulus representation, mainly to a fronto-parietal network, is required for a stimulus to become conscious (Dehaene et al., 2001; Dehaene and Changeux, 2005; Nakamura et al., 2005).

For example, NT or masked visual stimuli that are not consciously accessible still elicit activations in respective lower level sensory brain regions (Dehaene et al., 2006). Yet, only stimuli with enough energy such that their representation is propagated up to higher-order areas lead to a re-activation of sensory areas (re-entrant/recurrent process) and"ignition"of the conscious process (Lamme and Roelfsema, 2000; Dehaene and Changeux, 2005; Dehaene et al., 2006). Furthermore, evidence from masking studies shows that the mask interrupts recurrent processes (Lamme et al., 2002). Several electrophysiological studies could demonstrate an enhanced functional connectivity for a consciously perceived stimulus as compared to the identical stimulus that went unnoticed (Palva et al., 2005; Melloni et al., 2007). Furthermore, multivariate decoding patterns applied to visual cortex can predict the presented stimulus, even in the absence of a conscious percept (Haynes et al., 2005). Thus, despite being an important component of the NCC, there is more to conscious perception than the activation of essential nodes. The model that initially proposed essential nodes, which are active for different aspects/features of the stimulus, is the model by Zeki and Bartels (1999) and Zeki (2001, 2003), which proposes micro-consciousness in each of the active nodes. All representations (nodes) exist independently. However, a unitary conscious percept can only exist when they are bound together (Bartels and Zeki, 2006). Thus, even though the basic representation of features is different, and no fronto-parietal areas are engaged *per se*, the crucial element that forms a unitary consciousness is again how individual parts of the percept are connected.

Most NCC research still follows the conventional approach in cognitive neuroscience, in studying local brain activity or functional connectivity states starting with the onset of the stimulus. This implicitly or explicitly degrades preceding and fluctuating ongoing brain activity as irrelevant noise. However, numerous works (see Pre-stimulus Oscillatory Activity Predicts Conscious Perception) assume and illustrate the high importance of ongoing brain activity for upcoming sensory stimulation. This suggests that presentation of sensory stimuli may fall into a period, described by local activation patterns and interregional functional connectivity patterns, which may be favorable or unfavorable for conscious perception (see also Windows to Conscious Perception). First evidence for the importance of network connections in ongoing activity comes from a recent study by Hipp et al. (2011). The study used an ambiguous stimulus, two bars that could either be perceived as bouncing off or passing each other, and investigated networks associated with the different percepts. They found that pre-stimulus activity in a fronto-parietal beta network was predictive of a bouncing or passing percept. This indicates that the content of consciousness is predisposed by ongoing activity in a large neuronal network.

In summary, whether consciousness is represented by activity in a global workspace or by the integration of information of multiple essential nodes with micro-consciousness, the common factor is the necessary interconnection, without which conscious access seems impossible. In the following chapter we will argue that this is similarly true for prerequisites of consciousness.

### **WINDOWS TO CONSCIOUS PERCEPTION**

The framework proposed by Weisz et al. (2014) added a new angle to the discussion of determinants of consciousness, or NCC-pr to stay within the terminology of Aru et al. (2012). The rationale of the framework is motivated by what is seemingly a paradox: if local excitability is not a sufficient explanation for conscious perception of a stimulus (e.g., Dehaene et al., 2006) in a post-stimulus period then why should local excitability be the crucial pre-stimulus ingredient of upcoming conscious perception? Indeed, as summarized convincingly by Dehaene et al. (2006), temporally early sensory activity, i.e., not caused by reentrant processes, does not differentiate between consciously perceived and non-perceived (identical) stimuli in post-stimulus intervals. This however would be expected if the main feature leading to subsequent perception of NT stimuli is the excitability level in sensory areas (e.g., neurons closer to firing threshold inV1 discharging following weak afferent input in the case of perceived stimuli).

Weisz et al. (2014) argue that in addition (or potentially even as a substitute) to local excitability, specific functional connections between sensory and higher order areas are predicting conscious perception (**Figure 1A**). It is therefore a matter of pre-established pathways forming windows to consciousness (thus the frameworks name: Win2Con) that allow a stimulus to be forwarded and consequently become conscious. Due to the abundance of pre-stimulus alpha effects in the literature, Win2Con builds upon the "gatingby-inhibition" hypothesis (Jensen and Mazaheri, 2010), which proposes that alpha activity can functionally inhibit (and release) nodes along the sensory pathway and thereby govern which content reaches consciousness and which is suppressed. It seems likely that alpha band activity plays a role in pre-established networks, although our framework makes no specific predictions about the frequency band involved. Crucially, the Win2Con framework predicts that the functional pathways activated in this gating process are already pre-activated, thereby constituting open windows to

connections to higher order areas (e.g., prefrontal, parietal) are essential. **(B)** Local excitability changes in the somatosensory cortex (SII) as indexed by alpha decreases prior to perceived as compared to not perceived trials. **(C)** Additionally, local changes of connectivity (efficiency) have been observed increased with specific regions: mainly the left anterior temporal/inferior frontal areas, and other sensorimotor regions (e.g., premotor cortex, inferior parietal lobe). The power and connectivity results presented in **(B–D)** are from Figures 2–4 in Weisz et al. (2014).

consciousness. There is some evidence that this is in fact the case.

In their study, Weisz et al. (2014) used an NT paradigm to investigate pre-stimulus activity in the somatosensory domain (see Wühle et al., 2010 for the original study investigating NCC). Two tactile stimuli were presented successively, one of them always supra-threshold the other one at perception threshold. Participants reported whether they perceived one or two stimuli. Contrasting pre-stimulus power for perceived and not perceived trials of the NT stimulus, the authors reported enhanced alpha in secondary somatosensory cortex [as well as middle frontal gyrus (MFG)] for subsequently not perceived NT stimuli (**Figure 1B**), thereby replicating the finding of local excitability changes in predicting the NT stimulus perception. More relevant within the proposed framework, they also analyzed network properties (global and local graph theoretical measures, see **Box 1**) between perceived and not perceived NT stimuli during the pre-stimulus period. On a global level, enhanced small-worldedness occurred around 17 Hz prior to not perceived NT stimuli compared to perceived ones. As small-worldedness is calculated as the ratio between clustering of a network and path length, both parameters can potentially influence the metric (more details on the relationship between different network properties can be found in **Box 1**). The authors investigated the nature of this effect by calculating global clustering and path length (as well as global

efficiency) for the relevant frequency band and found that the enhanced small-worldedness prior to not perceived NT stimuli was due to a globally more clustered network compared to perceived ones. As global efficiency suggested shorter communication pathways prior to perceived stimuli, the authors concluded that networks preceding not perceived NT stimuli show more characteristics of local integration while those preceding perceived NT stimuli are more defined by global integration properties. These global differences underlying perceived and not perceived NT trials were further investigated using more fine-grained analyses on the frequency identified by the global analyses (17 Hz), namely local graph theoretical measures, which provide the location of specific network properties instead of a description applying to the entire network (**Figure 1C**). Confirming the global effects, increased local clustering and efficiency as well as reduced distance were observed for the secondary somatosensory cortex prior to perceived NT stimuli, presumably a key region for conscious somatosensory perception. Somatosensory cortex not only appears to be generally better integrated prior to perceived NT stimuli, but the communication flow is increased with specific regions (**Figure 1D**): mainly the left anterior temporal/inferior frontal areas and other sensorimotor regions (e.g., premotor cortex, inferior parietal lobe, etc.). Prior to not perceived NT stimuli, only the posterior cingulate gyrus showed enhanced connectivity.

#### **BOX 1 | Graph theory for electrophysiology: a very brief primer.**

Graph theory investigates different network properties of nodes (e.g., neurons, brain regions, or websites) and edges (e.g., axons, structural/functional connections, or hyperlinks). As the brain is a complex network of interconnected neurons, graph theoretical measures can also be applied to understand the dynamics of this complex system better (for reviews see Bullmore and Sporns, 2009; He and Evans, 2010; Sporns, 2012). Within the brain networks, the nature of nodes (i.e., voxels) and edges (i.e., connections) depend on different variables as measures and features of connectivity, brain mapping methods, and anatomical parcellation schemes (Rubinov and Sporns, 2010), all of which can influence the interpretation of the data.

With electrophysiology the following steps need to be taken to investigate frequency specific source space data with graph theory (see Bullmore and Sporns, 2009; and for details Weisz et al., 2014): First, after the projection of sensor space activity to source space (Schoffelen and Gross, 2009) a measure of association between individual nodes needs to be estimated (in Weisz et al., 2014, the imaginary part of coherence; for other approaches see Hipp et al., 2012), resulting in a node by node association matrix. Secondly, a binary adjacency matrix is generated by applying a threshold to the association matrix (for how to choose the threshold see van Wijk et al., 2010). The adjacency matrix (the graphical model of the brain network) can then be used to calculate network parameters of interest. Within the context of Win2Con the most interesting ones would be, for example:


The Brain Connectivity Toolbox (Rubinov and Sporns, 2010; https://sites.google.com/site/bctnet) offers a wide variety of functions in Matlab and C that calculate most of these measures.

The reported study by Weisz et al. (2014) provides first evidence for the Win2Con framework. Different sensory areas (here somatosensory regions) are differentially integrated within a network prior to the relevant stimulus, rendering its perception more or less likely. As proposed in the framework, local alpha modifications predicting the perception of an upcoming event (as suggested in the functional inhibition hypothesis) appear to depend on the level of integration between the region showing alpha modulations and the relevant network. In this sense, local excitability is necessary but not sufficient for conscious perception. Pre-established pathways within the relevant network are another prerequisite for NT stimuli to become consciously accessible. This is supported by findings of double dissociation of alpha modulations and graph theoretical measures: for example, while alpha power showed strong effects in the right MFG, effects of the graph theoretical measures were absent. *Vice versa*, significant changes of graph theoretical measures occurred in left anterior temporal/inferior regions without concomitant effects in alpha power. Accordingly, alpha power should not be mistakenly regarded as a proxy of functional connectivity.

Speaking of pre-established connections might trigger associations with other frequently investigated and reported networks such as, for example, the default mode network (Raichle et al., 2001). Findings from the study by Weisz et al. (2014) might nourish speculations that the default mode network constitutes such a network of pre-established pathways that modify the likeliness of subsequent conscious perception. In the current case of somatosensory NT stimulation, the posterior cingulate cortex, which supposedly is a core region of the default mode network, showed reduced connectedness with the somatosensory cortex prior to not perceived stimuli. Analogously, we want to point out that the idea of pre-established pathways relies on the assumption that, although globally increased connectivity might be a hint to an open window, it is not a general increase in connectedness of sensory regions that renders perception of an upcoming stimulus more likely, but connections with relevant areas, and integration of frontal and parietal regions that are crucially involved in the so-called global workspace (Dehaene et al., 1998).

#### **CONCLUSION AND FUTURE RESEARCH QUESTIONS**

In the current review we gave an overview of previous work regarding factors determining conscious perception with a focus on oscillatory alpha modulations prior to NT stimulation. In summary, lower alpha power in task relevant areas appears to be

### **FIGURE 2 | Schematic representation of the predictions of the functional inhibition theory and the windows to consciousness**

**framework.** The functional cortex images on top of each line plot show the hypothesized contrast of perceived vs. not perceived stimuli in terms of functional activity. **(A)** Functional inhibition hypothesis. Cortical excitability is modulated in sensory regions before NT stimulus presentation. Low alpha goes hand in hand with increased excitability and thus an NT stimulus entering sensory areas yields a stronger neural signal. The signal propagates to higher order areas, which send feed-back signals to sensory areas. The second peak in the sensory region reflects activity due to re-entrant activation. **(B)** Windows to consciousness framework. Under similar excitation levels, modulations of connectivity are expected. In case of pre-established pathways (high connectivity, dark green) early sensory activation, expected to be similar for perceived and not perceived trials, is propagated to high-level areas. This leads to re-entrant activation of sensory areas. Crucially, only late activity differences in sensory areas are expected.

predictive of conscious perception of upcoming NT stimulation. However, it is worth mentioning that there is sparse evidence that the local excitability changes are bound to sensory regions as only a few studies have estimated the sources of alpha modulations (van Dijk et al., 2008; Lange et al., 2013; Weisz et al., 2014). Also, these local excitability alterations may not always be predictive of NT stimulus perception, at least in a linear manner (Linkenkaer-Hansen et al., 2004). Importantly, we claim that there exists an explanatory gap between the standard interpretation of pre-stimulus alpha effects that emphasize the role of local excitability, and the very sparse evidence showing an implication of respective regions at early latencies (see **Figure 2** for a schematic presentation of the predictions of the functional inhibition and the Win2Con framework). Consequently, we argued that local alpha power is indicative but not sufficient for conscious perception (at least for stimuli of weak intensity). As was proposed in a recent study (Weisz et al., 2014) introducing the Win2Con framework, a prerequisite for conscious perceptions is the integration of relevant areas in a broad network. Thus, only when local sensory excitability meets pre-established pathways, i.e., integration of the relevant area, will an NT stimulus be perceived.

There are still a few open questions that it is important to address: Firstly, as mentioned above, the actual location of the frequently reported alpha modulations prior to stimulus perception is often not known. Sources within sensory areas would have different theoretical implications than downstream sources: for example, the distinction between visual cortex and parietal regions is not always evident from sensor topographies alone. While the first would underline the importance of the current state of a specific sensory system, the latter may point toward a generalized increased attentive state. Therefore, supportive evidence should come from source localization efforts and/or intracranial recordings in animals or, for example, epileptic patients. Secondly, while building upon the alpha literature makes sense in light of the current evidence, the question remains whether power modulations in sensory areas are indeed necessary or whether the connectivity states alone are sufficient to predict conscious perception. Thirdly, and somewhat related, the exact relation between power and connectivity or local and inter-areal synchronization needs to be better understood. Fourthly, are pre-stimulus networks and post-stimulus networks comparable and, if yes, how do they relate? Fifthly, what kind of impact does pre-stimulus activity have on stimulus processing as indexed by evoked/induced brain activity? This can be investigated, for example, by using correlational approaches. Sixth, networks within the alpha–beta frequency range are not only theoretically good candidates but, as elaborated above, there is evidence for their involvement in conscious perception. Nevertheless, the question about other frequencies ranges and their interplay remains open. Finally, how causal are pre-stimulus connectivity patterns? This could be investigated using real-time experiments in which particular states (as previously identified in offline studies) can be detected and stimuli presented at times of a specifically beneficial or detrimental state.

While the current review and introduction to our framework was geared toward the case of NT stimuli to illustrate our point, in real life many stimuli are clearly supra-threshold. In this case it is likely that sensory areas will be sufficiently activated to drive downstream areas, rendering the stimulus to be consciously perceived. However, it is also likely that trial-by-trial coupling fluctuations between sensory and downstream areas will influence post-stimulus activity patterns and thereby phenomenological experience. Apart from clever paradigms that allow for graded

quantification of subjective experiences related to a stimulation, analysis methods would need to sufficiently develop to allow for the assessment of single-trial connectivity using MEG/EEG.

#### **ACKNOWLEDGMENTS**

Philipp Ruhnau and Nathan Weisz were funded by the European Research Council (ERC StG 283404 – WIN2CON). We would like to thank Julia N. Frey for her helpful input on a previous version of the manuscript and Hayley Prins for proof reading the manuscript. Also, we would like to acknowledge the two reviewers for their very constructive comments.

#### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 03 July 2014; accepted: 09 October 2014; published online: 30 October 2014. Citation: Ruhnau P, Hauswald A and Weisz N (2014) Investigating ongoing brain oscillations andtheir influence on conscious perception – network states andthe window to consciousness. Front. Psychol. 5:1230. doi: 10.3389/fpsyg.2014.01230*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Ruhnau, Hauswald and Weisz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited andthatthe original publication inthis journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Intracranial spectral amplitude dynamics of perceptual suppression in fronto-insular, occipito-temporal, and primary visual cortex

## *Juan R. Vidal 1,2,3,4 \*, Marcela Perrone-Bertolotti 3,4 , Philippe Kahane5 and Jean-Philippe Lachaux1,2*

*<sup>1</sup> INSERM U1028, CNRS UMR5292, Brain Dynamics and Cognition Team, Lyon Neuroscience Research Center, Lyon, France*

*<sup>2</sup> Université Claude Bernard, Lyon, France*

*<sup>3</sup> Laboratoire de Psychologie et NeuroCognition, University Grenoble Alpes, F-38040 Grenoble, France*

*<sup>4</sup> CNRS, Laboratoire de Psychologie et NeuroCognition, UMR 5105, F-38040 Grenoble, France*

*<sup>5</sup> CHU Grenoble and Department of Neurology, INSERM U704, F-38043 Grenoble, France*

#### *Edited by:*

*Jaan Aru, University of Tartu, Estonia*

#### *Reviewed by:*

*Melanie Wilke, California Institute of Technology, USA Theofanis Panagiotaropoulos, Max Planck Institute for Biological Cybernetics, Germany*

#### *\*Correspondence and present address:*

*Juan R. Vidal, CNRS, Laboratory of Psychology and NeuroCognition, UMR 5105, University Pierre Mendès-France University of Grenoble-Alpes, Bâtiment Sciences de l'Homme et Mathématiques BP 47, 38040 Grenoble Cedex 9, France e-mail: juan.vidal@upmf-grenoble.fr*

If conscious perception requires global information integration across active distant brain networks, how does the loss of conscious perception affect neural processing in these distant networks? Pioneering studies on perceptual suppression (PS) described specific local neural network responses in primary visual cortex, thalamus and lateral prefrontal cortex of the macaque brain. Yet the neural effects of PS have rarely been studied with intracerebral recordings outside these cortices and simultaneously across distant brain areas. Here, we combined (1) a novel experimental paradigm in which we produced a similar perceptual disappearance and also re-appearance by using visual adaptation with transient contrast changes, with (2) electrophysiological observations from human intracranial electrodes sampling wide brain areas. We focused on broadband high-frequency (50–150 Hz, i.e., gamma) and low-frequency (8–24 Hz) neural activity amplitude modulations related to target visibility and invisibility. We report that low-frequency amplitude modulations reflected stimulus visibility in a larger ensemble of recording sites as compared to broadband gamma responses, across distinct brain regions including occipital, temporal and frontal cortices. Moreover, the dynamics of the broadband gamma response distinguished stimulus visibility from stimulus invisibility earlier in anterior insula and inferior frontal gyrus than in temporal regions, suggesting a possible role of fronto-insular cortices in top–down processing for conscious perception. Finally, we report that in primary visual cortex only low-frequency amplitude modulations correlated directly with perceptual status. Interestingly, in this sensory area broadband gamma was not modulated during PS but became positively modulated after 300 ms when stimuli were rendered visible again, suggesting that local networks could be ignited by top–down influences during conscious perception.

**Keywords: perceptual suppression, broadband gamma, conscious perception, anterior insula, primary visual cortex, visibility, intracranial EEG, contrast adaptation**

## **INTRODUCTION**

The flow of our conscious perceptual experiences evidently relate to the ongoing changes in our sensory inputs. In addition to establishing the limits between early sensory neural processing and subsequent conscious-related processing (Dehaene and Changeux, 2011) it has become a critical issue to scrutinize the conditions used in contrastive analyzes which typically compare brain activity for visible and invisible stimuli (Aru et al., 2012; de Graaf et al., 2012). For example, it might be relevant to *a priori* dissociate between perceptual invisibility that is caused by disabling sensory stimuli from reaching consciousness, which occurs when efficiently masking stimuli, from perceptual invisibility that is produced by suppressing a previously visible stimulus. Both invisibilities probably do not involve the same underlying neural mechanisms. In this study we focus on the latter which is called perceptual suppression (PS).

By definition, visual PS consists in making a visible object invisible despite ongoing retinal input. This has been achieved through flash suppression (Wolfe, 1984; Wilke et al., 2003, 2006; Tsuchiya and Koch, 2005). In binocular vision PS can also be induced by background motion, as in motion-induced blindness (MIB; Bonneh et al., 2001, 2014), by transient contrast decreases (May et al., 2003; Simons et al., 2006), or by presenting small visual transients throughout the visual field after short fixation (Kanai and Kamitani, 2003). Sudden contrast reductions have a strong impact so as to make complex visual scenes vanish from conscious perception for a prolonged period (Simons et al., 2006). The latter type of PS has been described and characterized at a psychophysical level, yet remains largely unexplored at the neural level, but could reveal important insights into the neural basis of conscious visual perception.

While previous studies explored the modulation of cortical spectral responses during PS (Wilke et al., 2006; Keliris et al., 2010; Panagiotaropoulos et al., 2012), we ignore if these responses are manipulation-specific, i.e., whether they depend on the type of

experimental manipulation that induces PS, or whether they generalize to other types of manipulations inducing suppressions as well, such as those induced by contrast decreases, thereby reflecting general mechanisms underlying states of stimulus visibility and invisibility. To explore a novel experimental approach to conscious perception and to study neural activity modulation by PS we characterized the neural correlates of PS induced by stimulus contrast decrease (Simons et al., 2006), for the first time within human intracranial electroencephalographic (iEEG) recordings.

In our task participants passively fixated a typical Troxler ringshaped stimulus while it underwent successive contrast increases and decreases, yet always remained physically on-screen. Within our experimental design, the same contrast level can be associated with either a visible or an invisible target. A behavioral account in our study showed that this effect elicited with specific parameters is perceptually robust and highly reproducible from trial-to-trial. We therefore implemented an adapted version of it for the subsequent intracranial recordings. In our analyzes we first focused on the global significant emergence of broadband gamma (50– 150 Hz) and low-frequency alpha/beta (8–24 Hz) responses as compared to a baseline period. We estimated the modulation of these signals in sensory, limbic, and frontal cortices. This first analysis revealed which type of signal amplitude modulations, increases or decreases, are elicited by contrast changes (increases or decreases), respectively, linked to stimulus visibility (VIS) and invisibility (INV). However, this analysis does not directly link local network activity with stimulus visibility status processing, due to a possible confound with the sign of contrast change. Therefore, in a second analysis we pinpointed perceptual processing by contrasting the two previous conditions with a third condition, the stimulus offset (OFF), which also elicits stimulus disappearance but without stimulus sensory processing (see Materials and Methods). Summarizing, our paradigm uses perceptual invisibility during contrast decrease (INV), physical invisibility by removal of physical stimulus (OFF) and physical visibility by contrast increase (VIS).

Here we report that PS induced by contrast reduction elicits a decrease in low-frequency activity in distant cortical networks. Conversely, broadband gamma responses, especially in lateral occipital-temporal cortices, are positively modulated for this

same physical manipulation which is associated with stimulus invisibility in our experimental design. Interestingly, when contrasting this condition with the two other conditions, respectively, stimulus visibility (by contrast increase) and stimulus invisibility (by stimulus offset), we found that in occipito-temporal cortices most effects were perceptual and were found in low-frequency amplitude modulations. In this same region, gamma-band activity modulation was divided between perceptual and sensory effects. Surprisingly, the temporal dynamics of the gammaband amplitude modulation found in primary visual cortex (PVC,=V1), revealed unexpected patterns. There was no gammaband response to stimulus contrast decrease (INV) and a slowly negative progressing gamma-band response to stimulus contrast increase (VIS). Interestingly, after 300 ms VIS responses displayed a positive and transient amplitude rebound, which could be signaling top–down feedback processing associated with the status transition from invisibility to visibility. Finally, we also contribute evidence on a role of frontal cortex in conscious perception by reporting that perceptual effects in the gamma-band appear earlier in fronto-insular networks than in occipito-temporal networks, conveying evidence supporting top–down signals in conscious perception that may arise from these networks. Globally, the results presented in this study shed light on a rich panel of underlying specific processes at work during PS with new insights for the neural basis and neural dynamics of conscious perception.

## **MATERIALS AND METHODS STIMULUS AND TASK** *Behavioral task*

The visual stimulus consisted of a centered dark gray fuzzy circle (Troxler stimulus, inner diameter: 10 cm; width 1 cm) on a gray background (**Figure 1A**). The image's original contrast level was changed for three different stimuli with respective contrast decreases of 10, 50, and 100% (stimulus removal). A small black fixation cross was positioned at the center of the image and stimuli were presented on a screen positioned at ∼1 m from the participants. The behavioral task consisted in the following sequence of events (**Figure 1A**): after a 1–2 s of fixation the stimulus at 100% of contrast was presented for 2 s, followed by the presentation for 1.2 s of one of the three other stimuli with reduced. Thereafter,

a white background screen was presented indicating three alternative response choices indicating full, partial, or null stimulus disappearance. Participants could respond during the 3 s duration of this period. The participants task was to rate on each trial his/her perceptual fading experience of the stimulus after the contrast change. Importantly, participants had the third option to indicate "partial disappearance" only when the stimulus did not fully disappear and remained partially visible. The disappearance probability of each stimulus was estimated by the ratio of the number of full disappearance reports divided by the total number of responses. As shown in **Figure 1B**, at 50% contrast decrease, the disappearance probability of the stimulus was near 95%, reflecting the strength of the PS effect. Similar effects have been shown in previous studies (May et al., 2003). The behavioral task was carried out on a laptop computer (screen: 60 Hz refresh rate).

#### *Intracranial recording task*

The same basic visual stimuli from the behavioral task was used in the experiment for the intracranial recordings. In this task, the stimulus contrast level indicated at 100%, which was equal to the 75% contrast level used in the behavioral task, was relatively reduced and increased to two different contrast levels: 50 and 150%. A small black fixation cross was positioned at the center of the circle and stimuli were presented on a screen positioned at ∼1 m from the participants. When increasing the contrast to its original level or higher the faded ring reappeared immediately. We used this PS effect in an original experimental design with two conditions, described in the following section.

A trial consisted in four successive contrast changes of the same ring-shaped stimulus (**Figures 2B,C**). Each trial started with the presentation of a central fixation cross for a duration of 1.5 s, followed by the onset of the stimulus at 100% contrast for 2.2 s. In the first condition termed INV the contrast changes occurred in the following order and lasted 1.1 s: 150, 100, and 150%. The latency of stimulus onset or stimulus change that includes a temporal jitter of 0–100 ms is preceded by the ∼ symbol. The contrast variations produced stimulus invisibility in the third step when the stimulus contrast was decreased 50% (**Figure 2B**). In the second condition termed VIS the contrast changes occurred in the following order: 50, 100, and 150%. In this series the stimulus became invisible in the second step and was visible again in the third step when the contrast was increased 50%. Importantly, through these different contrast manipulations we could achieve for the same stimulus physical contrast level (100%) two different percepts, either an invisible or a visible stimulus, as illustrated by the pink box relating panels B and C in **Figure 2**. After the fourth step which increased the stimulus contrast from 100 to 150%, the stimulus was set off during the presentation of a white screen (offset, OFF). In 50% of trials the stimulus decreased 50% contrast for a brief 0.2 s before full stimulus offset. We used the other 50% of all trials' offset (both conditions), who did not include this brief transient, in subsequent analyzes detailed in the data analysis section. Participants were instructed to fixate the central cross during all contrast changes and detect this very short stimulus disappearance before stimulus offset (white screen). This detection task was not related to prior perceived contrast changes and its only purpose was to keep

participants focused throughout trials. This experimental design has the advantage to reduce neural processing related to the preparation of subjective report in the period of interest which involves mainly passive fixation of successive stimulus contrast and visibility changes (pink box, **Figures 2B,C**). After a short task training period which concerned the maintenance of gaze fixation throughout contrast changes participants reported a vivid experience that the stimulus disappeared (PS) and subsequently reappeared after contrast increase (PSR; **Figures 2B,C**). There were between 60 and 120 trials per condition, depending on the number of blocks (=30 trials) completed by each participant. The intracranial task was carried out on a desktop computer (screen: 60 Hz refresh rate).

#### *Participants and intracranial recordings*

The behavioral task included 10 healthy participants (five males, five females, mean age 32.8 ± 2.0 SEM), all had normal or corrected-to-normal vision and provided written informed consent. Intracranial recordings were obtained from 9 neurosurgical patients (five females, four males, mean age: 28.7 ± 3.3 SEM)

with intractable epilepsy at the Epilepsy Department of the Grenoble University Hospital and at the Epilepsy Department of the Neurological Hospital in Lyon. All participants had normal or corrected-to-normal vision, gave written informed consent, and the experimental procedures were approved by the Institutional Review Board and by the National French Science Ethical Committee. Electrode implantation was performed according to regular procedures and all target structures for the presurgical evaluation were selected strictly according to clinical considerations with no reference to the current study. Eleven to fifteen semirigid, multi-lead electrodes were stereotactically implanted in each patient, with 10–15 recording sites on each electrode (2 mm wide, 3.5 mm center-to-center; Jerbi et al., 2009b; Lachaux et al., 2012). Intracranial EEG recordings (iEEG) were conducted using a video-iEEG monitoring system (Micromed), which allowed the simultaneous data recording from 128 depth-EEG electrode sites. All electrode contacts were identified on a postimplantation MRI showing the electrodes, and coregistered on a pre-implantation MRI. MNI and Talairach coordinates were computed using the SPM (http://www.fil.ion.ucl.ac.uk/spm/) toolbox.

#### *Data analysis*

Data were sampled at 512 Hz and each recording site was referenced to its adjacent neighbor, (bipolar montage). In this study we extracted spectral responses from neural signals using two spectral analysis techniques: wavelets and Hilbert transform. Based on prior knowledge from previous studies involving visual induced spectral responses in intracranial recordings (Vidal et al., 2010, 2011; Dalal et al., 2011; Hamame et al., 2014; Perrone-Bertolotti et al., 2014) we targeted two frequency bands of interest: the broad gamma band (50–150 Hz) and the low-frequency band (8–24 Hz). As proposed elsewhere (Lachaux et al., 2012), the broad gamma band is currently considered as a functional correlate of local cortical activation while the low frequency bands might reflect long-range interactions involved in top–down processing (Siegel et al., 2012). The amplitude responses within these bands were extracted with a Hilbert transform that is explained further in the text. However, in order to verify that such prior based frequency band selection indeed contained effective signal modulation we first computed a standard time-frequency (TF) wavelet decomposition within the 2–150 Hz band. To estimate the significance of these post-stimulus responses within this frequency range, as compared to baseline ([–200 to – 50]ms) power in a total time window from −200 to 800 ms post-stimulus, we used a paired *t*-test across trials at each recording sites. Illustrative examples of this analysis are shown in **Figure 3**.

To increase signal-to-noise ratio of spectral neural signals in the low and high frequency bands we thus opted to compute the amplitude envelope in the averaged high and low frequency intervals, respectively. We therefore computed the Hilbert transform to evaluate the significance of these neural responses with a

response emergence in PS, PSR and the comparison between the two conditions in **(A)** extra-striate visual area (MNI coordinate: [−39 −75 2]), **(B)** anterior insular cortex (MNI coordinate: [−37 18 9]) and **(C)** supramarginal gyrus (MNI coordinates: [58−21 18]). Broadband gamma power increase and low-frequency power decrease, as compared to pre-stimulus baseline, were

opposite effect pattern was observed for PSR. All TF T-maps are scaled between −8 and 8. **(D)** To study spectral emergence, i.e., the orientation of the different spectral responses as compared to the average baseline level, we averaged the amplitude in the late part of the neural response, bypassing the initial transient response induced by the changing sensory input.

lower degree of complexity across all electrodes. To obtain highfrequency activity amplitudes, for example between 50 and 150 Hz, we applied the following processing steps. First we bandpass filtered iEEG signals in multiple successive 10 Hz-wide frequency bands (e.g., 8–10 bands as [50–60 Hz], [60–70 Hz], etc.) using a zero phase shift non causal finite impulse filter with 0.5 Hz roll-off. For the low-frequency interval between 8 and 24 Hz we used 4 Hz-wide bands. Next, for each bandpass filtered signal we computed the envelope using a standard Hilbert Transform. For each frequency bin (ex: [60–70 Hz]) the time-varying amplitude was divided by its mean across the entire recording period of the experiment and multiplied by 100. This yields instantaneous amplitude envelope values as percentage-of-the-mean. Finally, the envelope signals computed for each consecutive band were averaged to provide one single time series across the entire recording session. The obtained envelopes had a sampling rate of 64 Hz. This procedure has been previously used in various studies in our group (Jerbi et al., 2010; Juphard et al., 2011; Ossandon et al., 2012; Vidal et al., 2012). To estimate the post-stimulus neural activity modulations vs. baseline (neural emergence), excluding the initial activity transient produced by neural sensory changes, we estimated the difference between the average amplitude in the 300–800 ms post-stimulus time interval with the averaged prestimulus (from −200 to −50 ms) baseline amplitude response by the means of a paired *t*-test, across all electrodes and corrected for multiple comparisons across this dimension with a false discovery rate test (FDR; Genovese et al., 2002). To estimate the difference in neural response amplitude between experimental conditions (visible, invisible and offset) averaged across time, we used a repeated measures analysis of variance (ANOVA) combined with a *post hoc* Tukey–Kramer test. All statistical *p*-values were FDR-corrected (False Discovery Rate, Genovese et al., 2002) for multiple comparisons across the number of electrodes. When these analyzes were performed on time resolved data, we equally corrected for multiple comparisons across time samples per ROI. We additionally considered effects significant and of interest when repeated on at least five consecutive time samples. All analyzes were performed using Matlab (The Mathworks, Inc., Natick, MA, USA).

In this study we focused mainly on two frequency ranges. We analyzed a low and high-frequency interval according to preliminary TF analysis (see Results section). The low-frequency interval ranged from 8 to 24 Hz including mainly alpha and beta band frequency response intervals. The high gamma band interval ranged from 50 to 150 Hz. This interval captures the broadband spectral responses that appear commonly in human intracranial recordings (Ossandon et al., 2012; Vidal et al., 2014). Note that although iEEG signals provide access to neuronal population activity across a wide range of frequencies especially in the gamma band, we focused here on broadband HFA because it has been closely related to population-level neuronal spiking activity (Manning et al., 2009; Ray and Maunsell, 2011) and it is increasingly used as a proxy for active cortical processing (Buzsaki and Wang, 2012; Lachaux et al., 2012). In line with human EEG studies on the link between low and high frequency bands and BOLD signal (Goldman et al., 2002; Laufs et al., 2003; Scheeringa et al., 2011) the high gamma band amplitude fluctuation, also appears to correlate with the fMRI BOLD signal (Logothetis et al., 2001; Lachaux et al., 2007a; Worrell et al., 2012) and has been shown to help neural decoding and real-time mapping of cognitive brain function and processes (Lachaux et al., 2007b; Jerbi et al., 2009a; Hamame et al., 2012).

## **RESULTS**

To use PS by stimulus contrast decrease for subsequent intracranial recordings we first investigated the strength of the PS by the level of contrast decrease in a group of healthy participants. The experiment revealed that small contrast decreases of −10% has only a marginal effect on disappearance probability (8% disappearance probability), as compared to the medium contrast decrease of −50%, which induced 89% of disappearance probability (**Figure 1B**). A repeated measures ANOVA revealed a significant difference [*F*(29) = 178.1; *p* < 0.001) between the disappearance probability of the three levels of contrast decreases (−10, −50, and −100%) and the *post hoc* Tuckey–Kramer test revealed that the −10% condition was different from the other two conditions. We concluded from this first study that decreasing half the contrast could achieve full PS of the visual stimulus, statistically indissociable from full stimulus removal (−100% contrast).

We recorded from a total of 976 electrodes widely distributed throughout the cortex (nine patients). The experimental paradigm is detailed in **Figures 2B,C**. We focused our analyzes on the timewindow indicated by the pink box (**Figures 2B,C**) to compare neural activity elicited by stimulus visibility (VIS) and invisibility (INV) at identical physical stimulus contrast (**Figures 2B,C**). Based on evidence from previous studies on spectral responses of visual processing in intracranial recordings (Vidal et al., 2010) we expected strong modulations within the broad high-gamma band (50–150 Hz) and a low-frequency interval grouping both alpha and beta frequency ranges (8–24 Hz). These signals were extracted with a Hilbert transform procedure (see Materials and Methods section) and subsequently statistically analyzed (detailed in the subsequent paragraph). From these results we chose three representative electrodes in three different cortical areas to proceed to a full TF decomposition with a wavelet procedure, and show the described effects can also be visible within the full TF interval. The results of this analysis are described in the next paragraph.

Time-frequency statistical analyzes revealed a significant poststimulus spectral power modulation relative to pre-stimuluschange baseline level (−200 to −50 ms). **Figures 3A–C** show statistically significant spectral power changes for the two experimental conditions, visible (VIS) and invisible (INV), obtained from three illustrative electrodes from three different participants, and the comparison statistics between the two conditions. First, stimulus invisibility induced a sustained power increase in a rather broad portion of the TF spectra (50–150 Hz). This increase occurred beyond the initial transient activity increase yielded by the contrast change. Simultaneously it induced a sustained power decrease in low-frequency bands, including alpha (8–12 Hz) and beta band (16–24 Hz; Middle panels in **Figures 3A,B**). In the right panel of **Figure 3A** the two distinct frequency intervals are marked as HF, high–frequency, and LF, low-frequency. An opposite response pattern was observed for stimulus visibility obtained during PSR, with a broadband gamma power suppression as the contrast increased (Left panels in **Figures 3A,B**). In the low-frequency range (8–24 Hz) power modulations occurred in opposite sign as for the respective high-frequency modulations. Stimulus contrast increases often elicited power increases in the low-frequency interval while stimulus contrast decreases induced power decrease in this band (**Figures 3A–C**). Though power modulations in both frequency intervals occurred in a variety of brain areas (occipital, temporal, frontal, limbic, and parietal), they did not systematically occur simultaneously at the same sites, as illustrated in **Figure 3C**, for an electrode located in the supramarginal gyrus.

Because contrast changes induced either sustained stimulus visibility or invisibility, beyond the contrast transition, we next focused our analysis on the interval after the initial transient neural response induced by the change in contrast, which occurs in the first couple of 100 ms. We therefore excluded the amplitude signals between 0 and 300 ms post-stimulus change (**Figure 3D**) and operated all subsequent statistical comparisons on the average signal amplitude in the 300–800 ms post-stimulus time window. Though some authors consider conscious processing to initiate in higher visual cortices after the sensory-gated feedforward sweep, i.e., around 100 ms (Lamme, 2003, 2006), others consider this to occur later in time and involving feedback processing from frontal areas, around 300 ms after stimulus onset (Dehaene et al., 2006). After this temporal limit in our recordings, stimulus invisibility by contrast decreases elicited gamma-band amplitude increase mostly in occipital, temporal and frontal cortex (**Figure 4A**). In a majority of medial cortices in temporal and frontal regions (i.e., limbic lobe), gamma-band activity decreased for invisible stimuli. Stimulus visibility elicited fewer responses, and showed mostly gamma-band decreases in temporal, occipital and frontal cortex (**Figure 4A**). Stimulus invisibility induced low-frequency amplitudes suppression on most recording sites across frontal, limbic, parietal, occipital, and temporal lobes (**Figure 4B**). Stimulus visibility induced mostly amplitude increases in all responsive cortical networks (**Figure 4B**). The electrode locations eliciting all the above significant effects within the different cortical lobes are depicted in **Figure 4C**. Importantly, though these effects occur, respectively, for periods of stimulus visibility and invisibility, they also correspond to opposite steps in contrast change (respectively, contrast increase and contrast decrease) and their singular contrastive analysis may not yield per se neural correlates of conscious perception (Aru et al., 2012; de Graaf et al., 2012). Moreover, they may also be affected by the previous states and state transitions, which also imply different contrast levels and contrast level transitions. See the discussion section for further implications of this particularity of our experimental design.

In order to further discriminate the functional processing that might be reflected by local networks we further characterized PS by comparing the amplitude responses elicited by contrast reduction (to 50%) to those elicited by full stimulus offset (**Figures 2B,C**), in line with previous studies and to dissociate sensory from perceptual processing components of neural responses (Wilke et al., 2006, 2009). Although the offset condition differ in terms of low-level physical properties such as time of

presentation and background color, we used this contrast mainly because it elicits a phenomenal perceptual experience of stimulus disappearance highly similar to the one produced by the condition of stimulus contrast reduction. We especially considered neuronal populations located in occipito-temporal and frontal cortical regions who process rather highly integrated visual information and whose activity is modulated by conceptual content of images and their conscious perception (Kreiman et al., 2000, 2002; Quiroga et al., 2007, 2008a,b; Vidal et al., 2010, 2014; Libedinsky and Livingstone, 2011; Panagiotaropoulos et al., 2012). To avoid any interference from the detection task at the end of the contrastchange series, we only used those trials (50%) which were not associated with a behavioral response (see Materials and Methods) but who were equally distributed among the VIS and INV conditions.

**(C)** Brain wide distribution of recording sites displaying either type of neural

emergence response.

By contrasting these three conditions, instead of two conditions, we were able to obtain more information on the involvement of the local neural network in three types of functional processing: (a) *perceptual processing (PERCEPTUAL)*, which differentiates conditions regarding stimulus visibility, independently of stimulus presence or absence, (b) *sensory processing (SEN-SORY)*, which differentiates conditions of stimulus processing independently of stimulus visibility or not, and (c) *suppression processing (SUPPRESSION)*, which differentiates the conditions eliciting stimulus invisibility while processing a sensory stimulus, from the two other conditions. Importantly, these effects are considered independently of sign. We compared the time-averaged amplitude of the three conditions (one way repeated measures ANOVA and Tuckey–Kramer *post hoc* test, FDR-corrected for multiple comparisons), respectively, for lowfrequency and high-frequency responses, and grouped the effects between these conditions into three previously defined functional categories: PERCEPTUAL effects ([INV = OFF]=VIS), SEN-SORY effects ([VIS = INV]=OFF) and SUPPRESSION effects ([VIS = OFF]=INV). In the high-frequency band we found all three response types in most cortical areas (**Figure 5A**). PERCEP-TUAL effects ([INV = OFF]=VIS) in the high-frequency range were found in frontal, temporal, and occipital cortices (**Figure 5A**). SENSORY effects ([VIS = INV]=OFF) were effectively represented in occipital, parietal and temporal cortex, and more surprisingly were also found in frontal sites (**Figure 5A**). Interestingly, SUPPRESSION effects ([VIS = OFF]=INV) were mostly absent in occipital cortex and mainly observed in medial and lateral cortices of the temporal lobe (**Figure 5A**). Low-frequency responses displayed proportionally overall more PERCEPTUAL effects ([INV = OFF]=VIS) in occipital, temporal and parietal networks (**Figure 5B**). SUPPRESSION effects ([VIS = OFF]=INV) were mostly observed in frontal and limbic cortices, while SENSORY effects ([VIS = INV]=OFF) were mostly present in the frontal and limbic lobe.

We now focus on the PVC. In this study one participant (P4) had three electrodes implanted in PVC which have been retinotopically mapped to mid- and upper right quadrant of the visual field (see **Figure 6** for a representation). We were specifically interested in the frequency-specific activity modulation across trials, considering previous studies on PS who studied spike rate and gamma-band activity modulation in V1 of the macaque brain. We found various interesting results (**Figure 6**):

(1) Stimulus offset (OFF) positively modulated broadband gamma activity during the whole temporal interval. The effect started very early due of the temporal smearing inherent to the spectral extraction method (Hilbert) used in our analysis. Interestingly, there was a total absence of amplitude modulation in the INV condition, which involved a stimulus contrast decrease entailing stimulus invisibility. In the VIS condition where the stimulus's contrast was increased, we first

**FIGURE 6 | Grand average of time-resolved high and low frequency responses and their comparison statistics for stimulus visibility (VIS), invisibility (INV), and stimulus offset (OFF) in the PVC**. Above **(A)**, left panel, a retinotopic receptive field map of recording sites, and right panel, localization of electrodes projected on MNI reference brain. **(A)** Broadband gamma responses, significant time-samples of neural emergence test and of condition contrasts. **(B)** Low-frequency responses, same conventions as in **(A)**.

observed a slow progressive negative amplitude modulation, which appeared unrelated to the contrast change of the stimulus but did relate to the perceptual state in the previous period (**Figure 6**). 312 ms after contrast increase we observed a positive amplitude re-bound that shortly after 400 ms became positively modulated as compared to baseline, and lasted until 670 ms.


Finally, we explored the dynamics of the PERCEPTUAL effects across different frontal and temporal recordings sites. It has been shown that neuronal discharges in temporal cortex (Sheinberg and Logothetis, 1997) and in lateral prefrontal cortex (Panagiotaropoulos et al., 2012) reflect unambiguously stimulus visibility at similar latencies, yet no study on PS has ever directly compared these latencies simultaneously in intracranial recordings to explore the information flow of perceptual information between these structures. Two participants within the group of patients who took part of our study had electrodes implanted in both regions (P4 and P7). Though we did not have access to multi-unit or single-unit recordings in those patients, we analyzed broadband gamma amplitude modulations which are often considered as a proxy of spiking activity at the population level (Nir et al., 2008; Manning et al., 2009; Ray and Maunsell, 2011; Buzsaki and Wang, 2012).

Twenty-four recording sites from participants elicited PERCEP-TUAL effects in the broadband gamma range in three regions of interest: temporal cortex (fusiform gyrus/inferior temporal cortex, ITC/FG, *N* = 11), anterior insula (AI, *N* = 7) and inferior frontal gyrus (IFG, *N* = 7). To estimate the latency of these average PERCEPTUAL effects ([INV = OFF] =VIS) we applied a repeated measure ANOVA (conditions: VIS, INV, and OFF) across all trials per region of interest across the full time-interval 0–800 ms, and corrected for multiple comparisons in time. We considered the first significant time sample of a minimum of five consecutive time sample. Here we report that PERCEPTUAL effects initiated, respectively, in the AI at 65 ms [*F*(1992) = 5.16; *p* < 0.01; *post hoc* T–K test, *p* < 0.05], in the IFG at 174 ms

[*F*(1735) = 8.44; *p* < 0.0005; *post hoc* T–K test, *p* < 0.05] and in the ITC/FG at 483 ms [*F*(2978) = 65.3; *p* < 0.0005; *post hoc* T–K test, *p* < 0.05] (**Figures 7A–C**). In all three ROI, the INV condition elicited a positive amplitude modulation while the VIS condition elicited a negative modulation (**Figures 7A–C**). We conducted the same analysis for the low-frequency responses at these three ROI. We found that PERCEPTUAL effects initiated, respectively, in the ITC/FG at 268 ms [*F*(3007) = 6; *p* < 0.05; *post hoc* T–K test, *p* < 0.05], in AI at 377 ms [*F*(2022) = 6.2; *p* < 0.05; *post hoc* T–K test, *p* < 0.05] and in the IFG at 424 ms [*F*(1786) = 5.15; *p* < 0.05; *post hoc* T–K test, *p* < 0.05] (**Figures 7D–F**).

## **DISCUSSION**

Currently the study of conscious visual perception can rely on a wide variety of experimental paradigms (Kim and Blake, 2005; Koch and Tsuchiya, 2007; Dehaene and Changeux, 2011), some of these produce PS which is described as the subjective experience of seeing visible items become invisible. In this study we focused on the neural correlates of conscious visual perception, using an experimental design which produced PS through the transient contrast decrease of a single and entire stimulus (May et al., 2003; Simons et al., 2006). This effect resembles the visual fading effect described by Troxler (1804) yet can be manipulated at wish after a short visual adaptation. By manipulating the temporal order of contrast decreases and increases we obtained in a time period of interest two conditions with equal and unchanging stimulus contrast which elicited two different perceptual states, stimulus visibility, and stimulus invisibility. We recorded human intracranial electrophysiological activity in nine patients and analyzed spectral responses in the broadband gamma 50–150 Hz range and in the low-frequency alpha/beta 8– 24 Hz range. In a first phase we analyzed globally these signals across recording sites in cortical ROI. We report that stimulus invisibility by PS produces a sustained energy decrease in lowfrequency bands across recording sites in many regions including occipital, temporal, limbic, parietal, and frontal cortex, while stimulus visibility achieved through suppression release produced a somewhat less strong opposite effect of energy increase. We also report a majority of broadband gamma activity increases during PS in occipital, temporal, and frontal cortices. Overall, stimulus invisibility through PS elicited neural responses across a larger number of recording sites than stimulus visibility. Moreover, by comparing these conditions to a shortly later stimulus offset we were able to describe signal effects as SENSORY, PERCEPTUAL, or SUPPRESSION related. We also report that PERCEPTUAL processing within the low-frequency band was largely present in occipital, temporal, parietal, and frontal cortices. Finally, in a second analysis phase we focused on time-resolved neural activity modulation to estimate the latencies of PERCEPTUAL effects in three ROI: AI, IFG, and ITC/FG. We were also able to analyze neural activity from PVC in one participant. Before discussing our results in the light of previous findings and theoretical accounts, we outline the main differences of our experimental approach as compared to previous PS paradigms, in order to understand some discrepancies between our results and those previously reported.

## **STIMULUS INVISIBILITY THROUGH TRANSIENT CONTRAST DECREASE**

Experimental paradigms that induce the perceptual disappearance of a stimulus such as MIB (Bonneh et al., 2001) and flash suppression (Wilke et al., 2003; Tsuchiya and Koch, 2005) have in common that stimulus invisibility is produced by the presentation of a competing stimulus in the same visual field. These sensory manipulations introduce competitive interactions between different stimuli in the visual field or between sensory streams thereby biasing sensory selection for further conscious processing (and perception) of the newly presented or changing stimulus at the expense of the constant and static ones. Stimulus invisibility could thus be related to the parallel processing of the novel stimulus that becomes visible and whose sensory processing could generate competing neuronal interactions for selective attention. In our PS's experimental approach stimulus invisibility was achieved without the processing of a novel visible stimulus, avoiding competing interactions between distinct sensory representations for sensory or attentional selection, which are known to modulate neural processing within early visual cortices and beyond in temporal networks (Moran and Desimone, 1985; Desimone and Duncan, 1995; Reynolds and Desimone, 1999; Kastner and Ungerleider, 2000; Reynolds et al., 2000; Fries et al., 2001, 2008; Bichot et al., 2005). However, at a local neural networks level, especially in sensory visual cortices, competitive interactions could occur between neurons involved in border or

contour detection very important in figure-ground segregation processes.

Because we achieve invisibility through contrast decrease, it is important to identify whether the neural responses we report resemble those reflecting directly the physical change or whether they could rather reflect the subjective status of stimulus perception. In our study physical contrast decreases implies local luminance increases of the stimulus. Luminance variations and contrast adaptation are known to modulate signal processing in PVC in animals and humans (Movshon and Lennie, 1979; Carandini and Ferster, 1997; Gardner et al., 2005; Peng and Van Essen, 2005; Geisler et al., 2007; Xing et al., 2014). But these effects on occipito-temporal cortices are yet poorly understood. Single neurons and neuronal populations in these regions display less sensibility to low-level stimulus features and respond robustly to high-level integrated objects such as stimulus semantic categories (Desimone et al., 1984; Kreiman et al., 2000; Quiroga et al., 2005; Vidal et al., 2010) and conscious perception (Quiroga et al., 2008b).

#### **BROADBAND GAMMA ACTIVITY MODULATION DURING PERCEPTUAL SUPPRESSION IN HUMAN CORTEX**

Previous electrophysiological measures from monkey PVC showed that luminance decreases produce an increase in neuronal spiking and gamma-band power as compared to luminance increases which tend to suppress neuronal population discharges (Xing et al., 2014). Xing et al. (2014) concluded on the existence of different cortical mechanisms behind brightness adaptation and argued that changes in neuronal firing in V1 neurons following brightness/darkness adaptation modulate the balance between local excitatory and inhibitory networks, thereby affecting global population discharge rates (Peng andVan Essen, 2005; Xing et al., 2014). The broadband gamma amplitude increase observed in our study following stimulus contrast decrease elicited an opposite effect, observed throughout various occipital, temporal and frontoinsular recording sites, but surprisingly not in PVC. Though we currently have no explanation for the differences in PVC between Xing et al. (2014) study and ours, we suggest that it may involve neuronal adaptation effects caused by the prolonged exposure to the same stimulus. These effects probably originate at retinal level (Burbeck and Kelly, 1984; Hunzelmann and Spillmann, 1984).

Perceptual suppression induced with general flash suppression paradigm has been shown to negatively modulate broadband gamma activity in macaque V4, especially when also negatively modulating multi-unit activity during stimulus offset (Wilke et al., 2006). However, such effects were absent in V1 (Wilke et al., 2006; Maier et al., 2008; Keliris et al., 2010) and in certain nuclei of the thalamus (Wilke et al., 2009). Here we report that the positive modulation of gamma-band activity during stimulus invisibility occurs on a majority of responding sites located in occipital and temporal cortex. These responses correlated with PERCEP-TUAL and SUPPRESSION effects and suggest the existence of local networks involved in the active processing of the subjective appearance and disappearance of a visual stimulus. On a related note, neuronal contrast polarity OFF (and ON) signals are processed early on within the visual pathway from the retina to V1 including the thalamus (Hartline, 1938; Kuffler, 1953; Tanaka, 1983; Reid and Alonso, 1995; Jin et al., 2008, 2011; Yeh et al., 2009). A similar type of neural response could exist to signal perceptual disappearances in higher levels of the cortical hierarchy. Noteworthy, this observation takes place in an experimental setting which does not involve a competing interaction with a newly presented stimulus or information stream, thereby discarding any relation to novel stimulus representation processing which might involve spatial, feature, or object selective visual attention, and usually increase gamma-band activity (Fries et al., 2001) but see Chalk et al. (2010) for opposite effects of attention on V1 neural activity. Yet, as mentioned before, our results remain conditioned by possible sensory adaptation effects which are inherent to the PS paradigm that was used. Future studies will need to clarify this relationship.

Finally, though we did not record eye-movements during this study, we suggest that the reported effects, especially in the PS condition (INV), do a priori not reflect saccadic effects. A recent intracranial study showed that peri-saccadic gamma-band activity is positively modulated in medial regions of PVC (Uematsu et al., 2013) without any further effects in all extra-occipital regions. Yet, in our study, we found condition specific effects throughout different cortical regions, and recordings sites in PVC elicited either no gamma-band response to negative contrast changes (INV), or delayed responses to positive contrast changes (VIS) beyond the reported saccade-induced latency effects.

## **GLOBAL LOW-FREQUENCY AMPLITUDE MODULATIONS BY STIMULUS INVISIBILITY**

Changes in low-frequency oscillations have been associated with state changes of cortical networks and cortical communication (Siegel et al., 2012); sustained power suppressions in sensory regions often relate with active local neural processing, sensory change processing but also modulation by cognitive processes such as attention and consciousness (Fries et al., 2001; Gaillard et al., 2009; Levy et al., 2013). Previous studies reported a sustained low-frequency activity decrease of the LFP in early visual cortex, V1–V4, following target invisibility by general flash suppression (Wilke et al., 2006; Maier et al., 2008). We report this same type of effect generalized across extra-striate visual cortices in the occipito-temporal region, such as lateral occipital cortex, temporal cortex (including fusiform gyrus, middle temporal gyrus) but also frontal and insular cortices. The extent of this finding across participants and within this wide variety of cortical areas beyond PVC stresses the generality of this effect and the cortically distributed processing involved in PS and stimulus invisibility. However, while low-frequency amplitude decreases occurred massively, they did not systematically coincide with simultaneous broadband gamma activity increases. This is similar to previous observations from monkey brain area V1 where either neuronal discharge rates or gamma-band power were modulated independently from alphaband oscillatory power (Wilke et al., 2006; Maier et al., 2008). In fact, in these studies either neuronal discharges or broadband gamma activity did not correlate to stimulus visibility status, thereby suggesting that very early sensory neural structures might not directly enable or be the initiator of conscious perception processing. Low-frequency power suppression across a large proportion of cortices, sensory and non-sensory, could thus indicate a profound change of state as has been suggested previously for beta-band oscillations (Engel and Fries, 2010; Panagiotaropoulos et al., 2013; Rey et al., 2014) hence not being directly involved in the processing of the content of conscious perception.

## **STIMULUS VISIBILITY-RELATED NEURAL MODULATIONS IN PRIMARY VISUAL CORTEX**

The neural emergence analysis of the gamma-band activity in PVC reveals three important pieces of information: (1) stimulus offset (OFF) induces an immediate positive response, terminating the adaptation period, (2) contrast decrease (INV) elicits no response, and (3) a late positive amplitude increase occurs for stimulus contrast increase, around 300 ms, while no early reaction similar to the OFF condition is observed. It appears as if the early progressive negative amplitude modulation in VIS (<300 ms) does not directly reflect the contrast increase.

The absence of gamma-band activity modulation during PS in V1 may be caused by a silencing of neuronal activity caused by forced adaptation of downstream retinal cell populations. If a contrast decrease reduces the sensory gain of edge neurons in these populations, the figure-ground segregation process might be compromised and produce a relative gain increase for background neurons, relayed upstream to V1. This could bias the competitive interactions and entail a filling-in phenomena. However, as suggested by previous studies (Wilke et al., 2006; Maier et al., 2008; Keliris et al., 2010) and current discussions on the role of V1 in conscious visual perception, it is possible that this structure though necessary may not be sufficient for visual awareness, and that this function rather concerns frontal and temporal cortices which could consolidate a cortico-thalamo-cortico network that conveys the content of conscious experience (Leopold, 2012; Panagiotaropoulos et al., 2014).

The neural emergence pattern of the VIS condition hints towards a local network activation around 300 ms. This event could reflect local cortical activation triggered by distant upstream networks. This interpretation is in line with theoretical concerns on the necessary involvement of frontal feedback signals in the ignition of conscious processing in cortical networks (Dehaene and Changeux, 2011; Boly et al., 2013). But if this delayed gammaband rebound reflects top–down processing, why did it not result in a PERCEPTUAL effect? On the one hand we can be critical about our contrasts effectiveness. On the other hand, this effect could constitute the first electrophysiological example of a salience-driven attention effect without the underlying conscious processing, as suggested by previous neuroimaging data (Watanabe et al., 2011) and as discussed by others (Leopold, 2012; Panagiotaropoulos et al., 2014). This interpretation, however, needs further investigation.

A previous human intracranial study using visual masking with string stimuli observed a general increase of broadband gamma activity across all recording sites between 200 and 300 ms (Gaillard et al., 2009). However, the authors did not single out, or did not have, activity from PVC, though they showed average lobespecific activity (ex: occipital lobe). They reported earlier visual masking effects in the gamma-band in the occipital lobe than in the frontal lobe. In our study, PERCEPTUAL effects were observed first in fronto-insular networks and then in temporal and occipital visual areas, but not PVC. Another human intracranial study also using visual masking but contrasting consciousness based on perceptual reports exclusively, focused on single recording sites within the occipito-temporal cortex (Fisch et al., 2009). In this study, the broadband gamma activity showed a high degree of category selectivity for complex visual stimuli such as faces, houses, objects, etc., and similar to the study of Gaillard et al. (2009) also reported differential effects related to conscious perception between 250 and 300 ms. However, there was no significant effect of conscious stimulus recognition in the broadband gamma range within low-level visual areas. Considering the lack of evidence of similar effects in V1 of the macaque brain during PS (Wilke et al., 2006), our results in the broadband gamma range suggest that PVC is not primarily involved in time in the emergence of visual consciousness. However, the sudden positive amplitude rebound suggest that a modulatory influence related to stimulus visibility, maybe from the involvement of top–down processes, reaches PVC around 300 ms. The choice of the three conditions in our contrastive analysis may also not be optimally matched so as to reveal conscious perception effects (i.e., PERCEPRTUAL). Importantly, the possibility of a significant contribution of adaptation effects to our result pattern remains to be discarded in future studies.

Low-frequency PERCEPTUAL effects started at 350 ms and were coincident with significant amplitude increases in the VIS condition and amplitude decreases in the INV condition. From a more global perspective, this result is coherent with low-frequency responses observed during PS in macaque V1 (Wilke et al., 2006) but also within subcortical structures such as in the pulvinar (Wilke et al., 2009). However, recordings in the latter only showed modulations of low-frequency power when the animal was actively involved in reporting its perception, and not when passively viewing the perceptual alternations. In our study this power modulation was not dependent on behavioral report. Moreover, our results are incongruent with the effects in the low-frequency band (i.e., beta band) reported by Gaillard et al. (2009) in human occipital lobe in both modulation latency and sign, which occurs earlier and is positive. As stated previously in the introduction, this difference may be due to specifics of physical stimulation, stimulus adaptation, and behavioral task within experimental designs used in these studies: PS and masking use within their contrastive analyzes qualitatively different types of invisibilities which probably also elicit different neural responses. In masking paradigms, stimulus invisibility is induced on the path toward conscious perception, however, never reaching it. In PS, stimulus invisibility occurs after the stimulus has reached a conscious status.

### **INSULAR AND FRONTAL CONTRIBUTIONS TO PERCEPTUAL SUPPRESSION AND CONSCIOUS PERCEPTION**

Most broadband gamma activity modulation by PERCEPTUAL effects located in occipito-temporal cortex (excluding PVC). Yet, we also found these responses in anterior insular cortex and inferior frontal cortex with overall onset latencies much earlier than those reported in visual areas. In the midst of the ongoing discussions on the respective participation of V1 and frontal regions in conscious perception (Leopold, 2012; Safavi et al., 2014) our results are coherent with current experimental results showing modulation of neuronal activity in frontal networks correlated to stimulus visibility (Libedinsky and Livingstone, 2011; Panagiotaropoulos et al., 2012). Because (1) it has been recently shown that AI is structurally connected to posterior visual areas including temporal cortex (Jakab et al., 2012), (2) that AI has been shown to concentrate Von Economo neurons (Nimchinsky et al., 1999) which could transmit at very high speed information through long-distance connections in the brain, and (3) based on the latencies of our broadband gamma PERCEPTUAL effects, we suggest that in our PS paradigm conscious perception's neural signaling emerges primarily in AI and may then propagate to IFG and other high-level cortical structures to generate top–down influences upon the downstream sensory networks. AI could also directly influence visual processing based on its direct connections with these networks (Jakab et al., 2012). This observation and proposal will need to be verified in future studies. The later emerging PERCEPTUAL effects in the lowfrequency band in AI may thus be signaling a consequence of the first (and probably feed the forward sweep) of conscious perception.

Why do insular networks produce increased broadband gamma activity for stimulus disappearance? First, anterior insula networks have been associated with the processing of perceptual saliency (Seeley et al., 2007; Corbetta et al., 2008) and the effects we report may reflect the high saliency value for sudden stimulus disappearance. This does, however, not explain the opposite pattern, broadband gamma decrease, for stimulus re-appearance (VIS). Moreover, neural activity decrease in this region is temporally related to upcoming lapses of attention and mind wandering (Weissman et al., 2006; Mason et al., 2007) thereby negatively affecting conscious sensory processing, while in our study it is related to a gain in conscious perception through stimulus visibility. Current models on the role of the insula suggest it performs a complex integration of interoceptive and perceptual information thereby possibly generating a unified conscious feeling of self (Craig, 2009). It is therefore unlikely that the effects reported in this brain region are related to low-level feature coding like the ones related to contrast coding in PVC. The fronto-insular perceptual effects we report could thus reflect the first emergence of conscious visual perception from the top of the cortical hierarchy (Hochstein and Ahissar, 2002).

#### **ACKNOWLEDGMENTS**

We thank all patients for their participation; the staff of the Grenoble Neurological Hospital epilepsy unit; Dominique Hoffmann, Patricia Boschetti, Véronique Dorlin for their support. This work was supported by MLA ANR, by the LABEX CORTEX (ANR-11-LABX-0042) of Université de Lyon, within the program "Investissements d'Avenir" (ANR-11-IDEX-0007). Juan R. Vidal, thanks Monica Baciu for her support.

#### **REFERENCES**


executive control. *J. Neurosci.* 27, 2349–2356. doi: 10.1523/JNEUROSCI.5587- 06.2007


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 07 July 2014; accepted: 14 December 2014; published online: 15 January 2015.*

*Citation: Vidal JR, Perrone-Bertolotti M, Kahane P and Lachaux J-P (2015) Intracranial spectral amplitude dynamics of perceptual suppression in frontoinsular, occipito-temporal, and primary visual cortex. Front. Psychol. 5:1545. doi: 10.3389/fpsyg.2014.01545*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2015 Vidal, Perrone-Bertolotti, Kahane and Lachaux. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

**OPINION ARTICLE** published: 19 September 2014 doi: 10.3389/fpsyg.2014.01063

## Is the frontal lobe involved in conscious perception?

#### *Shervin Safavi 1,2†, Vishal Kapoor 1,2†, Nikos K. Logothetis 1,3 and Theofanis I. Panagiotaropoulos <sup>1</sup> \**

*<sup>1</sup> Department Physiology of Cognitive Processes, Max Planck Institute for Biological Cybernetics, Tübingen, Germany*

*\*Correspondence: theofanis.panagiotaropoulos@tuebingen.mpg.de*

*†These authors have contributed equally to this work.*

#### *Edited by:*

*Jaan Aru, University of Tartu, Estonia*

#### *Reviewed by:*

*Wolfgang Einhauser, Philipps-Universität Marburg, Germany Jaan Aru, University of Tartu, Estonia*

**Keywords: prefrontal cortex, conscious visual perception, frontal lobe, binocular rivalry, perceptual suppression**

When studying the neural mechanisms underlying conscious perception we should be careful not to misinterpret evidence, and delineate these mechanisms from activity which could reflect the prerequisites or consequences of conscious experiences (Aru et al., 2012; De Graaf et al., 2012). However, at the same time, we need to be careful not to exclude any relevant evidence about the phenomenon.

Recently, novel paradigms have attempted to dissociate activity related to conscious perception from activity reflecting its prerequisites and consequences. In particular, one of these studies focused on resolving the role of frontal lobe in conscious perception (Frässle et al., 2014). Through a clever experimental design that contrasted blood-oxygen-level-dependent (BOLD) activity elicited during binocular rivalry with and without behavioral reports, Frässle et al. (2014) suggested that frontal lobe, or a large part of it, may not be necessary for conscious perception *per se*. Rather frontal areas are involved in processing the consequences of conscious perception like monitoring the perceptual content in order to elicit the appropriate report of the subjective experience. In particular, Frässle et al. showed that behavioral reports of conscious experiences resulted in increased and more widespread activity of the frontal lobe compared to a condition without behavioral reports, where spontaneous transitions in the content of consciousness were estimated through the objective measures like optokinetic nystagmus (OKN) and pupil dilation. The authors of this study concluded that "*frontal areas are associated with active* *report and introspection rather than with rivalry per se.*" Therefore, activity in prefrontal regions could be considered as a consequence rather than a direct neural correlate of conscious experience.

However, a previous study (Panagiotaropoulos et al., 2012) that measured directly neural activity in the macaque lateral prefrontal cortex (LPFC) using extracellular electrophysiological recordings could help to narrow down the role of frontal activity in conscious perception and exclude the contribution of cognitive or motor consequences in prefrontal neural activity during visual awareness. Specifically, the activity of feature selective neurons in the macaque LFPC was shown to be modulated in accordance with the content of subjective perception, without any confound from motor action (i.e., behavioral reports). Using binocular flash suppression (BFS), a paradigm of robust, externally induced perceptual suppression and without any requirement of behavioral reports, neurons in the LPFC were found to increase or decrease their discharge activity when their preferred stimulus was perceptually dominant or suppressed, respectively. Therefore, since neuronal discharges in the LPFC follow the content of conscious perception even without any motor action, the conclusion of Frässle et al. (2014) about the role of frontal lobe activity in rivalrous perception needs to be refined. Prefrontal activity can indeed reflect the content of conscious perception under conditions of rivalrous stimulation and this activity should not be necessarily considered as the result of a motor action or self-monitoring required for active report. Moreover, the results obtained by Frässle et al. (2014) do not anatomically preclude the entire prefrontal cortex from having a role in conscious perception. Specifically, the BOLD activity related to rivalry in their experiment is still present in the right inferior frontal lobe and right superior frontal lobe (Zaretskaya and Narinyan, 2014). Further, activation of dorso- LPFC in conscious perception of Mooney images was also reported in a study that explicitly controlled for activity elicited by motor action (Imamoglu et al., 2012).

It is true that the BFS-related prefrontal activity cannot conclude on a mechanistic, causal involvement of prefrontal activity in driving spontaneous transitions in conscious perception. This is because BFS is a paradigm of externally induced perceptual suppression and is therefore not directly informative about the role of recorded activity in spontaneous transitions. Therefore, the possibility remains open that the kind of prefrontal activity observed in the macaque LPFC during BFS is not a causal factor for conscious perception but rather reflects some other aspects of monitoring that are not directly related to motor action. For example, prefrontal activity could just reflect a read-out from other areas like the inferior temporal cortex (Sheinberg and Logothetis, 1997) that also reliably reflects the content of conscious perception. However, if this is the case, it triggers the question why this activity that closely follows the content of subjective perception is observed in the LPFC even in the absence of any behavioral report. Overall, it motivates further investigation to understand whether prefrontal activity

*<sup>2</sup> International Max Planck Research School for Cognitive and Systems Neuroscience, University of Tübingen, Tübingen, Germany*

*<sup>3</sup> Department of Imaging Science and Biomedical Engineering, University of Manchester, Manchester, UK*

has a mechanistic role in conscious perception or it might underlie some monitoring functions that are not necessarily bound to motor action.

Similar to this debate on the role of LPFC in visual awareness, the last decade witnessed disagreement on whether activity in primary visual cortex reflects subjective perception as monitored with electrophysiology and fMRI (Leopold and Logothetis, 1996; Tong, 2003; Maier et al., 2008; Keliris et al., 2010; Leopold, 2012). Measuring both electrophysiological activity and the BOLD signal in the same macaques engaged in an identical task of perceptual suppression finally provided the solution (Maier et al., 2008; Leopold, 2012). Therefore, in order to investigate and resolve the role of PFC in visual perception, one must take a similar approach that utilizes multiple measurement techniques simultaneously or in the same animal along with a careful experimental design. The experimental tasks should not only segregate the effect of various cognitive processes such as attention or introspection in comparison to awareness (Watanabe et al., 2011; Frässle et al., 2014), but also use an objective criterion to decode the content of conscious experience (Frässle et al., 2014), therefore separating perception-related activities from the subsequent behavioral report. Such an approach could therefore robustly delineate the prerequisites and consequences of conscious experience and reveal the true correlates of conscious perception.

Lastly, although such a multimodal approach could provide us substantial insights into the activity underlying the representation of conscious content, whether or not this activity has a causal role in mediating perception remains to be understood. Although a number of studies indeed point to a causal involvement of prefrontal cortex in conscious perception (reviewed in Dehaene and Changeux, 2011), a systematic study which directly interferes with prefrontal activity during a task of subjective perception is currently, to the best of our knowledge, missing. While utilizing objective criteria as indicators of perceptual transitions, systematic perturbation of the PFC (such as cooling, transcranial magnetic stimulation, microstimulation, or optogenetics) and observing concomitant changes in the temporal dynamics of perceptual transitions could reveal its causal contribution. Indeed, patients with frontal lesions are impaired in their ability to switch from one subjective view of an ambiguous figure to the other (for example see Ricci and Blundo, 1990, but also see a different case study from Valle-Inclán and Gallego, 2006).

We would like to conclude that in formulating our conclusions related to prerequisites, consequences and true correlates of conscious experiences, we need to have an *integrative view* on the available evidence. Our investigations and conclusions about the neural correlates of consciousness must not only entail betterdesigned experiments but also diverse experimental techniques (e.g., BOLD fMRI, electrophysiology) that could measure brain activity on different spatial and temporal scales (Panagiotaropoulos et al., 2014). Such a multi-modal approach holds great promise in refining our current understanding of conscious processing.

## **ACKNOWLEDGMENT**

This work was supported by the Max Planck Society and the International Max Planck Research School for Cognitive and Systems Neuroscience, University of Tübingen, Germany.

## **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 July 2014; accepted: 04 September 2014; published online: 19 September 2014.*

*Citation: Safavi S, Kapoor V, Logothetis NK and Panagiotaropoulos TI (2014) Is the frontal lobe involved in conscious perception? Front. Psychol. 5:1063. doi: 10.3389/fpsyg.2014.01063*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Safavi, Kapoor, Logothetis and Panagiotaropoulos. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

**REVIEW ARTICLE** published: 21 October 2014 doi: 10.3389/fpsyg.2014.01173

## Probing feedforward and feedback contributions to awareness with visual masking and transcranial magnetic stimulation

## *Evelina Tapia1\* and Diane M. Beck1,2*

*<sup>1</sup> Beckman Institute, University of Illinois Urbana-Champaign, Urbana, IL, USA <sup>2</sup> Department of Psychology, University of Illinois Urbana-Champaign, Urbana, IL, USA*

#### *Edited by:*

*Talis Bachmann, University of Tartu, Estonia*

#### *Reviewed by:*

*Michael Herzog, École Polytechnique Fédérale de Lausanne, Switzerland Tom Alexander De Graaf, Maastricht University, Netherlands*

#### *\*Correspondence:*

*Evelina Tapia, Beckman Institute, University of Illinois Urbana-Champaign, 405 N Mathews Avenue, Urbana, IL 61801, USA e-mail: evelinatapia@gmail.com*

A number of influential theories posit that visual awareness relies not only on the initial, stimulus-driven (i.e., feedforward) sweep of activation but also on recurrent feedback activity within and between brain regions. These theories of awareness draw heavily on data from masking paradigms in which visibility of one stimulus is reduced due to the presence of another stimulus. More recently transcranial magnetic stimulation (TMS) has been used to study the temporal dynamics of visual awareness. TMS over occipital cortex affects performance on visual tasks at distinct time points and in a manner that is comparable to visual masking.We draw parallels between these two methods and examine evidence for the neural mechanisms by which visual masking and TMS suppress stimulus visibility. Specifically, both methods have been proposed to affect feedforward as well as feedback signals when applied at distinct time windows relative to stimulus onset and as a result modify visual awareness. Most recent empirical evidence, moreover, suggests that while visual masking and TMS impact stimulus visibility comparably, the processes these methods affect may not be as similar as previously thought. In addition to reviewing both masking and TMS studies that examine feedforward and feedback processes in vision, we raise questions to guide future studies and further probe the necessary conditions for visual awareness.

**Keywords:TMS, visual masking, awareness, feedforward, feedback, alpha oscillations, alpha phase, alpha power**

## **INTRODUCTION**

The exact mechanism by which visual awareness arises and the neural circuits involved in generating this experience are greatly debated (Crick and Koch, 1995; Zeki and Bartels, 1998; Lamme, 2000, 2006a; Dehaene and Naccache, 2001; Dennett, 2001; Rees et al., 2002; Cooney and Gazzaniga, 2003; Crick and Koch, 2003; Lamme, 2003; Tong, 2003; Block, 2005; Lamme, 2006a; Zeki, 2008). However, many theories assume that some version of recurrent activity or feedback is necessary for awareness. Here we review findings from visual masking and transcranial magnetic stimulation (TMS) studies of visual suppression. In both paradigms visual awareness of a stimulus is disrupted at distinct points in time from the onset of the stimulus, and thus they have been used to elucidate the relationship between feedforward and feedback processes in awareness. The overlap in the windows of visual suppression with these two methods has been proposed to reflect a similar mechanism by which stimulus visibility is impaired (Breitmeyer et al., 2004a). That is, forward visual masking and TMS applied before the onset of a stimulus have been suggested to reflect interference with the feedforward visual processes; backward visual masking and TMS applied after the onset of the stimulus have been implicated in interference with the feedback visual processes. A closer examination of more recent studies, however, suggests that TMS effects that have been previously attributed to disruption of feedback processes

may also reflect interference with feedforward visual processing. Additionally, there is increasing evidence that TMS can modulate alpha frequency oscillations that in turn can impact visual awareness. Such findings complicate the interpretation of TMS effects on awareness. For example, the documented influences of prestimulus alpha oscillations on awareness raise the possibility that the visual masks or TMS stimulation that precede the target may impact awareness by changing the brain state prior to the onset of the target. We argue that despite the current lack of clarity regarding their neural mechanisms, both visual masking and TMS are useful methods for studying the neural conditions necessary for visual awareness.

## **THE VISUAL SYSTEM AND THE FEEDFORWARD-FEEDBACK FRAMEWORK**

The visual system has abundant anatomical feedforward and feedback connections that are organized in a hierarchical manner (DeYoe and Van Essen, 1988; Felleman and Van Essen, 1991; Van Essen et al., 1992; Merigan and Maunsell, 1993; Felleman et al., 1997; Lamme and Roelfsema, 2000; Bullier, 2001). Initially, information enters the cortex in a feedforward manner. Electrophysiological studies in humans indicate that the visual signal from the retina reaches the primary visual cortex, V1, in 55–70 ms after stimulus onset (Wilson et al., 1983; Baseler and Sutter, 1997; Luck et al., 1997; Vanni et al., 2001; Foxe and Simpson, 2002; Di Russo

et al., 2003; Boehler et al., 2008). However, it is worthy to note that the magnocellular and parvocellular pathways have different response latencies (Maunsell and Gibson, 1992; Schmolesky et al., 1998), with the magnocellular pathway responding earlier and maintaining approximately 10–15 ms advantage over the parvocellular pathway across areas in the early visual cortex (Schmolesky et al., 1998). From V1 information travels to temporal, parietal, and frontal cortices via feedforward connections in the ventral and dorsal streams (Ungerleider, 1985). Importantly, information not only propagates forward, but laterally and backwards; that is, hierarchically higher areas send signals to lower areas via feedback connections (DeYoe and Van Essen, 1988; Felleman and Van Essen, 1991; Van Essen et al., 1992; Merigan and Maunsell, 1993; Lamme and Roelfsema, 2000; Bullier, 2001) and information within an area is transformed via lateral neural connections (Felleman and Van Essen, 1991).

The role of feedback processes in vision as well as when and how they are initiated is still debated. Feedback amplifies and focuses activity of neurons in lower visual areas (Hupé et al., 1998, 2001). However, the function of such modulation is contested. Some argue that feedback is involved primarily in attentional modulation of the feedforward sweep (Macknik and Martinez-Conde, 2007), while others postulate that feedback modulates activity in early sensory areas based on expectations or to minimize prediction error (Rao and Ballard, 1999; Panichello et al., 2012). Several other models fall within a general "frame-and-fill" approach where feedback serves to fill in details of an initially established scene (Bullier, 2001; Hochstein and Ahissar, 2002; Bar, 2003; Ahissar and Hochstein, 2004; Bar et al., 2006; Chen et al., 2007; Kveraga et al., 2007; Ahissar et al., 2009; Peyrin et al., 2010). Regardless of its specific function, it is argued that this feedback activity is an essential component of an emergent visual awareness (Di Lollo et al., 2000; Enns and Di Lollo, 2000; Lamme, 2000, 2006a; Lamme and Roelfsema, 2000; Lamme et al., 2002; Tong, 2003; Breitmeyer, 2007; Fahrenfort et al., 2007, 2008). While these theories concur that feedback is necessary for awareness, we note that they should not be construed as arguing that feedback is sufficient for awareness. For example, feature integration can be modulated over hundreds of milliseconds even though the actual stimuli fail to reach awareness (Otto et al., 2006; Plomp et al., 2009; Scharnowski et al., 2009; Rüter et al., 2010), suggesting that while information is maintained in the visual system (presumably due to recurrent processing) additional factors determine whether it eventually becomes a conscious percept (e.g., see Herzog et al., 2007). In the context of visual awareness we explicitly define recurrent feedback as activity that encompasses recurrent processing within and among adjacent areas as well as reentrant activity from hierarchically higher to lower brain areas, the latter of which reside in the early visual cortex. Importantly, recurrent feedback occurs after and as a result of the initial feedforward signal. We will use "recurrent" and "feedback" as synonyms throughout.

It has been a challenge to establish the exact timing of these neural events because they are dependent on stimulus properties, species-specific neural architecture, and experimental procedures. For example, because magno and parvo cells exhibit distinct spatial frequency and contrast sensitivities (Kaplan and Shapley, 1986; DeYoe and Van Essen, 1988; Livingstone and Hubel, 1988; Van Essen et al., 1992; Schyns and Oliva, 1994; Sincich and Horton, 2005), the physical properties of the stimulus modulate signal arrival time in V1 and higher visual areas (Baseler and Sutter, 1997; Schmolesky et al., 1998; Alexander et al., 2005; Foxe et al., 2008). Nonetheless, some have estimated that activity occurring prior to 100 ms post-stimulus onset in the human brain corresponds to a feedforward signal, whereas feedback impacts activity in a later time period, 100 ms or more after stimulus onset (e.g., Fahrenfort et al., 2007; Boehler et al., 2008). However, if the definition of feedback encompasses any recurrent activity, as ours does, then such activity could also occur before 100 ms (e.g., Nowak et al., 1995; Foxe and Simpson, 2002). As will become clear in subsequent sections, it is difficult to pinpoint specifically when feedback might be playing a role in awareness, but by considering both visual masking and TMS studies together, we argue that the presence of both feedforward and feedback processes can be inferred.

It has been argued that some natural stimuli, such as real-world objects, animals, or scenes, can be perceived on the feedforward sweep; that is, observers can successfully identify such stimuli with presentation times that are too short to allow for feedback (Thorpe et al., 1996; VanRullen and Thorpe, 2001; Rousselet et al., 2002; VanRullen and Koch, 2003; VanRullen, 2007; Schmidt and Schmidt, 2009; Koivisto et al., 2014a). In these studies feedback is typically conceived although not explicitly defined as activity from higher to lower neural areas and, in particular, activity from frontal regions involved in decision making projecting to occipitotemporal areas. Others have argued that identification of such natural stimuli may rely on very fast feedback processes (Bar, 2003; Bar et al., 2006; Kveraga et al., 2007). However, we note that the concept of feedback processes as well as masking methods used in these ultra-rapid presentation paradigms do not strictly preclude recurrent processing and thus they cannot be taken as strong evidence against the idea that awareness requires feedback activity. We return to this issue in the next section.

Numerous theories and methods of visual masking exist, and the nuances of these models are beyond the scope of this review. Most models fit within the general framework of feedforward and feedback processing that we discuss in this paper. However, for completeness sake, we direct an interested reader to several excellent reviews that discuss alternative methods and theories of visual masking (Breitmeyer and Ganz, 1976; Breitmeyer, 1984; Bachmann, 1994; Francis, 1997, 2000; Di Lollo et al., 2000; Enns and Di Lollo, 2000; Breitmeyer and Öˇgmen, 2000, 2006; Francis and Herzog, 2004; Macknik, 2006; Ansorge et al., 2007; Kouider and Dehaene, 2007; Hermens et al., 2008; Ghose et al., 2012; Goodhew et al., 2013; Bachmann and Francis, 2014).

## **SUPPRESSING STIMULI FROM AWARENESS: BACKWARD MASKING**

#### **VISUAL MASKING**

Visual masking occurs when perception of one stimulus, a target, is reduced by the presence of a second stimulus, a mask. The strength of masking is quantified as the reduction in the visibility of some aspect of the target (e.g., its shape). In forward masking the mask stimulus precedes the to-be-discriminated target; in backward masking the mask follows the to-be-discriminated target. Metacontrast is a specific case of backward masking in which spatially contiguous but non-overlapping stimuli are used. Because the target and mask do not overlap spatially, and thus do not stimulate the same retinal cells, any interaction between the target, and the mask must occur primarily at the cortical level. In line with this idea, target visibility is strongly impaired under dichoptic presentation conditions in which target and mask stimuli are presented separately to each eye (Kolers and Rosner, 1960; Schiller and Smith, 1968; Weisstein, 1971; Breitmeyer and Kersey, 1981). Here, the visual signals from each stimulus can only interact at post-retinal levels.

By varying the stimulus onset asynchrony (SOA) between the target and the mask one can track the change in the visibility of the target with respect to the onset of the mask. Metacontrast typically yields U-shaped target visibility functions (see **Figure 1**). The exact SOAs of optimal masking vary from study to study due to differences in experimental parameters and criterion the subject is asked to adapt (Kahneman, 1968; Breitmeyer and Öˇgmen, 2006). In general, however, metacontrast masking produces the strongest suppression of target visibility at approximately a 50 ms SOA while preserving stimulus visibility at the earlier SOAs (e.g., Breitmeyer, 1978; Enns and Di Lollo, 1997; Breitmeyer et al., 2008; for reviews see Breitmeyer and Öˇgmen, 2000, 2006; **Figure 1**). This pattern of results indicates that the strongest interaction between target and mask neural events occurs when they are separated by several tens of milliseconds rather than immediately.

The non-monotonic, U-shaped visibility of target function differs markedly from integration masking in which masking is most effective at an SOA of 0 ms and decreases in effectiveness monotonically with increasing SOAs (e.g., Hellige et al., 1977). This pattern of results suggests (nearly) immediate interactions between neural target and mask signals rather than after some delay. In other words, the very same SOAs that produce maximal integration masking (typically obtained when the target and mask spatially overlap) produce little or no masking during metacontrast (when target and mask do not spatially overlap). Whereas integration masking is largest when the mask is near in time to the feedforward signal of the target, metacontrast masking specifically circumvents that form of masking and instead affects some later processes beyond the initial feedforward signal.

Metacontrast and integration masking are also thought to differ in terms of where in the visual processing stream they occur. Integration masking is thought to occur when the target and mask stimulate common retinal and early cortical cells, resulting in an integration of target and mask signals starting at the earliest stages of processing (Weisstein, 1972; Breitmeyer, 1978; Hellige et al., 1979; Michaels and Turvey, 1979; Breitmeyer and Öˇgmen, 2006). Metacontrast masking, on the other hand, is designed to minimize retinal integration and is thought to primarily reflect cortical interactions. It has been argued, however, that even these non-spatially overlapping stimuli may in fact be integrated (as opposed to interrupted, as it is sometimes described), although presumably at later levels of the visual system than the classic

from Breitmeyer et al., 2006). **Lower**: stimulus visibility in a transcranial magnetic stimulation (TMS) paradigm (adapted from Tapia et al., 2014). When the average retinocortical transmission time of 60 ms (gray region) is taken into account, we can directly compare timing of neural events elicited by target and mask or TMS stimuli in EVC as indicated by the bolded horizontal axis. Both methods suppress stimulus visibility in comparable time windows, reflecting interference with the same visual process. See text for more details.

"integration masking" effects (Herzog and Koch, 2001; Otto et al., 2006; Plomp et al., 2009; Scharnowski et al., 2009; Rüter et al., 2010). In general, when masking methods are used to investigate awareness, care should be taken regarding the assumptions about the mechanism by which stimuli were suppressed from awareness. With this in mind, whenever relevant to our discussion we will only draw on evidence from masking studies with overlapping stimuli if cortical interactions between target and mask stimuli can be inferred (also see Breitmeyer and Öˇgmen, 2006, for a further discussion on this point). Also, note that the latency differences between magnocellular and parvocellular channels are accommodated within the range of retinocortical transmission times discussed below, and thus we do not separate out the contributions of these pathways to visual masking effects and awareness.

#### **TRANSCRANIAL MAGNETIC STIMULATION**

Another way in which to bypass sensory signal interactions in areas before V1 is to directly stimulate visual cortex with TMS. The rapid changes in the magnetic field generated by a TMS coil can induce electrical activity in the brain through the scalp and skull, thus non-invasively stimulating the brain area under the coil. In their seminal paper, Amassian et al. (1989) applied TMS to the occipital pole at varying SOAs relative to the stimulus onset. Reminiscent of metacontrast masking results, they found that occipital TMS reduced stimulus visibility when applied 80–100 ms after stimulus onset. Since then further research has confirmed that single-pulse TMS reduces target visibility in distinct temporal epochs when applied to the occipital pole, stimulation of which has since been shown to include V1, V2, and even V3 (Kammer et al., 2005b; McKeefry et al., 2009; Thielscher et al., 2010; Salminen-Vaparanta et al., 2012). This suppression of visibility effect has been more consistently reported as a window centered at a post-stimulus SOA of 100 ms (Amassian et al., 1989; Paulus et al., 1999; Corthout et al., 1999a,b, 2002; Heinen et al., 2005; Kammer et al., 2005b; Kammer, 2007a; Sack et al., 2009; Camprodon et al., 2010; de Graaf et al., 2011a, 2012a; Koivisto et al., 2011b; Railo and Koivisto, 2012; Emmanouil et al., 2013; Allen et al., 2014; Tapia et al., 2014; **Figure 1**). Several studies have also reported an additional and very late post-stimulus TMS effect at 200 to 280 ms after the onset of the stimulus (Allen et al., 2014), some in tasks that required animal categorization and relied on figure-ground segmentation processes (Heinen et al., 2005; Camprodon et al., 2010; Koivisto et al., 2011a).

Unlike in visual masking, where both target and mask enter the visual system via the retina, in TMS masking studies an externally induced TMS event applied directly over EVC affects an internal event in EVC that was initiated at the retina. Given that the retinocortical transmission time varies from 55 to 70 ms and averages around 60 ms (Wilson et al., 1983; Baseler and Sutter, 1997; Vanni et al., 2001; Foxe and Simpson, 2002; Boehler et al., 2008), any effect of TMS must first account for this retinocortical transmission time. For example, if we assume that it takes approximately 60 ms after the stimulus onset before TMS can directly interact with the incoming visual signal, then an effect of TMS on visibility at the 100 ms SOA would mean that TMS had its effect on the visual information approximately 40 ms after the sensory signal reached EVC.

After a retinocortical transmission adjustment is made, we can begin to compare SOAs in visual and TMS masking studies (see also Railo and Koivisto, 2012). **Figure 1** shows target visibility functions in metacontrast and post-stimulus TMS studies with the 60 ms adjustment. The 50 ms backward masking effect dovetails with post-stimulus TMS effects at an EVC-adjusted SOA of 40 ms (i.e., 100 ms post-stimulus onset). In other words, visual and TMS masking paradigms impair stimulus visibility at comparable time windows, suggesting that they affect the same visual process(es) that are critical for visual awareness (Breitmeyer et al., 2004a; Lamme, 2006b; Breitmeyer, 2007).

#### **NEURAL MECHANISMS**

Data regarding the neural effects of backward masking is consistent with the idea that the mask interferes with feedback rather than an initial feedforward signal of the target. Single cell responses during backward masking have been recorded from a number of different primate brain areas involved in visual processing. In area V1 metacontrast masks suppress spike activity in a later time window (post-100 ms) while the early neural response is generally unaffected (Bridgeman, 1980; Macknik and Livingstone, 1998). Similarly, masking specifically disrupts signals responsible for differentiating figure from ground which occur in a later time window (post-100 ms), whereas it has no effect on the early signals (pre-100 ms) which reflect orientation differences of the elements comprising the figure-ground stimuli (Lamme et al., 2002). Because figure-ground segmentation depends on feedback from extrastriate areas to V1 (Lamme et al., 1998), these results suggest that the mask interferes with the feedback processing of the target. Furthermore, only the late, but not the early component of V1 responses correlates with behavioral reports of stimulus visibility, with higher neural activation observed for seen stimuli (Bridgeman, 1980; Lamme et al., 2000; Supèr et al., 2001).

Electroencephalography recordings with humans reveal similar findings to those of single-cell recording studies. Fahrenfort et al. (2007) presented texture-defined target stimuli and partially overlapping backward masks to human observers during a target detection task. In trials where the target was seen (i.e., not masked), three stages of visual processing indicated the presence of a figure (target) against a background. The first stage, which occurred during the first 110 ms after stimulus onset and was apparent in occipito-temporal electrode activity, was interpreted to reflect an initial feedforward activation of the visual system extending into the ventral stream. A second stage, which started at 110 ms after stimulus onset and was apparent in occipital electrodes, was interpreted as reflecting a reactivation of occipital regions due to recurrent feedback. A distinct third stage, which occurred around 200 to 300 ms and was apparent in occipito-temporal electrodes activity, was interpreted as reflecting an additional wave of recurrent activity. Importantly, the authors found that in trials where target was masked (i.e., not seen), both the second and third stages were no longer apparent; only the first stage of processing was unaffected by masking. In other words, the stages indicative of recurrent feedback were affected most, supporting the position that backward masks interrupt feedback processing of the target stimuli in the visual cortex. Together, single cell recording and human neuroimaging studies suggest that backward masking interrupts the later but not the initial neural response; these two stages of processing have been attributed to feedback and feedforward processing, respectively, (Fahrenfort et al., 2007, 2008; Boehler et al., 2008).

With this mechanism of masking in mind let us return briefly to studies that employ overlapping target and mask stimuli to make inferences about feedforward and feedback processes and their contribution to visual awareness. As discussed earlier, stimuli that overlap in space activate identical cells along the visual processing pathway allowing masking to occur due to sensory signal integration at pre-cortical levels. Because these studies typically employ pattern masks, successful discrimination of natural stimuli under ultra-rapid presentation conditions (i.e., target-to-mask SOAs of less than 100 ms) likely indicate poor integration masking rather than scene processing that does not require feedback; that is, fairly high target accuracy even at the shortest SOAs (Bacon-Macé et al., 2005; Walther et al., 2009; Loschky et al., 2010) may indicate that the masks used to interfere with natural scenes are simply

in effectively integrated with the feedforward signal of the target. Indeed, both faces and scenes are most effectively masked by face and scene stimuli, respectively, (Loffler et al., 2005; Loschky et al., 2010), suggesting that other masks (e.g., noise) may be too simple or may not be effectively integrated with the rich feedforward signal from natural images. With these methodological considerations in mind we argue that data obtained using ultra-rapid presentation paradigms do not strictly preclude recurrent processing and thus they cannot be taken as strong evidence against the idea that awareness depends on feedback processes (see also Koivisto et al., 2014a). In fact, as mentioned earlier, some evidence exists to suggest that TMS over the occipital pole might interfere with feedback signals from scene stimuli in a 100 ms and later time window (Camprodon et al., 2010; Koivisto et al., 2011a).

The neural mechanisms behind TMS-induced reduction of stimulus visibility are less clear, due primarily to unknown variation in retinocortical transmission time. If we accept a retinocortical transmission time of 60 ms (Railo and Koivisto, 2012), then the timing of TMS-induced masking is comparable to that of backward visual masking. Thus, the classic post-stimulus TMS effect at 100 ms, like visual masking, could be attributed to interference with feedback processing in the EVC (Corthout et al., 1999a,b; Breitmeyer et al., 2004a; Lamme, 2006b; Breitmeyer, 2007; Allen et al., 2014). Even given some variability in retinocortical transmission time, it has been argued that such a late effect is unlikely to be due to TMS interference with the initial feedforward sweep (Celebrini et al., 1993; Merigan and Maunsell, 1993; Nowak et al., 1995; Schmolesky et al., 1998; Lamme and Roelfsema, 2000) and instead is likely due to interference with feedback processes (Breitmeyer et al., 2004a; Lamme, 2006b; Breitmeyer, 2007). The idea that feedback to EVC is required for visual awareness is supported by joint backward masking and TMS studies. Here in trials where target's visibility is suppressed by a visual mask, if TMS is applied at around 100 ms after the onset of the mask, target stimulus visibility greatly increases while that of the mask is suppressed by TMS. Such unmasking of the target and masking of the mask by TMS has been taken as evidence of recurrent feedback processing in EVC (Amassian et al., 1993; Ro et al., 2003).

To directly test similarities and differences between metacontrast and TMS, Railo and Koivisto (2012) compared masking effects using identical target stimuli with the same group of participants. Both methods impaired subjective target visibility ratings in comparable time windows; the optimal masking SOA was 33 ms in metacontrast and 75 ms in TMS trials. Assuming a retinocortical transmission time of 60 ms, Railo and Koivisto (2012) suggest that TMS interference with visual processing occurs slightly earlier than with metacontrast masks. Interestingly, neither TMS nor metacontrast impaired subjects' ability to simply detect the target in a 2-alternative forced choice location task. These data suggest that while feedback may be necessary for subjective reports of awareness, the intact feedforward signal may be sufficient to enable above chance detection in a forced-choice location task (VanRullen and Koch, 2003; VanRullen, 2007).

Although the timing of the TMS and visual masking effects obtained by Railo and Koivisto (2012) can be accommodated in the common framework of interference with feedback processes in EVC, the fact that the TMS effect occurs slightly earlier (depending on actual retinocortical transmission time) than visual masking raises an alternative view. It has been suggested that TMS in this time window may interrupt (some of the) feedforward in addition to feedback activity (Sack et al., 2009; Koivisto et al., 2011b; de Graaf et al., 2012a, 2014; Miyawaki et al., 2012; Railo and Koivisto, 2012). There are several reasons to consider this hypothesis. The range of SOAs that are lumped into the classical TMS suppression window not only vary from 60 to 140 ms but can extend over tens of milliseconds (Amassian et al., 1989; Corthout et al., 1999a,b, 2002; Kammer et al., 2005b; Sack et al., 2009; Camprodon et al., 2010; de Graaf et al., 2011b, 2012a; Koivisto et al., 2011a,b; Jacobs et al., 2012b; Railo and Koivisto, 2012; Allen et al., 2014; Tapia et al., 2014). This suggests that the earliest effects might be occurring as sensory information is just arriving in EVC and raises a possibility that TMS interferes with multiple visual processes in a wide window (Koivisto et al., 2011b; de Graaf et al., 2014).

The hypothesis that the 100 ms TMS suppression window may reflect interference with (some of the) feedforward in addition to (multiple recurrent) feedback processes (Camprodon et al., 2010; Koivisto et al., 2011b; de Graaf et al., 2012a, 2014; Miyawaki et al., 2012; Railo and Koivisto, 2012) is only now being considered in empirical investigations. Here, paradigms that allow differentiating between feedforward and feedback processes are especially useful. The framework of visual awareness we have adapted for this review specifically states that feedback is required for awareness, while feedforward processes are sufficient to"prime"selective motor responses (Lamme and Roelfsema, 2000; Breitmeyer, 2007). This view is supported by masked priming studies (Klotz and Wolff, 1995; Klotz and Neumann, 1999; Vorberg et al., 2003). Here, a visual masking paradigm is adapted to measure priming by varying the similarity between the target-prime and mask-probe. Responses to the mask-probe are faster when the two stimuli match on a to-be-discriminated feature (e.g., color) as compared to when they do not. Interestingly, the target-prime affects responses to the mask-probe both in trials when the target-prime is seen and when its visibility is suppressed by the subsequent mask-probe. We and others (e.g., Lamme and Roelfsema, 2000; Chen and Treisman, 2009; Tapia and Breitmeyer, 2011) have interpreted this pattern of results as indicating that the feedforward sweep of activity elicited by the target-prime is sufficient to produce priming even though the target-prime fails to reach awareness due to the interruption of its feedback processes by the maskprobe. Masked priming effects have been consistently reported for various stimuli in a metacontrast paradigm (e.g., Klotz and Neumann, 1999; Schmidt, 2002; Vorberg et al., 2003; Breitmeyer et al., 2004b, 2005, 2007; Enns and Oriet, 2007; Breitmeyer and Hanif, 2008; Kentridge et al., 2008; Ro et al., 2009; Schmidt and Schmidt, 2010; Tapia et al., 2010, 2011, 2013; Tapia and Breitmeyer, 2011).

If post-stimulus TMS over EVC at around 100 ms interrupts solely feedback processes, we would also expect similar (i.e., feedforward-supported) effects in TMS paradigms; that is, we would expect to obtain priming from a TMS-masked target. Alternatively, if post-stimulus TMS also interferes with (some

of the) feedforward processes during this time window, performance on tasks that rely on feedforward activity should be impaired; that is, we would expect priming to be diminished when the target is suppressed by TMS. Only a handful of studies have investigated this hypothesis to date, but they all show that priming is either reduced (Sack et al., 2009; Railo et al., 2012) or entirely absent (Jacobs et al., 2012a; Persuh and Ro, 2013) at TMS SOAs ranging from 60 to 100 ms post-stimulus, suggesting that TMS in this range might be interfering with feedforward processes.

It is difficult to draw clear conclusions about feedforward and feedback processes from these studies not only because there are only a few of them, but also because the experimental procedures used differ among the studies. TMS stimulation parameters (e.g., intensity, shape of coil) should be systematically explored in the priming task as they have been shown to differentially affect performance in other paradigms (Beckers and Hömberg, 1991; Kammer et al., 2005a; Hallett, 2007; de Graaf et al., 2011a; Reichenbach et al., 2011). Additionally, differences between behavioral tasks should also be examined. For example, it is not known whether priming effects differ between paradigms in which TMS is applied either between presentation of target-prime and maskprobe (Persuh and Ro, 2013), during (Railo et al., 2012) or at the end of this stimulus sequence (Sack et al., 2009; Jacobs et al., 2012a; Railo et al., 2012). Moreover, assessing target-prime visibility during the same (e.g., Persuh and Ro, 2013) or separate sets of trials (e.g., Jacobs et al., 2012a) might also lead to different results (Lin and Murray, 2014). A combination of any of these factors might influence when and if priming with TMS is observed. Nonetheless, we believe that this line of inquiry will yield significant insights into feedforward and feedback contributions to visual awareness once these experimental procedures are fine-tuned.

In summary, although it has been previously argued that the 100 ms post-stimulus window reflects TMS interference with feedback processing in EVC, data from the newest studies suggest that the classical TMS window of suppression might in fact encompass both the feedforward and feedback processes, with lower SOAs tapping into the feedforward and later SOAs tapping into the feedback activity (Koivisto et al., 2011b; de Graaf et al., 2012a, 2014; Miyawaki et al., 2012). Hence, while there is ample evidence to suggest that metacontrast and TMS interfere with feedback processes, future research should inform the effects of TMS on feedforward activity as well as how and why this method of masking differs from that of metacontrast.

## **SUPPRESSING STIMULI FROM AWARENESS: FORWARD MASKING**

#### **VISUAL MASKING**

Given that feedforward processes precede and initiate feedback processes, it should also be possible to interfere with the initial feedforward sweep of activity and reduce stimulus visibility. Paracontrast is thought to be one such example. It is a specific case of forward masking that, like metacontrast masking, involves spatially non-overlapping target and mask stimuli. Unlike in metacontrast masking, here the mask stimulus precedes the target, and is thus thought to interfere with

feedforward processing of the target. As in backward masking, varying the SOA between the target and the mask reveals distinct windows of visual suppression (**Figure 2**). In paracontrast strongest suppression of target visibility is obtained at SOAs around −170 to −100 ms and −10 ms (Cavonius and Reeves, 1983; Kaitz et al., 1985; Öˇgmen et al., 2003; Breitmeyer et al., 2006).

#### **TRANSCRANIAL MAGNETIC STIMULATION**

As in forward visual masking paradigms, TMS can also be applied before the onset of a visual stimulus. Pre-stimulus TMS impairs target visibility at a range of SOAs spanning from -80 to -10 ms (Corthout et al., 1999b, 2002, 2003; Laycock et al., 2007; de Graaf et al., 2011a; Jacobs et al., 2012b, 2014). Interestingly, pre-stimulus TMS at -20 ms shows retinotopically specific effects, that is reducing stimulus visibility only in specific locations, while TMS at -50 ms exhibits a global reduction of stimulus visibility regardless of where the stimulus was presented (Jacobs et al., 2014). Poststimulus TMS effects shortly after the onset of the target (SOA effects at 0, 10, 20–60 ms; Beckers and Hömberg, 1991; Paulus et al., 1999; Corthout et al., 1999a,b, 2002, 2003) have also been reported (**Figure 2**).

#### **NEURAL MECHANISMS**

Behavioral and especially neural effects of paracontrast are much less studied than those of metacontrast. This is likely due in part to the fact that paracontrast produces much weaker suppression than metacontrast (Alpern, 1953; Weisstein, 1972). Single-cell recordings in primate V1 reveal that paracontrast reduces the initial neural activity associated with the onset response of the target and slightly reduces subsequent excitatory after-discharge due to target offset (Macknik and Livingstone, 1998). Thus, it has been proposed that paracontrast masks interfere primarily with the incoming feedforward activity of the target (Breitmeyer and Öˇgmen, 2006; Breitmeyer, 2007).

The neural consequence of the pre- and early post-stimulus TMS reduction of visibility is under question. Post-stimulus effects of SOAs up to 60 ms have been difficult to replicate consistently (Kammer, 2007a,b). Moreover, the SOAs of optimal suppression in pre-stimulus TMS studies, like in paracontrast, are quite varied (Beckers and Hömberg, 1991; Paulus et al., 1999; Corthout et al., 1999a,b, 2002, 2003; Laycock et al., 2007; de Graaf et al., 2011a; Jacobs et al., 2012b, 2014). In fact, some pre-stimulus TMS effects have been attributed primarily to eye-blink artifacts (Corthout et al., 1999b, 2011) raising questions as to whether it is a neural effect at all. More recent work, however, has shown that in trials where no eye-blinks occurred stimulus visibility was still significantly impaired with pre-stimulus TMS at −80 and −60 ms SOAs (Jacobs et al., 2012b, 2014).

Even with the masking effect localized to the cortex, the relationship between the pre-stimulus TMS effects and paraand metacontrast masking is still under debate. The early poststimulus TMS effects at SOAs up to 60 ms have been explicitly attributed to TMS interruption of the feedforward sweep (Corthout et al., 1999a). However, given retinocortical transmission time, TMS applied earlier than that would presumably affect the visual cortex before the visual signal arrives there. Hence, it has been suggested that the pre-stimulus and early post stimulus TMS effects reflect changes in pre-stimulus brain activity, such as changes in alpha power or phase, rather than a direct effect on the evoked activity of the target (de Graaf et al., 2011a; Jacobs et al., 2012b, 2014). Such suggestions also raise the possibility that paracontrast masks also change brain states, such as those related to alpha power and phase. Moreover, although paracontrast and pre-stimulus and early post stimulus effects on stimulus visibility (when TMS SOAs have been adjusted for retinocortical delay) overlap in time, they do not align as nicely as with backward visual and TMS masks (see **Figure 2**). Of course, these conclusions critically depend on actual retinocortical transmission time. If one accepts that the backward masking effects of TMS and visual masks are comparable, then it is tempting to conclude that paracontrast and forward masking effects with TMS tap into the same visual mechanism of processing, this time interference with the feedforward signal (Corthout et al., 1999a,b; Breitmeyer et al., 2004a; Breitmeyer, 2007). As we have argued above, the post-stimulus TMS window likely reflects TMS interference with the incoming feedforward signal. If pre-stimulus TMS also affects the feedforward sweep, it may do this indirectly, by altering the brain state. Clearly, more research is needed on this issue.

Transcranial magnetic stimulation lends itself nicely to investigating temporal dynamics of visual processing because, unlike visual masks, it can be used to selectively interrupt activity in distinct visual areas (e.g., Pitcher et al., 2009). The timing of visual suppression effects when different regions are stimulated can inform us about the timing of feedforward and feedback processes between the stimulated areas. In a seminal study Pascual-Leone and Walsh (2001) provided evidence that feedback to V1 from V5/MT+ was necessary for the perception of motion. TMS over V5/MT+ produces the perception of moving spots of light (i.e., moving phosphenes). However, when TMS was applied to EVC between 5 and 45 ms after it was applied to V5/MT+, the perception of motion significantly decreased. Because TMS over EVC only interfered with motion perception subsequent to the V5/MT+ stimulation (motion suppression peaked at 25 ms post- V5/MT+ TMS), the result has been interpreted as TMS interference with feedback coming from V5/MT+ to EVC. Similar conclusions have been reached by others; that is, the perception of motion requires a feedforward and feedback exchange of activity between EVC and V5/MT+ (e.g., Silvanto et al., 2005a,b; Laycock et al., 2007; Koivisto et al., 2010).

This exchange of activity between EVC and extrastriate cortex is not limited to V5/MT+. Koivisto et al. (2011a) applied TMS to EVC and lateral occipital cortex (LOC) while participants categorized whether a briefly flashed image contained an animal or not, and rated their subjective visibility. TMS over EVC impaired categorization speed and subjective stimulus visibility ratings at SOAs from 90 to 210 ms after stimulus onset, whereas TMS over LOC affected categorization speed and subjective ratings only at the post-stimulus SOA of 150 ms. Such a pattern of data is consistent with the hypothesis that recurrent interactions between EVC and later regions, in this case LOC, are necessary for visual awareness, with the first suppression intervals produced by EVC and LOC stimulation corresponding to interference with feedforward signals and the later suppression window (at 210 ms) corresponding to interference with feedback from LOC into EVC.

TMS studies discussed thus far suggest that at least one source of feedback to EVC, which is needed for awareness, comes from extrastriate regions. However, a number of other brain areas have been implicated in awareness that might also act as a source of feedback to EVC. It is commonly argued that awareness arises due to interactions between fronto-parietal and occipitotemporal areas (e.g., for a review see Beck et al., 2001; Driver and Vuilleumier, 2001; Rees, 2001; Crick and Koch, 2003; Baars, 2005; Dehaene et al., 2006; Tononi and Koch, 2008). Studies with patients (e.g., unilateral neglect) and with TMS suggest that fronto-parietal regions are not incidental to awareness but critical to it (Driver and Vuilleumier, 2001; Turatto et al., 2004; Beck et al., 2006). Nonetheless, the exact dynamics of interactions between fronto-parietal and occipito-temporal regions are still under investigation.

One candidate for the source of feedback to EVC is the parietal lobe because of its abundant feedforward and feedback connections with occipito-temporal cortex (Felleman and Van Essen, 1991; Webster et al., 1994; Lewis and Van Essen, 2000). To test

this possibility, Koivisto et al. (2014b) applied TMS to the EVC and the intraparietal sulcus (IPS). They found that shape visibility was impaired by EVC TMS at post-stimulus SOAs of 60, 90, and 120 ms, while TMS over IPS disrupted performance in the same task only at an SOA of 90 ms (Koivisto et al., 2014b). These results complement those from the authors' earlier study where TMS over LOC impaired visibility in a relatively late time window (150 ms; Koivisto et al., 2011a), except that the IPS effect occurred earlier. In other words, both studies implicate recurrent interactions between EVC and higher-level areas, point to the importance of both the feedforward and feedback signals to awareness, and begin to address when and where that feedback is coming from.

Recently it was shown that phosphene sensations can be elicited with TMS applied over the parietal cortex, i.e., regions corresponding to the P3/P4 electrode sites (Marzi et al., 2009), and that these percepts are similar albeit less vivid than occipitally induced phosphenes (Fried et al., 2011; Mazzi et al., in press). Additionally, when TMS is applied to parietal areas that elicit phosphenes, occipital cortex exhibits activity 20 to 40 ms after parietal stimulation (Parks et al., 2013 and in preparation). The occipital activity is in line with the existence of feedback connections between parietal and occipito-temporal regions (Felleman and Van Essen, 1991; Webster et al., 1994; Lewis and Van Essen, 2000) as well as other data showing that TMS to parietal cortex can modulate activity in EVC (Ruff et al., 2008; Silvanto et al., 2009). What is not yet clear is whether the experience of parietal phosphenes (i.e., phenomenal awareness) requires the interplay of activity between parietal and EVC. Indeed, it is currently unknown whether occipital and parietal phosphenes arise from the same (Fried et al., 2011) or different neural mechanism given that parietally induced phosphenes have been reported in the blind visual field of two hemianopic patients (Mazzi et al., in press; see also Tapia et al., 2014).

The frontal lobes have also been implicated in visual awareness. The general "frame-and-fill" approach (Bullier, 2001; Chen et al., 2007) to visual processing posits that magnocellular channels project a rapid but coarse feedforward representation of the stimulus to higher cortical areas in the dorsal pathway and to the prefrontal cortex (Bullier, 2001; Peyrin et al., 2010; Tapia and Breitmeyer, 2011). Then, projections from these areas, activated by the initial magnocellular signals, potentiate or "frame" the processing along the slower ventral parvocellular pathway that carries the "fill" information, e.g., detailed form and color that are necessary for constructing a high-resolution representation of a visual object (Chen et al., 2007; Tapia and Breitmeyer, 2011; Breitmeyer, 2014). A specific version (Bar, 2003; Bar et al., 2006; Kveraga et al., 2007) of this"frame-and-fill"approach additionally posits that the prefrontal cortex projects directly to and modulates processing in the inferotemporal cortex (IT).

Finally, microstimulation of or TMS to frontal eye fields (FEF) have been shown to modulate neural responses in striate and extrastriate visual areas (Moore and Armstrong, 2003; Ruff et al., 2006, 2008; Silvanto et al., 2006; Taylor et al., 2007). The FEFs are considered a part of the dorsal attention network (Corbetta and Shulman, 2002). Given the link between attention and awareness and the connectivity of the FEF, these regions seem a reasonable candidate for awareness-related feedback to EVC. However, this feedback is yet to be shown to be critical to awareness. Given this complex neural network, the exact mechanism by which frontoparietal regions generate feedback to EVC and contribute to visual awareness is yet to be determined and all of these candidates need to be probed further.

#### **THE NCCs OF VISUAL AWARENESS**

Awareness is an emergent property of the brain and arises amidst other equally complex processes. Therefore, trying to pinpoint the neural correlate of consciousness (NCC) or more specifically of visual awareness may be misleading. For instance, there may be prerequisite conditions for the "true" NCC (or NCC-proper) to emerge as well as events that consistently arise as a result of awareness. The distinctions among these three NCCs have been discussed eloquently and in detail elsewhere (e.g., Aru et al., 2012; de Graaf et al., 2012b) and are inherently difficult to tease apart. Here we concentrate on the prerequisites of consciousness, or NCC-prerequisites, as some of the reported TMS effects are likely to fall into this category. Prerequisites include conditions set prior to the onset of a stimulus that by themselves cannot elicit a percept of that stimulus. Thus, as argued above, the pre-stimulus TMS effects (and possibly some of the early post stimulus effects) very likely achieve their effects by modulating brain states that influence how subsequent sensory information will be processed (Thut et al., 2006; Gilbert and Sigman, 2007; Mathewson et al., 2009; Summerfield and Egner, 2009).

Ongoing alpha oscillations (8-12 Hz) not only reflect a brain state that has been implicated in visual awareness, but they are also modulated by TMS. Visibility of a masked target in metacontrast fluctuates as a function of power and phase of occipito-temporal EEG alpha (Mathewson et al., 2009, 2010, 2012). Increased prestimulus alpha power has been associated with lower detection rates using a wide range of stimuli (e.g., Ergenoglu et al., 2004; Palva et al.,2005; Romei et al.,2008; van Dijk et al.,2008;Wyart and Tallon-Baudry, 2008; Busch et al., 2009; Mathewson et al., 2009). These changes in power have been linked to changes in attentional state (Worden et al., 2000; Thut et al., 2006; Mathewson

et al., 2014). Others have argued that alpha power is an indication of the general level of excitability of the visual cortex, with high alpha power representing a general inhibition of ongoing processing (Klimesch et al., 2007; Mathewson et al., 2011). Consistent with this idea, Romei et al. (2008) have shown that identical TMS pulses over visual cortex are less likely to elicit visual phosphenes when alpha power over posterior cortical areas is high. Together these results suggest that the power and phase of alpha oscillations are indicative of a brain state that influences whether a subsequent stimulus will reach awareness.

Interestingly, even when alpha power is high there appear to be mechanisms that modulate awareness. Specifically, stimulus visibility varies as a function of alpha phase and is observed only under high alpha conditions (Mathewson et al., 2009, 2011, 2012). For instance, in metacontrast paradigm, depending on whether the target appears during a peak or trough in the alpha cycle it will be more or less likely to be detected (Mathewson et al., 2009, 2010, 2012). Similarly, alpha phase predicts whether or not participants will experience a TMS induced occipital phosphene (Dugué et al., 2011). Moreover, it is possible to induce alpha with either periodic visual stimuli (Mathewson et al., 2010, 2012; de Graaf et al., 2013; Spaak et al., 2014) or repetitive TMS (rTMS) over parietal cortex (Thut et al., 2011; Jaegle and Ro, 2014) and produce phasedependent changes in stimulus detection performance. Romei et al. (2010) stimulated occipital and parietal cortex with rTMS at alpha (10 Hz), theta (5 Hz), and beta (20 Hz) frequency and found that stimulation only at the alpha frequency significantly correlated with stimulus visibility. Together these findings suggest that alpha is causally involved in shaping perception and, hence, both its phase and power reflect the brain state that can be labeled as NCC-prerequisites. Interestingly, activity in frontoparietal attention areas has been shown to correlate with the posterior alpha that predicts detection of visual stimuli (Mathewson et al., 2014), suggesting once again that pre-stimulus alpha may reflect attentional states.

Given the timing of alpha and optimal backwards masking SOAs, it is possible that alpha oscillations impact both the target and mask. Specifically, optimal backward masking SOAs with visual and TMS masks (when retinocortical transmission time is accounted for) fall within the half cycle of alpha. Indeed, the timing is such that if the target appears in an inhibitory phase of alpha, the mask will fall in the excitatory phase, potentially increasing the chances that the feedforward signal from the mask will interfere with the target (see **Figure 3**). Of course, it is equally likely that the target appears during the excitatory phase of alpha, resulting in poorer masking and better detection of the target. Such a mechanism could explain why metacontrast masking rarely occurs on 100% of trials. Alpha phase alone cannot account for backward masking more generally, however, because alpha power is low when a study participant is fully engaged in the task. In this case, phase has little to no effect on stimulus visibility, yet metacontrast masking still occurs during low alpha power states, albeit at a reduced rate compared to trials occurring during high alpha power states (Mathewson et al., 2009). Further research is needed to understand the relationship between alpha power and phase, and paracontrast and pre-stimulus TMS effects. Because TMS pulses in a range of modalities have been shown to reset

Finally, with regard to the NCCs we note that most data collected on consciousness are just that, correlates. We still do not know what conditions give rise to the experience of consciousness. However, we argue that the visual masking and TMS literature reviewed here suggests that feedforward and feedback signals are not only correlated with awareness, but necessary for it. We are not suggesting, however, that such signals are sufficient for awareness. We do not suppose, for instance, that any set of neurons wired in a recurrent fashion should result in awareness (Herzog et al., 2007). Indeed, feedback likely confers some other advantage such as allowing for integration of information (Oizumi et al., 2014) or resonance between top-down expectations and bottom-up input (Grossberg, 2013).

## **CONCLUSION**

To summarize, visual masking paradigms and TMS to EVC affect stimulus visibility in distinct time windows. Backward visual (metacontrast) and post-stimulus TMS masking effects have been thought to occur due to interference with feedback processes that are required for visual awareness, while forward visual (paracontrast) and pre-stimulus as well as early post-stimulus TMS effects have been proposed to reflect interference with the initial feedforward activity. Recent empirical evidence, however, suggests that the parallels between metacontrast and TMS masking might not be as straightforward as previously thought. While metacontrast occurs due to the mask interfering with feedback processes of the target, post-stimulus TMS possibly interrupts not only the feedback but also (some of the) feedforward processes. Additionally, forward masking (paracontrast) was thought to reflect interference with feedforward processing, but recent work looking at alpha oscillations raises the possibility that pre- and early post-stimulus TMS influence stimulus visibility by affecting the brain state prior to target onset. Future research should also inform the involvement of feedforward activity in post-stimulus TMS masking window and elucidate how and why this method of masking differs from that of metacontrast. Additionally, assessing the exact retinocortical transmission time for visual stimuli and specific time windows of suppression with TMS and visual masks should help disentangle feedforward and feedback contributions to visual processing in these paradigms. Finally, exploring the contributions of oscillatory alpha power and phase will help establish the necessary and sufficient conditions to visual awareness by parsing out prerequisites from NCC-proper.

## **ACKNOWLEDGMENTS**

We thank Bruno Breitmeyer and Christianne Jacobs for sharing their data for our figures. This work was supported by NIH R01EY022605 to Diane M. Beck.

#### **REFERENCES**

Ahissar, M., and Hochstein, S. (2004). The reverse hierarchy theory of visual perceptual learning. *Trends Cogn. Sci.* 8, 457–464. doi: 10.1016/j.tics.2004.08.011

Ahissar, M., Nahum, M., Nelken, I., and Hochstein, S. (2009). Reverse hierarchies and sensory learning Reverse hierarchies and sensory learning. *Philos. Trans. R. Soc. Lond. B Biol. Sci.* 364, 285–299. doi: 10.1098/rstb.2008.0253


performance in humans. *Brain Res. Cogn. Brain Res.* 20, 376–383. doi: 10.1016/j.cogbrainres.2004.03.009


EEG alpha and fluctuations in visual awareness with rhythmic visual stimulation. *J. Cogn. Neurosci.* 24, 2321–2333. doi: 10.1162/jocn\_a\_00288


in early visual cortex: a TMS study. *J. Cogn. Neurosci.* 24, 819–829. doi: 10.1162/jocn\_a\_00172


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 13 June 2014; accepted: 26 September 2014; published online: 21 October 2014.*

*Citation: Tapia E and Beck DM (2014) Probing feedforward and feedback contributions to awareness with visual masking and transcranial magnetic stimulation. Front. Psychol. 5:1173. doi: 10.3389/fpsyg.2014.01173*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Tapia and Beck. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Using brain stimulation to disentangle neural correlates of conscious vision

## *Tom A. de Graaf 1,2 \* and Alexander T. Sack1,2*

*<sup>1</sup> Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, Netherlands <sup>2</sup> Maastricht Brain Imaging Centre, Maastricht, Netherlands*

#### *Edited by:*

*Jaan Aru, University of Tartu, Estonia*

#### *Reviewed by:*

*Tony Ro, City University of New York, USA Evelina Tapia, University of Illinois Urbana-Champaign, USA*

#### *\*Correspondence:*

*Tom A. de Graaf, Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 EV, Maastricht, Netherlands e-mail: tom.degraaf@ maastrichtuniversity.nl*

Research into the neural correlates of consciousness (NCCs) has blossomed, due to the advent of new and increasingly sophisticated brain research tools. Neuroimaging has uncovered a variety of brain processes that relate to conscious perception, obtained in a range of experimental paradigms. But methods such as functional magnetic resonance imaging or electroencephalography do not always afford inference on the functional role these brain processes play in conscious vision. Such empirical NCCs could reflect neural prerequisites, neural consequences, or neural substrates of a conscious experience. Here, we take a closer look at the use of non-invasive brain stimulation (NIBS) techniques in this context. We discuss and review how NIBS methodology can enlighten our understanding of brain mechanisms underlying conscious vision by disentangling the empirical NCCs.

**Keywords: NIBS,TMS,TES, tDCS, visual awareness, consciousness, NCC**

#### **INTRODUCTION**

The search for neural correlates of consciousness (NCCs) continues. While "consciousness" has been a philosophical and scientific topic of interest throughout the ages, the surging development of brain research technology has caused something of a renaissance in the last quarter century. Quick advancements in functional magnetic resonance imaging (fMRI) and electro-/magnetoencephalography (EEG/MEG), combined with seminal contributions from high-profile pioneers (e.g., Crick and Koch, 1990), provided the NCC research program in humans with quite some momentum (Dehaene and Naccache, 2001; Rees et al., 2002a; Crick and Koch, 2003; Koch, 2004), while groundbreaking animal experiments were performed as well (Cowey and Stoerig, 1995; Leopold and Logothetis, 1996). Today, the neuronal mechanisms underlying "visual awareness," "conscious perception," or "subjective experiences," are regular topics in empirical research.

## **EMPIRICAL NCCs**

#### **CONSCIOUSNESS**

No article on "consciousness" is likely to be very meaningful without a clear delineation of what mental faculties exactly are referred to. We previously outlined our preferred rough taxonomy of types of consciousness (de Graaf et al., 2012b). It included (1) self-awareness, (2) higher-order consciousness, (3) "medical awareness" or state-consciousness, and (4) "consciousness-asexperience" or content-consciousness.

*Self-awareness* is the overarching concept of a continuous and controlling self, a being that is defined by the contrast to surroundings and other beings. Research clustered under self-awareness could include such topics as self-recognition, agency, and awareness and situation of the persona inside the body.

*Higher-order consciousness* in our schema is the somewhat folk psychological conception of consciousness, where the abilities to think, reason, and reflect are crucial. It involves typically human faculties such as the realization of past and future, the ability to "think about thinking," and is likely analogous to, for example, reflective consciousness (e.g., Edelman and Tononi, 2001).

*Medical awareness*, which for simplification we may also refer to as the more common "state consciousness," is a conception of consciousness as a certain state of being. Patients in a coma or under anesthesia may not be in a conscious state, people under the influence of drugs may be in an alternative conscious state. It is required to be in at least some minimal state of consciousness to achieve:

*Consciousness as experience,* which for simplification we may refer to as the more common "content consciousness." At any moment in time, provided we are awake (or at least dreaming), we have experiences. They include phenomenal properties (the "what-it-is-like" of the experience) and psychological or "access" properties (the abilities to report, remember, or act on the experienced information; Chalmers, 1996; Block, 2005).

#### **NCC PARADIGMS**

Early on, pioneering consciousness researcher Baars (1989) pointed out that NCCs can be obtained by contrasting conditions with conscious experience to conditions without conscious experience (the *contrastive method*, e.g., Aru et al., 2012). Using neuroimaging paradigms, such as fMRI or EEG, one can contrast these two experimental conditions to isolate the brain mechanisms specific to the conscious condition. Ideally the conscious condition and the non-conscious condition differ from each other as little as possible in terms of stimulation parameters. Over the years, a range of paradigms has been developed for NCC research that

elements are rotated. The presence or absence of the conscious percept can be correlated to brain activity. Another form of illusion is filling-in, in which a constant peripheral stimulus is sometimes perceived and other times not. *Multistable paradigms*: Constant visual stimulation leads to a changing conscious percept. A well-known example is binocular rivalry, in which both eyes receive incompatible images and conscious perception fluctuates between the two stimuli. Some brain processes will covary with perception, others will not. *ON–OFF paradigms*: Paradigms in which sometimes a stimulus is perceived, and other times not. It thus involves presence vs

allows useful contrasts with no, or minimal, changes in stimulus parameters. Outlined in **Figure 1**, we have developed informally our taxonomy of NCC research paradigms (also presented in de Graaf and Sack, in press). We make no claim to either exhaustiveness or originality/priority in this regard (see e.g., Kim and Blake, 2005), it is simply a grouping that we have found useful to maintain an overview of the many paradigms in NCC research.

One main division is between paradigms to obtain NCCs and paradigms that research neural correlates of unconsciousness (NCUs). The latter are not always referred to as such, but investigations of brain activity elicited by inputs that do not make it to consciousness are clearly valuable in the greater scheme of NCC research. For example, patients with (often right) parietal damage may fail to consciously see (report) stimuli in the opposite visual field ("neglect"), especially when bilateral stimuli are presented ("extinction"). Yet brain imaging studies uncovered activity in early and extrastriate visual regions, in response to these unseen stimuli (Rees et al., 2000, 2002b; Vuilleumier et al., 2001). Goebel et al. (2001) studied fMRI activation in extrastriate regions in blindsight patients. Blindsight is another neuropsychological condition with relevance to consciousness, since in this condition with

OFF condition. The same stimuli and setup can be useful for NCU studies, if brain activity is contrasted in the OFF condition vs rest. In other words; which brain processes still obtain if a stimulus is presented but not consciously perceived? *Continuous flash suppression*: A variant of the binocular rivalry paradigm, in which conscious perception is heavily biased towards one eye through repeated salient stimulation of that eye, while the second eye receives a weaker stimulus. That weaker stimulus of interest is thus suppressed for prolonged periods of time, allowing analysis of brain processes nevertheless responding to it.

damaged (connections to) primary visual cortex, patients can correctly report on ("guess") various features of stimuli that are not consciously perceived (Weiskrantz, 2009). One patient studied by Goebel et al. (2001) experienced conscious motion perception in only one visual hemifield, even though bilateral hMT/V5 cortices were intact. Interestingly, activity in hMT/V5 in both hemispheres was nearly identical with respect to BOLD responses to contralateral visual stimulation. In other words, sustained hMT/V5 BOLD activity did not seem to reflect the presence or absence of visual awareness.

Neural correlates of unconscious processing can also be studied in fully intact brains. In one clever fMRI experiment house and face stimuli were presented either consciously (congruent dichoptic stimulation: e.g., a green house on red background presented to both eyes) or not consciously (incongruent dichoptic stimulation: e.g., a green house on red background in one eye and a red house on green background in the other eye "canceled each other out" at the binocular level). "Face areas" and "place areas" of the brain still responded (although to a lesser extent) to completely invisible pictures (Moutoussis and Zeki, 2002). As a second example, an influential study (Dehaene et al., 2001) could show that even words are processed in extrastriate regions when they are not consciously perceived due to visual masking. In some extrastriate regions, the effects were moreover case-independent. Collectively, these experiments seem to demonstrate that activation in specialized higher-order visual regions is not in itself sufficient for conscious perception. Today, an increasingly popular paradigm that can be used to study NCUs is continuous flash suppression (CFS; Tsuchiya and Koch, 2005). This useful variation to the classical binocular paradigm, in which a stimulus in one eye is suppressed by salient flashing patterns of stimulation in the other eye, can be implemented to suppress visual stimuli for very long durations.

A second main distinction that may be useful is between "strong" and "weak" paradigms. "Strong" paradigms in this context allow changes in conscious percept, or variations in conscious percept, without *any* changes in stimulation parameters. "Weak" paradigms, in contrast, implement small changes in stimulus parameters to determine conscious content. This difference is easily understood in the context of "ON–OFF paradigms," where stimuli are consciously perceived (ON) or not (OFF). For example in visual masking, experimenters can determine, through the timing between targets and masks, whether targets will be perceived or not (Breitmeyer and Ogmen, 2006). Since this involves a change in stimulus parameters, it is a "weak" ON–OFF paradigm. Alternatively, in a "strong" ON–OFF implementation the stimulus parameters could be fixed at some threshold level, relying on spontaneous neuropsychological fluctuations to lead to a conscious percept on some trials (ON) but not on other trials (OFF).

"Strong" and "weak" refer only to the level of isolation of brain processes in relation to visual awareness, not the scientific value of the paradigm. While constant stimulus parameters allow a "cleaner" isolation of the brain processes related to conscious perception, weak paradigms afford the experimenter control and certainty about the presence, absence, or contents of visual awareness. Both thus have their advantages and disadvantages, and appropriate applications depend on experimental question and brain imaging setup.

All of the various paradigms in **Figure 1** could yield a number of brain processes correlated to conscious vision, including BOLD activations, fMRI connectivity patterns, EEG/MEG event-related potentials, changes in oscillatory power or phase coherence, and so on. All such brain mechanisms would, by definition, correlate to conscious contents. And they would therefore, in the literal sense, be NCCs: neural processes that correlate to consciousness. But in the conceptual and philosophical domain, "NCC" can have quite a specific and involved meaning. So from now on, we refer to such experimental findings strictly as *empirical NCCs.*

#### **CORRELATES AND "TRUE CORRELATES"**

After all, another, or perhaps "true," meaning of NCC's has traditionally been the *actual* brain mechanisms responsible for conscious perception. Definitions abound, but an influential definition of a neural correlate of consciousness comes from Chalmers (Chalmers, 2000):

*An NCC is a minimal neural system N suchthatthere is a mapping from states of N to states of consciousness, where a given state of* *N is sufficient, under conditions C, for the corresponding state of consciousness. (Chalmers, 2000, p. 31)*

Clearly,"NCC"here is much more refined and constrained than the"empirical NCCs"obtained in neuroimaging research using the contrastive method.

This realization has quite a long history, as pointed out by Miller (2007, p. 162). For instance, Crick (1994) noted: "it does not follow that these particular neurons are the real seat of awareness. They may by their firing, influence other neurons... that are the *true* correlates of awareness" (Crick, 1994, p. 218). Logothetis (1998, p. 541) asked: "Do neurons responding only when a stimulus is perceived actually mediate the conscious experience of this stimulus?" He pointed out that, although his data favored such an interpretation, they "cannot prove it unequivocally" (Logothetis, 1998). There have been others (e.g., Revonsuo, 2000) who noted what Miller (2001, 2007) calls the "constitution/correlation problem;" brain processes that correlate to conscious perception may not necessarily be constituent of that conscious experience. Koch, lastly, points out that it makes sense to distinguish "core NCCs" from "total NCCs," where core NCCs are responsible for the contents of conscious experience, whereas the total NCC reflects the core NCC plus all enablingfactors and is thus required as a whole to obtain a particular conscious experience (Koch, 2004, as discussed in Block, 2005).

#### **THREE ROLES FOR EMPIRICAL NCCs**

In fact, increasing numbers of philosophical (Revonsuo, 2001; Noë and Thompson, 2004; Block, 2005; Miller, 2007; Hohwy, 2009; Neisser, 2012) and empirical researchers (Miller, 2001, 2007; Koch, 2004; Bachmann, 2009; Melloni et al., 2011; Aru et al., 2012; de Graaf et al., 2012b; Kanai and Tsuchiya, 2012; Sergent and Naccache, 2012) have been coming to the conclusion that empirical NCCs are only part of the way there. To make this explicit, as a prelude to outlines for future research opportunities, two similar review papers (Aru et al., 2012; de Graaf et al., 2012b) recently focused on the three fundamentally distinct functional roles that any (part of an) empirical NCC resulting from the contrastive method could reflect. They are *neural prerequisites, neural consequences,* or *neural substrates* of conscious experience.

#### **NEURAL SUBSTRATES**

Neural substrates of a particular conscious experience are the brain events that *directly* caused (epiphenomenalism), reflected (dualism), or were identical with (materialism) the phenomenal experience in our experiment. They were both necessary and sufficient. "Sufficient" in the sense of Chalmer's definition: only these brain events were required for the experience and nothing more was needed. They were "necessary" only in our current empirical situation (imagine any concrete NCC neuroimaging experiment), and in a non-philosophical sense ("necessary" is rather a loaded term in the context of consciousness), because without these brain events the experience would not have occurred. Hypothetically, perhaps other brain events could have served as substrates for the same, or a similar experience. But in our experiment, it was these brain processes that instantiated the experience. In the current context, there is no clear difference between what we have called "neural substrates," and the "NCC" of Chalmers, the "real" NCC, "true" NCC, or the "NCC-proper." We refer to it as "neural substrate of conscious experience" to differentiate it from all the other different types of NCCs discussed here, and to remain consistent with our earlier outlines (de Graaf et al., 2012b; de Graaf and Sack, in press).

#### **NEURAL PREREQUISITES**

Neural prerequisites of consciousness are brain events that are necessary for the conscious experience to occur, but not sufficient. They are *empirical* NCCs, since they consistently co-vary with conscious experience. That is because, in the implemented experimental setup, in the real world, the neural substrates do not arise without them. But importantly, if somehow through some hypothetical and counterfactual scenario the neural substrates *did* arise without the prerequisites, the conscious experience would be there and unchanged. To understand this, one might imagine highly advanced brain stimulation techniques targeting specifically and only the neural substrates. Or, somewhat more realistically, a different experimental paradigm could give rise to the same neural substrates of a conscious experience, via a different route and thus through different neural prerequisites. Or perhaps it would be possible to attain an identical conscious experience in a dream, without any external stimuli, leaving out some neural prerequisites that obtained in our experiment. In sum, different chains of brain events might have served as prerequisites to the same neural substrates. But, coming back to our perspective as an empirical researcher, we are talking about *empirical* NCCs that factually *did* result from a concrete brain imaging experiment with a given experimental paradigm. And we are outlining possible functional roles of these empirical NCCs. So, in the empirical situation at hand, some of our empirical NCCs could, and likely would, have functioned as neural prerequisites of the studied conscious experience, rather than being the neural substrates. It is useful to distinguish these functional roles.

Expanding on our previous outline (de Graaf et al., 2012b) and as discussed in detail in de Graaf and Sack (in press), it may be useful to further distinguish two possible "types" of neural prerequisites: *content-invariant* prerequisites, and *content-specific* prerequisites. Content-invariant prerequisites would co-occur with any conscious experience. Much of the findings on brain events enabling a conscious state would pertain to content-invariant prerequisites. For example, connectivity between reticular formation and precuneus might be required for any conscious experience to arise (Silva et al., 2010). Content-invariant prerequisites are interesting, but would be relatively easy to identify by eliciting different contents of consciousness across different NCC paradigms and seeing which empirical NCCs are consistently observed.

Content-specific prerequisites would be much more tricky to dissociate from neural substrates, since they co-occur, by definition, with and only with each occurrence of a particular conscious experience. For example, a particular visual image presented for a sufficiently long duration will consistently lead to a specific conscious experience of it, but also to a (in part) specific cascade of non-conscious feature processing steps in early visual regions. To complicate things further, in reality the distinction between content-specific and content-invariant prerequisites may not be a dichotomy, but something of a continuum. There may be modality-specific prerequisites, feature-specific prerequisites, perhaps even concept-specific prerequisites, and so on.

As a concrete example, some binocular rivalry results have implicated primary visual cortex in conscious vision (Polonsky et al., 2000; Tong and Engel, 2001), and we learn from blindsight patients that this region appears to be crucial for conscious vision (Weiskrantz, 2009). Yet, Crick and Koch (1995) argued that primary visual cortex is unlikely to be part of neural substrates ("true correlates") of consciousness. Primary visual cortex activation may therefore be a neural prerequisite, and perhaps particular processes within it content-specific prerequisites. As Silvanto (2008) argues, these processes may be crucial for conscious vision to arise, even though the conscious experience is localized elsewhere in the brain.

In another example, Beck et al. (2006) reported on the role of parietal cortex in change blindness. Finding that parietal rhythmic transcranial magnetic stimulation (rTMS) slowed reaction times to change and reduced the proportion of detected changes, they concluded *"It is important to note that we are not arguing that the parietal cortex is the neural locus of consciousness, but rather that the functions associated with parietal cortex, such as attention and visual short term memory (VSTM), may be necessary prerequisites to visual awareness"* (Beck et al., 2006, p. 716). They suggested that the functional relevance of parietal cortex should be tested in other paradigms to determine whether this role is general rather than specific to change blindness (in our terminology; whether it is a content-invariant prerequisite). As a clear prelude to discussing the value of non-invasive brain stimulation (NIBS) in the current framework, we may focus on another quote from the Introduction of the same article. After discussing prior neuroimaging work: *"In all these studies, it remains possible that the parietal activity found was a consequence of subjects' awareness and did not play either a necessary or causal role in producing that awareness"* (Beck et al., 2006, p. 712).

#### **NEURAL CONSEQUENCES**

There may be brain events that consistently co-occur with a conscious experience, that are neither necessary nor sufficient for the experience to arise. They are not substrates, and they are not even required for the substrates to arise, so they would – with regards to the phenomenal experience – not be missed. Yet, they are there, because they consistently follow a conscious experience.

Again, consequences can be content-invariant or contentspecific. Content-invariant consequences would be empirical NCCs across the range of contents of consciousness. They could include attention effects, if the simple occurrence of an experience grabs your attention, or response preparation and memory processes (we're evolved to act on and learn from consciously perceived information). Consequences can therefore be – while useless to the neural substrates of a conscious experience – rather useful for the organism. Citing Seth (2009),Aru et al. (2012) point out that, indeed, meaningful neural consequences of conscious experience are a logical consequence of assigning any functionality to conscious perception. As with prerequisites, while contentinvariant consequences could be isolated through variations in NCC paradigms and stimuli, content-specific consequences are more difficult to distinguish from neural substrates. If a picture of a beach elicits in me strong emotional memories of a long-lost friend, stimuli depicting beaches could consistently elicit in me a cascade of brain events that would not only be content-specific, but even participant-specific.

### **DISENTANGLING EMPIRICAL NCCs**

There are thus threefundamentally different roles one might assign to (part of) any empirical NCC. And things may become even more complex, if it turns out that combinations of and interactions between content-specific and content-invariant brain events are responsible for conscious experiences. That would make separations of neural prerequisites and substrates difficult, or even somewhat arbitrary. Therefore, as new evidence continues to inform neurobiological models of consciousness, we should follow the advice of properly thinking through the cascade of brain events that underlies conscious experience, reframing the question as we go (Hohwy, 2009; Feinberg, 2012; Neisser, 2012). This way, theoretical and computational models will become increasingly sophisticated (Dehaene et al., 2003; Dehaene and Changeux, 2011; Oizumi et al., 2014). For the moment, however, research with the contrastive method (Aru et al., 2012), using paradigms such as those outlined in **Figure 1**, continues to specify and increasingly constrain empirical NCCs. There are different strategies to try and disentangle the functional roles of empirical NCCs (Aru et al., 2012; de Graaf et al., 2012b). In the remainder of this article, we focus on the contributions of NIBS in this context.

#### **EMPIRICAL NCCs**

Before continuing on to our review of NIBS as a tool for NCC research, it is useful to provide a very quick and rough overview of some empirical NCCs that have been obtained in neuroimaging research. These form, after all, the "starting point" for brain stimulation experiments to probe functional roles of specific empirical NCCs.

While oversimplified, it seems fair to claim that *early visual cortex* (by which we mean V1, V2, V3), certain *extrastriate visual/temporal cortices*, *parietal cortex*, and *frontal cortex* have been linked to conscious vision. We have above already mentioned important studies demonstrating relations between visual awareness and early visual regions. The relevance of increasingly modular extrastriate cortices in vision in general is uncontroversial, and for example bistable vision paradigms have shown that extrastriate regions such as the fusiform face area (FFA) or the parahippocampal place area (PPA) reflect conscious percept rather than the constant visual input (Tong et al., 1998). Bistable paradigms (Kleinschmidt et al., 1998; Lumer et al., 1998; Lumer and Rees, 1999; Sterzer et al., 2002), but also masking (Dehaene et al., 2001), and other NCC paradigms (Rees et al., 2002a; Rees, 2007) have generally reported frontoparietal activations in fMRI. Frontoparietal activations are stronger to trials in which stimuli are consciously seen versus not seen (e.g., Dehaene et al., 2001; Lau and Passingham, 2006), are time-locked to specifically endogenous perceptual switching (Kleinschmidt et al., 1998; Lumer et al., 1998), and are related to attentional/perceptual effects often mentioned in the context of consciousness such as change blindness (Beck et al., 2001) and attentional blink (Marois et al., 2000). The

following section will describe how NIBS has contributed to our understanding of the functional roles of these empirical NCCs.

#### **NON-INVASIVE BRAIN STIMULATION BASICS OF NIBS**

Non-invasive brain stimulation includes primarily TMS and transcranial electric stimulation (TES), the latter including transcranial direct current stimulation (tDCS) and transcranial alternating current stimulation (tACS).

Transcranial magnetic stimulation involves a strong capacitor linked to a coil. A single TMS pulse with the popular figure-eight coil requires a brief electrical current through the overlapping windings inside the coil, which leads to a short, focal, rapidly changing magnetic field extending perpendicularly from the TMS coil. If the coil is placed tangentially on a human head, this magnetic pulse extends into the brain where it induces an electrical field in neural tissue and ultimately causes action potentials (Wassermann et al., 2008). A single pulse can stimulate neurons at rest, with observable behavioral (motor response; e.g., Barker et al., 1985) or perceptual (phosphenes; e.g., Marg and Rudiak, 1994; Kammer, 1999) consequences. A single pulse administered to a region during processing can disrupt the spatiotemporal organization of regional processing and thus induce behavioral (Pascual-Leone et al., 1991), cognitive (Sack et al., 2002a,b), or perceptual (Amassian et al., 1989) impairments (Pascual-Leone et al., 2000). Multiple TMS pulses in a rhythmic sequence, rTMS, can not only impair function online, but also have effects on cortical excitability outlasting the stimulation protocol (Robertson et al., 2003). More complex and powerful new protocols have been developed [theta-burst stimulation (TBS); Huang et al., 2005], but traditionally simple repetitive stimulation of a cortical region with low frequency (∼1 Hz) has been shown to result in decreased cortical excitability while stimulation at high frequency (∼5 to 20 Hz) has been shown to result in increased cortical excitability (Dayan et al., 2013).

TES can have similar effects on cortical excitability. It involves a power source connected to (minimally) two electrode patches. tDCS refers to a continuous flow of low-intensity electrical current (typically around 1–2 mA) from one electrode patch (anodal) to the other (cathodal). Cathodal stimulation of a brain region hyperpolarizes membranes, ultimately resulting in decreased cortical excitability, while anodal stimulation depolarizes membranes, ultimately resulting in increased cortical excitability (Nitsche and Paulus, 2011; Paulus, 2011). Thus, depending on stimulation parameters both rTMS and tDCS can increase or decrease the excitability, and therefore efficacy of contribution, of any cortical region close enough the surface of the brain (Dayan et al., 2013). That includes a great number of empirical NCCs to target. Another form of TES is tACS. In tACS the electrical current does not flow continuously from one electrode to the other, but instead flows rapidly back and forth between the two, reversing direction at an externally fixed frequency. As we will see below, this can be used to modulate oscillatory brain activity. There are still other implementations of TES (e.g., transcranial random noise stimulation), but these fall outside the scope of the current review. Collectively, TMS and TES can be referred to as NIBS, which has already been, and likely still will be, of great value to NCC research.

#### **THE NIBS CONTRIBUTION**

What is the added value of NIBS, over and above the contrastive method of neuroimaging with the paradigms described above? With neuroimaging, it is difficult to conclude whether or not a region or process is functionally relevant. It remains an open question whether or not, for example in fMRI, a regional BOLD response reflects neural processing that is imperative for the task at hand. For conscious vision, this means it is hard to know whether the conscious percept would have been the same if this particular BOLD response had not obtained. In contrast, manipulating brain activity directly with NIBS as an independent variable, and then evaluating the effects on conscious vision, *does* allow one to draw such conclusions on functional relevance. If a conscious percept is abolished, because a brain region is disrupted, then one way or another the disrupted region was functionally relevant to conscious vision. We do think that advanced neuroimaging paradigms and analyses (e.g., connectivity analysis; Friston et al., 2003; Roebroeck et al., 2005; Friston, 2011) can actually make substantial contributions to separating neural substrates, prerequisites, and consequences (de Graaf et al., 2012b). But NIBS can certainly complement this.

#### **NIBS AND CONSCIOUS VISION**

Since brain stimulation can make a unique contribution to the disentangling of empirical NCCs, it may be useful to gain an overview. In this section we therefore provide an exemplary, though non-exhaustive, review of current literature on how NIBS has contributed to our understanding of brain events underlying conscious vision. It should become clear that most if not all of this work goes beyond the concept of empirical NCCs, revealing functional relevance of brain regions both before and after visual stimulus onset and interactions between regions that influence the quality or quantity of conscious vision.

#### **PHOSPHENES**

The simplest TMS protocol immediately reveals the value of TMS for NCC research. A single TMS pulse, applied over occipital cortex, can actually induce a conscious visual percept, called a *phosphene* (e.g., Marg and Rudiak, 1994; Kammer, 1999). This in itself is interesting, because phosphenes likely involve activity of early visual cortex, such as regions V1/V2/V3 (Thielscher et al., 2010). Unless substantial feedback from these areas to subcortical regions turns out to be paramount to conscious vision, this suggests that many subcortical processing steps elicited by exogenous visual stimulation may not be necessary for conscious experience *per se*. A simultaneous TMS-fMRI study could address the matter of subcortical responses to a perceived phosphene, but this appears to remain an empirical question.

With EEG, however, a recent study did manage to implement a strong ON–OFF paradigm to isolate phosphene-specific responses across the (surface of the) brain. Taylor et al. (2010) determined the TMS intensity that led to phosphene perception in approximately half of all trials (the phosphene threshold). By concurrently measuring EEG responses they could show that, in phosphene-present trials versus phosphene-absent trials *ceteris paribus*, widespread electrophysiological responses ranging from occipital-posterior to frontal regions were specific to conscious

perception of phosphenes. These ON-specific responses became apparent quite late, starting from 160 ms after the TMS pulse, suggesting involvement of recurrent processing in conscious vision.

The necessity of recurrent processing for conscious perception of phosphenes had been shown earlier, in a landmark TMS paper by Pascual-Leone and Walsh (2001). TMS applied over the human motion area hMT/V5 is known to elicit moving phosphenes. But when single TMS pulses below phosphene threshold were administered to early visual cortex (V1/V2), thus a cortical region *earlier* in the visual hierarchy (Felleman andVan Essen, 1991), the perception of moving phosphenes from hMT/V5 pulses was diminished or abolished. Importantly, this was only the case for TMS pulses applied to early visual cortex *after* the TMS pulses to hMT/V5. A TMS pulse to V1/V2 preceding the phosphene-eliciting pulse to hMT/V5 had no effect. Thus, recurrent projections from this extrastriate area to V1/V2 were *necessary* for the conscious perception of moving phosphenes. Feedback to early visual cortex activity thus seems to be a neural prerequisite or substrate, not a consequence. In an interesting follow-up study, Silvanto et al. (2005a) reversed the paradigm, administering supra threshold TMS pulses to early visual cortex, thus inducing stationary phosphenes, and evaluating the effect of preceding sub threshold TMS pulses to hMT/V5. They could show that such sub threshold TMS pulses, too weak to elicit moving phosphenes in isolation, could nevertheless affect the quality of phosphenes elicited by subsequent supra threshold early visual cortex TMS. In short: the stationary phosphenes started moving!

Another line of research on phosphenes has involved the study of frontoparietal influences on early visual cortex. Starting with frontal cortex, the bilateral cortical regions known as the frontal eye fields (FEF) have been known to be involved in eye movements and attention (Corbetta, 1998; Corbetta and Shulman, 2002). These regions were also related to successful perception of visual stimuli, for instance in physiological studies with monkeys (Thompson and Schall, 1999). Electrical stimulation of monkey FEF neurons 50–175 ms prior to a visual stimulus actually improved detection of that stimulus (Moore and Fallah, 2001), and follow-up work demonstrated an increase in sensitivity of area V4 neurons by FEF stimulation (Moore and Armstrong, 2003). This and other work (e.g., Ekstrom et al., 2009) suggests a top-down influence of FEF on early visual regions, with the possible consequence of improved conscious vision. Using TMS over FEF in human subjects, Grosbras and Paus (2003) confirmed this hypothesis, showing that single TMS pulses over FEF preceding visual stimuli could improve visual target detection in a masking paradigm. Later work by Silvanto et al. (2006) demonstrated that TMS over FEF also directly affected human motion area hMT/V5, as TMS over FEF decreased the threshold for perception of moving phosphenes. In a similar vein, TMS applied to parietal cortex (posterior parietal cortex, PPC) decreased the threshold for stationary phosphenes elicited by TMS over early visual cortex. But, interestingly, only if PPC was stimulated in one hemisphere; bilateral PPC stimulation canceled out this effect (Silvanto et al., 2009). Below, we will return to TMS applied to frontal and parietal cortices in the study of conscious vision, but first we will address a rapidly growing field known as "TMS masking."

#### **TMS OVER VISUAL CORTICES**

Transcranial magnetic stimulation pulses applied to early visual cortex at rest can elicit phosphenes. But if pulses are applied to the same cortical structures (Kastner et al., 1998; Kammer, 1999), they can also disrupt ongoing processing of a visual stimulus (Amassian et al., 1989, 1993; Beckers and Hömberg, 1991; Ro et al., 2003; de Graaf et al., 2011a,c, 2012a; Koivisto et al., 2011a; Jacobs et al., 2012a,b; Salminen-Vaparanta et al., 2012) and thus abolish it from conscious perception altogether. This demonstrates that early visual cortex activations in the NCC are not consequences, but actually crucial for a conscious percept to arise. It does not, unfortunately, distinguish between substrates and prerequisites. But it does strongly inform (e.g., recurrent) models of visual awareness, due to the chronometric potential of TMS studies (for a recent review see de Graaf et al., 2014).

By applying TMS pulses at a range of stimulus onset asynchronies (SOAs), researchers could demonstrate that recurrent interactions between early visual cortex and higher-order regions are necessary for conscious perception not only of phosphenes, but also when regular visual stimuli were used (e.g., Ro et al., 2003). For example, Silvanto et al. (2005b) applied TMS pulses at different SOAs to early visual cortex or to hMT/V5, measuring conscious perception of motion stimuli. Chronometrically, first only early visual cortex TMS disrupted conscious perception of motion, then only TMS over hMT/V5 disrupted conscious perception, and subsequently only early visual cortex TMS again disrupted conscious perception of motion (see also Koivisto et al., 2010). Also for stationary stimuli, the necessity of recurrent projections from extrastriate cortex (this time lateral-occipital cortex) to early visual cortex was recently demonstrated (Koivisto et al., 2011b). A very interesting recent demonstration of how recurrent processing leads to conscious vision involved the TMS-masking of Kanizsa-type illusory stimuli. TMS pulses applied to early visual cortex *only* successfully masked such illusory percepts in SOAs *following* the SOAs in which extrastriate cortex was functionally relevant (Wokke et al., 2013).

Another interesting application of TMS in the masking paradigm involves what has been referred to as "TMS-induced blindsight" (Ro, 2010). TMS applied to early visual cortex can abolish vision, but this can be measured and evaluated in two ways: with direct subjective reports ["did you see the stimulus (feature)?"], or with (forced-choice) stimulus discrimination tasks. In contrast to blindsight patients, participants with fully intact brains have not had years of training and possible brain reorganization, so it would be useful to probe "blindsight" behavior in them. TMS has been used to this effect, and successfully demonstrated blindsight-like behavioral patterns across a range of stimuli and tasks. In trials without reported awareness of TMS-masked stimuli, "unseen" stimuli could still affect saccade responses (Ro et al., 2004), reaching movements (Christensen et al., 2008; Ro, 2008), emotion recognition (Jolij and Lamme, 2005), and even orientation and color discrimination (Boyer et al., 2005). The occurrence of such dissociations seems to depend on SOA (Koivisto et al., 2010; Jacobs et al., 2012b; Allen et al., 2014). Depending on stimuli and tasks, some have reported that TMS does affect subjective and objective measures, as well as priming measures, as a whole

across SOAs (Sack et al., 2009; Jacobs et al., 2012a). And it has been shown recently that the experimental paradigm and analysis can make quite a difference as well. Lloyd et al. (2013) showed that, in their experiments, obtained TMS-induced blindsight effects disappeared when performing signal detection theory analysis, suggesting that response criteria may play a large role and should be controlled for. New evidence on this exciting topic is continually added, such as a very clever study by Allen et al. (2014) who attempted to determine whether retinotectal and/or geniculate subcortical pathways underlie TMS-induced blindsight effects, by manipulating their stimuli such that the retinotectal (and magnocellular) pathways were bypassed. They found that blindsight-like performance still obtained for such stimuli.

#### **TMS OVER PARIETAL CORTEX**

Interestingly, we can make a smooth transition from TMS masking to TMS studies of the role of parietal cortex in conscious vision, because recent reports suggest that phosphenes can be elicited by TMS over parietal cortex as well (Marzi et al., 2009). And while one recent study did not obtain consistent visual suppression by TMS pulses over these parietal regions (Tapia et al., 2014), Koivisto et al. (2014) did report evidence for parietal (inferior parietal sulcus) TMS masking of specifically subjective conscious vision.

Quite a number of TMS studies have addressed the functional role of parietal cortex in conscious vision, even if the distinction between attention and consciousness was not always clear. In bistable vision paradigms, offline TMS-induced "virtual lesions" of parietal cortex affected the rate of perceptual switching (Carmel et al., 2010; Kanai et al., 2010, 2011), where the direction of effect (increased or decreased switch rates) was found to depend on the exact parietal region stimulated (Kanai et al., 2011). Effects of online parietal TMS on binocular rivalry have also been shown (Miller et al., 2000; Zaretskaya et al., 2010). Mechanisms in parietal cortex may thus play a role in the maintenance of conscious percepts, as also suggested by the finding that parietal TMS pulses can actually induce fading of a continuous peripheral stimulus from consciousness (Kanai et al., 2008). A more general role of resource allocation among competing percepts is suggested by the finding that decreasing excitability of right parietal cortex by continuous TBS (an inhibitory rTMS protocol) rather increased the durations of target disappearance in a motion-induced blindness paradigm (Nuruki et al., 2013). Parietal cortex has further been shown functionally relevant for visual awareness in TMS studies on change blindness (Beck et al., 2006; Tseng et al., 2010), attentional blink (Kihara et al., 2007), and conscious perception of an illusory gestalt (Zaretskaya et al., 2013).

One line of TMS research on the role of parietal cortex in visual awareness we find particularly intriguing, and it concerns TMSinduced extinction (Pascual-Leone et al., 1994). Patients with "neglect," and the symptom of "extinction," were described above. Using TMS over parietal cortex, extinction-like behavior could be replicated in healthy participants (Pascual-Leone et al., 1994), which may be attributable to a biasing of interhemispheric competition for attentional resource allocation. Parietal TMS effects "suppressing" contralateral space and occasionally "enhancing" ipsilateral space have been found repeatedly (Seyal et al., 1995; Hilgetag et al., 2001; Meister et al., 2006; Muggleton et al., 2006; Oliveri and Caltagirone, 2006; Eshel et al., 2010; Bien et al., 2012; Szczepanski and Kastner, 2013; for a recent review on attentional enhancements by NIBS see Duecker et al., 2014). Interestingly, and in keeping with the idea of interhemispheric competition, TMS delivered over both parietal cortices simultaneously actually abolishes this effect (Dambeck et al., 2006).

#### **TMS OVER FRONTAL CORTEX**

Parietal and frontal cortex are frequently mentioned together, as a "frontoparietal network," because they simply do often co-occur in neuroimaging studies in general, and NCC studies particularly. For some NCC paradigms, frontal regions have subsequently been shown to be functionally relevant. For example, we already saw how TMS pulses over FEF can improve visual target detection (Grosbras and Paus, 2003). In subsequent work, TMS applied over FEF concurrently with fMRI was shown to have BOLD effects in low-level visual cortices, compatible with observed effects on psychophysics (Ruff et al., 2006). We already saw that FEF pulses facilitated perception of moving phosphenes from hMT/V5 TMS (Silvanto et al., 2006), and Amassian et al. (2008) reported frontal TMS-induced facilitation of complex phosphenes elicited by early visual cortex pulses, as well as facilitated reporting of weakly illuminated letter stimuli.

For bistable vision paradigms, there is surprisingly little published research with frontal brain stimulation. Neuroimaging has often implicated both frontal and parietal activity in perceptual switching in bistable paradigms (see references above). For example, it was shown with fMRI that specifically frontal activity occurs earlier for spontaneous percept reversals than externally induced ("replay") reversals (Sterzer and Kleinschmidt, 2007), suggesting a causal role (Sterzer et al., 2009). On the other hand, there is recent evidence that if the gradual transition of percept reversals (Knapen et al., 2011) or percept reporting (Frässle et al., 2014) are controlled for, frontal activations conventionally obtained in bistable vision studies with fMRI may be reduced or no longer found. When it comes to NIBS, while parietal disruption leads to effects (see above), we are not aware of studies demonstrating functional relevance of frontal regions for spontaneous perceptual reversals in passive bistable vision so far. (Virtual) lesions of frontal cortex appeared to affect voluntarily induced reversal rates, though no such effect on passive viewing reversals was found (Windmann et al., 2006; de Graaf et al., 2011b). Yet the frontal lobe is large and null results remain fundamentally limited (de Graaf and Sack, 2011). It could be worthwhile to combine fMRI with neuronavigated brain stimulation in future studies to further elucidate the role of frontal regions in passive bistable vision. For the moment we dare not say whether, specifically in the context of bistable paradigms, frontal NCCs are neural consequences, prerequisites or substrates.

There does seem to be evidence for frontal involvement in visual awareness as a metacognitive process. TMS over dorsolateral prefrontal cortex (DLPFC) decreased detection of visual change (Turatto et al., 2004). FMRI research controlling for objective visual task performance found a specifically frontal activation corresponding to subjective visual report (Lau and Passingham, 2006), and in a group of frontal lesion patients, predominantly

subjective visual awareness in a masking paradigm was decreased (Del Cul et al., 2009). In line with this, a theta-burst TMS study in healthy volunteers inhibited bilateral DLPFC and investigated metacognitive sensitivity to visual stimuli. Concretely, a response-bias free measure was calculated, quantifying how well participants' subjective reports of visibility could discriminate their correct or incorrect responses to the same visual stimuli. Frontal TMS decreased this metacognitive sensitivity, and further analysis suggested that this was specifically due to decreased visibility on trials with correctly identified visual stimuli (Rounis et al., 2010).

## **ADVANCED NIBS PROTOCOLS AND THE STUDY OF CONSCIOUS VISION**

We have seen examples of inspiring TMS experiments that directly tested, and demonstrated, functional relevance of brain regions for conscious vision, as well as functional relevance of particular projections between brain regions for conscious vision. Evidence for the latter, while revealing, was generally indirect; for example with the inference of feedforward and feedback projections between early visual cortex and motion area hMT/V5 from the relative temporal patterns of TMS-induced perceptual disruptions (Silvanto et al., 2005b). But new paradigms of NIBS may more directly address the functional relevance of brain interactions, as we discuss below. Similarly, new protocols may widen the range of NCCs that can be tested for causal involvement in conscious perception.

#### **ENTRAINMENT AND PHASE COHERENCE**

Starting with the latter, TMS or tDCS entrainment protocols may be used to test the causal involvement of oscillatory NCCs. The power (Thut et al., 2006; van Dijk et al., 2008) and phase (Mathewson et al., 2009) of alpha oscillations in parietal–occipital cortex, for example, has been related to attention/perception, indexing the successful detection of visual targets (see also Busch et al., 2009; Mathewson et al., 2010, 2012; de Graaf et al., 2013) and see Jensen and Mazaheri (2010), Britz and Michel (2011), and Mathewson et al. (2011) for recent reviews. The relevance of these alpha oscillations had already been probed with TMS by demonstrating that their power (Romei et al., 2008) and phase (Dugué et al., 2011) at TMS pulse onset directly reflected visual cortex excitability as measured by phosphene perception. The hypothesis arose that rhythmic TMS might actually phase-lock and/or amplify such oscillations if the rhythm was of compatible frequency (Thut et al., 2011a). Thut et al. (2011b) could indeed demonstrate that alpha-frequency TMS amplifies alpha oscillations in the brain, and Romei et al. (2010) showed that such a TMS entrainment protocol had attentional/perceptual consequences. Specifically for alpha-frequency, and in a retinotopically specific location, visual performance was enhanced, directly demonstrating the causal relevance of these oscillatory patterns for conscious vision. Rhythmic TMS was recently used to confirm also the functional relevance of alpha phase for visual perception (Jaegle and Ro, 2014). Helfrich et al. (2014) recently demonstrated, using simultaneous EEG and tACS, that alpha oscillations can successfully be induced by tACS at the same frequency, moreover with perceptual consequences. The idea

of TMS (or tACS) entrainment to test functional relevance of endogenous oscillations has moreover been extended to other frequencies and paradigms (Romei et al., 2011, 2012; Neuling et al., 2012).

Another recent development in the field of NIBS is the use of tACS over different regions to bring interregional oscillations either in coherence, or out of phase. This method was pioneered by Polanía et al. (2012), who applied tACS over frontal and parietal cortex at theta-frequency (and control frequencies). Importantly, either the frontal and parietal oscillatory stimulation were at the same phase (0◦ phase-lag), or in anti-phase (180◦ phase lag). Matching prior correlational evidence from EEG, specifically the in-phase frontoparietal stimulation, at the appropriate frequency, led to enhanced task performance. Anti-phase stimulation actually resulted in a performance decrement. Thus, NIBS can now also be used to study the functional relevance of previously observed oscillations, *and* the relevance of interregional coherence of oscillations, implicated in conscious vision.

A very recent example successfully demonstrates this approach. Strüber et al. (2013) took advantage of the fact that a particular bistable visual stimulus, the "motion quartet," involves inter hemispheric communication in one perceptual interpretation (horizontal motion across the vertical meridian) but not in the other perceptual interpretation (vertical motion across the horizontal meridian). Perceptual switches in this paradigm had previously been linked to changes in the synchronization of gamma-band oscillations, so the authors applied 40 Hz (and 6 Hz as a control) tACS to bilateral occipital cortices, at either 0 degrees phase-lag or 180◦ phase-lag. Specifically for anti-phasic tACS at specifically 40 Hz, they observed both perceptual effects in the form of less perceived horizontal motion and effects on coherence in the gamma-band as measured by EEG.

### **CONCLUSION**

A review of the literature suggests that researchers in the field of NCC are often aware of the conceptual limitations around neuroimaging-based, empirical NCCs, either implicitly or explicitly advising caution with regards to interpretations of functional roles. We believe that the explicit division of empirical NCCs into three possible functional roles, which are neural prerequisites, neural substrates, and neural consequences, remains a useful one. Review of the literature also demonstrates how valuable NIBS can be, and indeed has already been, for the enlightenment of these functional roles as a complement to neuroimaging research. The growing number of brain stimulation experiments, and the continuous development of new stimulation techniques and protocols, is a testament to the potential and value of brain stimulation. Hopefully, many more applications of NIBS in the context of NCC research can be expected. And hopefully, the framework of three NCCs will prove useful in this endeavor.

#### **ACKNOWLEDGMENTS**

Tom A. de Graaf is supported by the Netherlands Organization for Scientific Research (NWO 451-13-024). Alexander T. Sack is supported by European Research Council under the European Union's Seventh Framework Programme (FP7/2007-2013)/ERC Grant agreement no. 263472. We thank Dr. Juha Silvanto and Dr. Christianne Jacobs for reviewing earlier versions of the manuscript.

#### **REFERENCES**


Rees, G., Wojciulik, E., Clarke, K., Husain, M., and Frith, C. (2000). Unconscious activation of visual cortex in the damaged right hemisphere of a parietal patient with extinction. *Brain* 123(pt 8), 1624–1633. doi: 10.1093/brain/123.8.1624


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 15 June 2014; accepted: 26 August 2014; published online: 23 September 2014.*

*Citation: de Graaf TA and Sack AT (2014) Using brain stimulation to disentangle neural correlates of conscious vision. Front. Psychol. 5:1019. doi: 10.3389/fpsyg.2014. 01019*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 de Graaf and Sack. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## It is time to combine the two main traditions in the research on the neural correlates of consciousness: C L D **= ×**

## *Talis Bachmann1\* and Anthony G. Hudetz <sup>2</sup>*

*<sup>1</sup> University of Tartu, Tallinn, Estonia*

*<sup>2</sup> Medical College of Wisconsin, Milwaukee, WI, USA*

#### *Edited by:*

*Anil K. Seth, University of Sussex, UK*

#### *Reviewed by:*

*George Mashour, University of Michigan Medical School, USA Jakob Hohwy, Monash University, Australia*

#### *\*Correspondence:*

*Talis Bachmann, University of Tartu, Kaarli Puiestee 3, 10119 Tallinn, Estonia e-mail: talis.bachmann@ut.ee*

Research on neural correlates of consciousness has been conducted and carried out mostly from within two relatively autonomous paradigmatic traditions – studying the specific contents of conscious experience and their brain-process correlates and studying the level of consciousness. In the present paper we offer a theoretical integration suggesting that an emphasis has to be put on understanding the mechanisms of consciousness (and not a mere correlates) and in doing this, the two paradigmatic traditions must be combined. We argue that consciousness emerges as a result of interaction of brain mechanisms specialized for representing the specific contents of perception/cognition – the data – and mechanisms specialized for regulating the level of activity of whatever data the content-carrying specific mechanisms happen to represent. Each of these mechanisms are necessary because without the contents there is no conscious experience and without the required level of activity the processed contents remain unconscious.Together the two mechanisms, when activated up to a necessary degree each, provide conditions sufficient for conscious experience to emerge. This proposal is related to pertinent experimental evidence.

**Keywords: consciousness, neural correlates of consciousness, contents, level, state, visiual perception, thalamus, excitatory postsynaptic potentials**

### **INTRODUCTION**

Scientific research on consciousness has made a major leap forward after influential papers by Bernard Baars, Francis Crick, and Christof Koch outlining the logic and perspective directions of the contrastive analysis (Baars, 1989; Crick and Koch, 1990, 2003). To discover the neural correlates of consciousness (NCC), they suggested to compare brain-process recordings collected in the conditions when subject is conscious (C) with those when that subject is not conscious (U). The difference between these two conditions was operationalized as the NCC, the minimal necessary neural correlates expressed by specific signatures of brain processes differentiating these two conditions as NCC = C − U. There are two main traditions of research using this methodological approach: (1) studying the general states of consciousness versus unconsciousness for revealing NCC (Tradition-1) and (2) studying the correlates of the contents of consciousness in a conscious subject who in some of the experimental trials (or subconditions of trials) has subjective experience of the target stimulus and in some other conditions does not (Tradition-2). (See articles by Steriade, 1996; Laureys, 2005; Massimini et al., 2005; Seth et al., 2006; Rosanova et al., 2012; Sarasso et al., 2014 for an overview of typical tradition-1 studies and Del Cul et al., 2007; Aru and Bachmann, 2009a,b; Lamy et al., 2009; Hesselmann et al., 2011; Pitts and Britz, 2011; Aru et al., 2012a; Sekar et al., 2013 for typical tradition-2 studies.) Tradition (1) has made some progress in finding NCC for the *state* of consciousness while tradition (2) has found a lot of data on NCC in terms of its specific *content*. Ironically, an observer cannot infer the state of consciousness with certainty without testing the presence of some subjective content

in consciousness. In other words, there is no empty or contentless consciousness and even though a highly skillful meditation may perhaps reduce the contents to a minimal extreme (Hohwy, 2009), an experience of some unchanging "empty" state of mind will still have a felt emptiness or oneness as a content. Thus, states and contents seem to be unseparable in that as soon as the unconscious state changes to a conscious state, some content of consciousness inevitably appears. Nevertheless, the above mentioned two traditions of research have been developing in relative mutual isolation.

Recently, a methodological crisis in studying the NCC was spotted (Aru et al., 2012b; de Graaf et al., 2012). It appears that with standard contrastive analysis not only the NCC of the consciousness itself are extracted from brain recordings, but the observed neural signatures may also pertain to processes necessarily preceding those directly responsible for consciousness itself (correlates of the prerequisites of consciousness or NCCpr) and processes that are aftereffects or consequences of consciousness (NCCae). It appears that it is not at all easy to distinguish the NCCpr and NCCae from the processes directly equivalent to the neural processes minimally sufficient for the conscious experience itself (i.e., the NCC proper, or constituents of consciousness, to use the distinction made by Miller, 2001, 2007). We believe – together with Hohwy, 2009 – that the theoretical picture used to explain and understand the results of research on NCC is confused partly because the above mentioned two traditions, the study of conscious state vs. content, have progressed separately. "It seems then that neither the content nor the state-based approach taken in isolation from one another will help us discover the NCC" (Hohwy, 2009, p. 435). Thus, we argue that Tradition-1, by concentrating on the brain mechanisms necessary for the general state of consciousness, has been neglectful with regard to the systematic experimentation in the domain of finding the NCC for subjective contents and because of this, it has had difficulties in properly specifying the NCC. Neither the difference in the exact contents nor the difference in the exact timing of the subjective experience of these contents along the subsecond timescale have been studied. We also argue that Tradition-2, by concentrating on the brain processes that correlate with subjective awareness of specific target stimuli has tended to overlook the data and regularities obtained in the research on NCC of conscious states. In observing and analyzing brain-process signatures as candidates for NCC, no attempt has been made to clearly distinguish the contributions of the state and content systems to these signatures.

Aside from the issue of the exact timing of emergence of conscious contents and their NCC in terms of brain-imaging signatures, an attempt to bridge to state-content interrelationship is the former study by Owen, Laureys and colleagues capitalizing on the contrastive analysis of fMRI signals between vegetative state (VS) patients and normal alert controls (Owen et al., 2006). VS patients who were clinically defined as being unconscious were asked to imagine playing tennis while their pattern of fMRI activation was compared to that obtained in healthy control subjects reporting the corresponding phenomenal content. The two brain activation patterns were quite similar, which the authors used to suggest that VS patients have conscious contents. Hohwy (2009) criticizes this approach based on the grounds that what the investigators tested really was volition rather than qualia. In our opinion this critique is insufficient for several reasons: (1) the subjective experience of volition is also a kind of consciousness with its content, although possibly different in form from sensory experience fo qualia, (2) in order to express volition, patients had to first semantically process the command and be able to differentiate between the requested choices. Thus it seems that either we really have some residual form of consciousness in VS patients or the brains of VS patients are capable of surprisingly intricate and precise information processing without the concomitant subjective experience simply executing automatisms of processing sensory input up to cognitive and semantic stages (supporting the zombie mode of complex information processing). On the other hand, the so-called zombie modules seem to exist indeed if we accept data by Kotchubey (2005) and Owen et al. (2006) as evidence for complex perceptual-cognitive processing of stimulation contents by the unconscious brains. The serious shortcoming of this type of research consists in the lack of reliable independent measures clearly associated with phenomenal subjectivity. We do not know whether the specific contents processed by the brain in a specific state (e.g., VS or a locked-in state) are also reflected phenomenally in the first person perspective. Because verbal reports from subjects in states for which we cannot definitely say whether these are conscious or unconscious cannot be reliably obtained, we are left with the option to study *the effect of experimental manipulations of the brain's state systems only when these manipulations are applied when the human subject is in a state allowing subjective report*. When this type of research has developed well enough and the corresponding mechanisms of consciousness will have become more precisely specified, experiments with animal models can be then carried out in a more meaningful way with regard to the problem of NCC.

Although many of the clinical criteria for qualifying a state as a conscious, minimally conscious or unconscious state and as a state of sleep or behavioral arousal are objectively physiological (e.g., EEG) and behavioral (e.g., reflexes), at present we can rely on subjective reports and evaluations of the contents of mind as the only available method to assess presence of consciousness reliably. An important dimension which allows measurement of variable states from the point of view of subjectivity is the scale of the *levels of consciousness* expressed in terms of the clarity, vividness and fullness of subjective conscious experiences. Vague, barely noticeable, fragmented, poorly differentiated experiences stand at the one end of the scale and clear, vivid, differentiated and stable experiences at the other end. Sedated patients typically describe their experiences in terms of this kind of qualia continuum where the quality of contents changes along with the change of state from unconsciousness to hypnagogic experiences to vague experiences up to the full blown clear and vivid perception of the world and private thoughts (Newton et al., 1990; Andrade et al., 1994; Zacny et al., 1994; Hudetz and Pearce, 2010). Consciousness contents have different levels of expression that can vary vith invariant environmental stimuli input as a function of the varying state of the brain. Thus, operationally the effects of the state system on the content system should be possible to measure as the variations in the level of subjective experience as measured by the contents typical for the gradations of the level (Bachmann, 2012). Importantly, there are scalable attributes of the level common to different modalities and within-modality contents which makes these attributes universal.

We believe that by combining the two approaches within a theoretical synthesis some – but not all – of the controversies in NCC research can be solved. We suggest a theoretical synthesis where the activity of the mechanisms examined in each of the two traditions are the prerequisites (NCCpr) of consciousness, but together they form the combination jointly sufficient for consciousness (i.e., NCC proper). In a highly simplified form, we propose an expression for consciousness: C = L × D, where C stands for consciousness, L stands for the critical *level* of activity of neuromodulatory systems regulating the state of the brain, and D stands for the *data*, i.e., the contents of consciousness provided by the specific representational systems of the brain. In order to ease the abstractness of the concept, we give a few examples on what we mean here. In an experiment aimed to manipulate the state level L, for example, with an incremental dosing of anesthetics, a suitable task-response can be studied simultaneously at matched performance levels (accompanied with subject's scaling of the level and/or contents of his/her subjective experience) while doing fMRI, MEG, EEG or something else. In doing this, matching the task performance may be important to minimize the contribution of NCCae on the recorded neuronal signatures. One could also look at other changes in the brain, e.g., functional connectivity as candidates of NCC proper or its reliable signature. Similarly, EEG/ERP recordings can be used for comparing subjective reports under varying degrees and means of sedation by anesthetics, including measurements of possible NCCae (e.g., Veselis et al., 2009). Some additional examples will be given later on in this article. Obviously, in order to follow the strategy of the combined use of the L-system and D-system based approach, these systems should be known sufficiently well in terms of neuroanatomy and ways of functioning.

Before elaborating this concept further, let us have a closer look at the putative brain mechanisms and processes that have been proposed as necessary for consciousness both in terms of its content and state, and after having done so, we will return to the problem of NCC.

## **TWO MUTUALLY INTERACTING BRAIN SYSTEMS RELATED TO CONSCIOUSNESS**

To experience something subjectively, the brain has to be in the state of consciousness, but also capable of processing sensory data which constitute the contents of experience. In other words, the presence of specific sensory contents in conscious awareness is not automatically granted, but requires involvement of cerebral *interactive mechanisms* that aid the explication of preconsciously processed information. Neurobiological experimental data, deep brain stimulation data, and anesthesiological data converge in showing that modulations mediated by the thalamocortical "nonspecific" system targeted at the specific system of encoding the sensory-perceptual information are necessary for subjective contentful experience to emerge (Brooks and Jung, 1973; Bachmann, 1994; Purpura and Schiff, 1997; Llinás and Ribary, 2001; Ribary, 2005; John, 2005; Hudetz, 2006; Alkire et al., 2008; Hudetz and Alkire, 2011; Giacino et al., 2012; Liu et al., 2013). An important aspect of this dual functional architecture consists in the way these two systems interact at the cellular level. There is massive presynaptic targeting of the apical dendrites of the layer-5 pyramidal neurons in cortex originating from the non-specific system while the layer-5 pyramidal neurons are the prime part of the specific afferent system at the cortical level serving the function of encoding specific stimulation content. This specific afference arrives presynaptically at the more somatic parts of these nerve cells. It is suggested that this is the cellular level mechanism where nonspecific and contextual top-down modulatory influences interact with primary specific afferent input and that the extent or pattern of this interaction highly correlates with subjective states of consciousness (Brooks and Jung, 1973; Purpura and Schiff, 1997; Llinás and Ribary, 2001; Ribary, 2005; Alkire et al., 2008; Pillay et al., 2014). The layer-5 pyramidal neurons are naturally suitable to be a mechanism where perceptual contents are modulated up to consciousness, but at the same time this mechanism allows both pre-conscious and unconscious information processing. Although inputs to cell soma and dendrite tuft can perform their own functions autonomously (Berger et al., 2001; Larkum et al., 2004), these inputs are also capable of intense integrative interaction. Therefore, the same system can perform information processing in either the preconsious (non-interacting) or the conscious (interacting) domain. Membrane activity caused by somatically targeted sensory input is "available" for effective (i.e., hypothetically awareness-providing) modulation if it has sufficient frequency (Larkum et al., 1999a) and optimal timing with respect to the apical dendrite targeted

input (Larkum et al., 1999b; Larkum, 2013). The autonomous function of the two neuron compartments is possible also due to the fact that the level of membrane depolarization is progressively attenuated distally from soma, and also from the tuft, as well as for the very fact that an absence of temporal coincidence of somatic and distal-dendritic input dissociates the two subcellular activities and thus the potential to act as an integrative device remains unused (Berger et al., 2001; Larkum, 2013).

For conscious perception, it is *necessary that both systems* be active in concert – (i) the bottom-up, specific content-signaling data system D receiving afferents by the layer-5 pyramidal neurons' basal compartment, and (ii) the modulation system sending intracortical top-down associative signals and/or non-specific thalamo-cortical signals leading to up-states necessary for a sufficient consciousness level L (Bachmann, 1994; Ribary, 2005; Larkum, 2013). When only one of the two principal input streams to pyramidal neurons – somatic specific afferents and dendritic presynaptic nonspecific afferents – is active without the "partnering" effect of the other system, consciousness is expected to fade away. Formally: D(0) × L(z) = C(0); D(x) × L(0) = C(0). Only D(x) × L(z) = C(x), with x being the contents of the experienced subjective state.

It is likely that for the hypothetical dual-input consciousness mechanism to become ignited, both neuron compartments – somatic and apical dendritic – must receive enough presynaptic input in order to generate the wide excitatory postsynaptic potential (EPSP) plateau-waves carrying a burst of spikes representing an up-state. One good candidate for being the core cellular level mechanism in this system is the backpropagation activated calcium spike firing (BAC firing) mechanism described by Matthew Larkum and colleagues. For the layer-5 pyramidal neurons to generate plateau-wave-based spiking, the temporal coincidence of somatic sodium channel-related presynaptic input (i.e., data for the subsystem D) and calcium channel-related presynaptic input targeted at the apical compartment of the cell dendrite (i.e., modulation by the subsystem L) is necessary (Larkum et al., 1999b; Larkum, 2013). It is important to acknowledge that dendritic calcium spikes are a direct target of anesthetics (Potez and Larkum, 2008) and after release from the anesthetic effects, calcium electrogenesis in layer-5 pyramidal neurons dramatically increases (Murayama and Larkum, 2009). This supports the notion of the principal importance of the BAC firing mechanism for consciousness. Importantly, suprathreshold input to the neuron's body (responsible for the bottom-up inflow of sensory signals by D) produces fewer action potentials of the cell than triggering of the dendritic Ca2<sup>+</sup> spikes. This once more substantiates the importance of modulatory brain processes in addition to the straightforward sensory afference and provides a convincing argument for the common effects of biased perception being under the contextual and arousal systems control. [See also works by E. Roy John (e.g., John, 2005) and Rodolfo Llinás (e.g., Llinás et al., 1998) on the putative significance of coincidence detection for the effectively working consciousness mechanism.]

In addition (or alternatively to) the thalamocortical resonance or interaction theory, the role of cortico-cortical interactions in the NCC should not be excluded or overlooked. For example, Mashour and coworkers (Lee et al., 2013) demonstrated the selective alterations in fronto-parietal interactions during general anesthesia with three different anesthetic agents. The use of different drugs in the same investigational setting is important in order to be able to identify an anesthetic-invariant mechanism of loss and return of state consciousness (John, 2001), removing agent-specific effects, and by the same token, at least some of the NCCpr or NCCae. A particularly interesting aspect of the work of Lee et al. (2013) is that fronto-parietal communication was preferentially suppressed by all three anesthetics in the feedback (top-down) direction. This important finding is consistent with what was proposed by Del Cul et al. (2007) and predicted by prior preclinical anesthesia studies by Imas et al. (2005a), suggesting that cortical long range connectivity, especially in its top-down aspect, may be critical for consciousness. On the other hand, based on the general knowledge that non-specific thalamocortical effects target more frontal and central than posterior areas it is possible that the typical effect on the top-down aspect of cortical effective connectivity reflects also (or first of all) the effect of the extended thalamo-cortical modulation system. Further, indirect support for the nonspecific brain systems regulating conscious thresholds for visual stimuli comes from a recent study by Park et al. (2014) who showed that the threshold varied with the phase of heartbeats. It is difficult to assume that heartbeats are directly associated with processing specific visual input, but it is easy to understand that and how heartbeats reflect activation levels of the non-specific systems involving the autonomic system contribution to arousal.

In some situations, specific input is processed by the brain and/or stored in its long term memory system but there is insufficient L-system modulation for conscious awareness of the sensory input or stored memory-information. Examples include subjects having fallen asleep, undergone fainting, or brought into anesthetic states. Brains of sleeping or anesthetized subjects receive localized specific input that comes from sensory receptors, but associative input and modulating arousal input from the reticulothalamic activation system is insufficient (Magoun, 1958; Hassler, 1978; Purpura and Schiff, 1997; John, 2005; Ribary, 2005; Hudetz et al., 2009; Hudetz and Alkire, 2011; Zhou et al., 2011). Information integration to consciousness is absent (Alkire et al., 2008). On the other hand, when the L-system is artificially stimulated, arousal and consciousness accompanied by sensing the contents of ambient stimulation can be brought about, cortical information-integration can be augmented, release from visual masking produced, or artificial sensations evoked (Smirnov et al., 1978; Tasker et al., 1980; Bachmann, 1994; Giacino et al., 2012; Pillay et al., 2014). Artificial stimulation of the intralaminar thalamic sites that are part of the nonspecific system augments visual evoked potentials in response to grating stimuli (Hunsperger and Roman, 1976). Visual, auditory and somatosensory research using recording of brain potentials has repeatedly shown that the level of expression and timing of the negative-polarity potential N1 (with post-stimulus latency equal to about 50–200 ms depending on the modality and stimulation characteristics) strongly correlates with conscious perception (Uttal and Cook, 1964; Wagman and Battersby, 1964; Hassler, 1979; Alter et al., 1990; Cauller and Kulics, 1991; Bachmann, 1994; Imas et al., 2006; Schubert et al., 2006; Schoenfeld et al., 2011; Auksztulewicz et al., 2012; Auksztulewicz and Blankenburg, 2013; Sinke et al., 2014; Pitts et al., submitted). For example, while early post-stimulus potential components are more or less invariant between sleep or sedation on the one hand and awake conditions on the other hand (or even increased in sedation – Imas et al., 2006), N1 is considerably suppressed in the unconscious state. Similarly, visual cortical neurons in anesthetized animals show early activity in response to light flashes similar to the unanesthtized state, but long-latency responses significantly decrease under sedation (Hudetz et al., 2009). Also, in the crowding effect when target information is left out of awareness, N1 is suppressed although P1 ERP component still reflects basic stimulus characteristics regardless of crowding (Chicherov et al., 2014). When similar monocular textures fuse into subjectively visible images, conspicuous occipitally recorded negative component is present, but absent when fusion into visibility is not achieved (Fahrenfort et al., 2012). Moreover, recently it was shown that whether subjective experience of a visual stimulus in hysterical blindness is present or not is reflected in the amplitude of N1 (Schoenfeld et al., 2011). Importantly, while N1 appeared as an authentic NCC, fMRI recording data could not differentiate between consciously seen vs unseen stimuli. For our purposes it is essential that the electrogenesis of negativity of the surface-recorded brain potentials considerably owes to the apical dendritic activity in the upper layers of the cortex (Larkum, 2013). The stimulus-evoked negativity can be easily explained as resulting from the excitatory synaptic input to the distal compartment of the apical dendrites of layer-5 pyramidal neurons. The fact that consciousness of a stimulus emerges with a relatively long delay after 100 ms (Bachmann, 1994, 2000) is strongly consistent with the fact that N1 is relatively slow compared to the timing of the early arrival of stimulus signals to cortex as indexed by the faster ERP components. This is despite the fact that the early ERP components with latencies under 100 ms can vary depending on whether the target stimulus presented in invariant physical conditions becomes consciously experienced or not (Aru and Bachmann, 2009b) suggesting that they are signatures of the NCCpr and not NCC proper. Furthermore, an ERP signature termed visual awareness negativity (VAN) most conspicuously present as recorded by lateral occipital electrodes also correlates with stimulus awareness (Railo et al., 2011). (However, see Bachmann, 2009, concerning the problem that VAN duration is too short compared to the duration of subjective experience when subjects hold target-stimulus in explicit immediate memory. This problem appears to find a solution in the correlation of slow negative potentials with conscious awareness as discussed later in this article.)

Thus, taking into account the experimental regularities reviewed above, let us operationally regard EEG potential negativity as a signature of the involvement of the L-system in modulating the neural activity of the D-system. This notion does not mean that modulation by the L-system *always* leads to conscious experience of the stimuli; it is likely that a certain level of the L-activity modulating D-activity is necessary. This in turn implies the existence of a threshold level and/or threshold duration of the processes producing ERP negativity, with sub-threshold levels of the measured negative cortical potential measurable in principle. Consistent with this, Intaité et al. (2014), demonstrated pre-reversal negativity as a signature of the change in conscious perceptual contents. Several hundred ms before subjective reversal of the perceptual contents of an invariant ambiguous stimulus, ERP negativity was enhanced when subsequently the conscious contents were changed.

Not only brief transient ERPs with negative polarity correlate with stimulus awareness timing, but also the slower negative deflections of the brain potentials such as, the contingent negative variation, slow cortical potential waves, or contralateral delay activity appear to be good candidates of a reliable and robust correlate of consciousness, especially as related to expectancy states and application of general-purpose brain-process resources for a specific task involving focused attention (Rösler et al., 1997; Birbaumer, 1999; Devrim et al., 1999; He and Raichle, 2009; Murayama and Larkum, 2009; Fischer et al., 2010; Murd et al., 2010; Stamm et al., 2011; Birbaumer et al., 2012; Pun et al., 2012; Jo et al., 2014; Li et al., 2014). Basically, the slow potential dynamics reflects changes in cortical excitability (He and Raichle, 2009; Raichle, 2011). This notion is substantiated by the known intimate relationship between the slow cortical potential and the thalamocortical neuromodulatory system (i.e., the system controlling L, according to our notation). This conceptualization is consistent with observations that trans-cranial magnetic stimulation (TMS) of the occipital cortex elicits state-dependent effects when performed in a state of increased arousal (brought about by caffeine administration) as compared to quiet wakefulness (placebo condition; Murd et al., 2010), or during NREM sleep compared to awake state (Stamm et al., 2011). When caffeine was applied or wakeful state was tested, TMS evoked a stronger early negative component (N1) and enhanced slow negativity in remote cortical areas compared to the placebo condition or NREM sleep condition. Because TMS is a task-irrelevant, nonspecific signal in terms of the contents of consciousness, these results speak in favor of the stance that (slow) negativity reflects nonspecific effects on cortical excitability. Provided that slow negativity correlates with (readiness for) long range effective connectivity, TMS is expected to facilitate cortical communication in conscious states in order to allow sufficient information integration. Studies by Tononi, Massimini and associates indeed show this (e.g., Massimini et al., 2009; Ferrarelli et al., 2010). They tested the state-dependence of the cortical propagation of a TMS-induced EEG signal and found that in the unconscious condition, either in slow wave sleep or anesthesia, long-range cortical communication was suppressed, and restricted to a local region surrounding the stimulation site. In terms of temporal dynamics, the late (sustained) component of TMS-evoked signal was also suppressed, in agreement with results obtained previously with the direct measurement of flashevoked unit activity in the visual cortex of rats (Hudetz et al., 2009).

In accordance with the conceptualization presented above, Yasuda et al. (2011) showed that with the loss of consciousness at sleep onset, N1 decreased, P2 increased (!), and CNV was increasingly more positive. In sleep the CNV was absent.

We believe that a step forward in revealing the NCC would be achieved by the use of a combined strategy consisting in three key requirements: (i) *causal* mechanistic effects are used and studied instead of correlational ones in the objective domain of

research so that microelectrode stimulation, optical stimulation, TMS, psychopharmacological intervention, or other methods of purposeful manipulation of neural tissue in selected brain areas allow to study the resulting effects on brain processes by brainimaging or similar methods; (ii) the *relative involvement of the D- and L-systems* is specifically examined in the context of objective causal manipulations and measurements; (iii) psychophysical procedures are used in combination with (i) and (ii) allowing the *precise measurement of subjective contents* of *consciousness* as they emerge, unfold, and decay along the objective time axis. Correspondences between precise timing of the neural signatures and psychophysical phenomena pertaining to the domains of (i), (ii), and (iii) will hopefully make it possible to find causal relationships that are more likely to be indicative of the real constituents of consciousness in terms of the underlying neural processes. An important accompanying research agenda should be to use animal models with mammalian species, executing studies (i) and (ii) and by analogy, invoking and postulating the corresponding variability of the experienced contents (iii). In what follows, we will provide a few *ad hoc* interpretations of the published research suitable for differentiating the D- and L-system effects in the context of the problem of NCC. Thereafter, we will conclude by envisaging possible experimental approaches for future research in service of the view that C = L × D.

## **EMPIRICAL FACTS CONSISTENT WITH C = L × D**

There are surprisingly few research data on the NCC as based on experiments explicitly relating content- and state-related system effects within the same study that utlize the methodology of contrastive analysis. Measurements and discussions have been evolving around the specific D-system contributions with rarely – if at all – mentioning the L-system contribution. This conclusion is especially valid with regard to the real-time examination of the NCC as they unfold along the fine timescales of psychophysical experimental paradigms. Although a lot of psychophysical and cognitive neuroscience research has been published with results that can be *interpreted* according to our C = L × D "formula," this kind of research lacks the power of *direct evidence*. Nevertheless, here we discuss both – studies with experimental protocols and designs more directly combining the L-system and D-system variables as well as studies allowing indirect interpretations of their data in terms of L- and D-system interactive contributions. We do this, being well aware of the methodological weaknesses of the *ad hoc* type of argumentation about the second sub-variety of the pertinent research as mentioned above.

Alert patients regularly treated by chronically implanted microelectrodes in order to relieve the suffering from Parkinson's disease were used as volunteer subjects in a visual mutual masking experiment (Bachmann, 1994). They were asked to discriminate spatially overlapping brief stimuli presented successively with stimulus onset asynchrony (SOA, values less than 100 ms) leading to masking. Compared to the control condition without the pre-test microelectrode stimulation, awareness of the visual targets improved considerably in the main condition with a preliminary activating stimulation targeted at the thalamic nuclei which were the known part of the L-system (including the central medial intralaminar nucleus). Artificially activating the L-system produced improvement of perception of the specific contents represented by the D-system. The effect was obtained also with invariant physical conditions of the independent variables.

A very brief near-threshold Gabor patch was presented for orientation discrimination in the study by Li et al. (2014). MEG responses as an equivalent of slow cortical potential (SCP) were recorded in the conditions allowing comparison of trials with (i) strong and marginal awareness and unawareness, (ii) correct and incorrect behavioral responses, and (iii) subjective confidence in the correctness of response, all with invariant values of the principal independent variables. After the effects of objective performance and confidence were both removed from data, the SCPs were found to be a strong and reliable correlate of consciousness while the neural signatures correlating with veridicality of response and confidence were mostly related to transient brain activity. Because long-lasting EPSPs at the apical dendrites of the (mostly layer-5) pyramidal neurons are considered as the main contributors of surface-recorded SCPs and as the equivalent MEG contrasts between aware/unaware conditions stood for the expression of SCPs, we can conclude that modulation of the D-system (carrying the contents of Gabors) by the L-system (modulating the EPSPs of the content-carrying neurons of the D-system) may be an instance of the NCC. In light of these results, it is inviting to speculate about the possible reasons why LFP responses indicative of correct stimulus perception in the slower theta band preceded the single-neuron gamma band response by about 50- 100 ms in the experiment reported byRey et al. (2014). It is possible that the slower LFP activity in the theta band range represents the L-system activity and the gamma band single-cell responses correspond to the D-system activity. If so, the contextual and/or expectancy-related or arousal-related L-system activity may precede the bottom-up afferent signaling from the specific modal stimulus so that it enhances EPSPs by presynaptic signals targeting the apical dendrites, but not much the high-frequency firing which needs also a sufficient input to the somatic compartment of the D-system nerve cells in the temporal cortex. Only as soon as there is enough coincidence of the L-system and D-system input, firing commences and explicit recognition results. This conjecture is consistent with that of Larkum (2013) with respect to the BAC firing mechanism.

Thus, in spite of suggestions by several authors (Llinás, for example) it seems that a change in thalamocortical 40Hz gamma oscillations, at least in the local sense, may not be the critical factor in the involvement of the L-system for the NCC. Various data support this. In anesthesia with various select agents, the power of both spontaneous and sensory evoked gamma oscillations is *enhanced* (Imas et al., 2005b; Sellers et al., 2013), although preliminary findings suggest that high-frequency gamma (70- 140 Hz) may be suppressed (Hudetz et al., 2011). Nevertheless, the long-range cortical interactions at 40–50 Hz gamma frequencies do seem to be selectively altered in an opposite direction to what should have been expected from the traditional gamma band hypothesis (Imas et al., 2005a,b). All this is consistent with recent findings by Aru et al. (2012a) and Hermes et al. (2014) with respect to the lack of a band-specific gamma correlate of conscious perception. As suggested by several authors (e.g., He and Raichle, 2009; He, 2014; Miller et al., 2014), broadband and/or

scale-fee activity of neuronal populations seems to be associated with surface potentials, dendritic integration and very likely with awareness. It appears that BAC firing as a potential mechanism subserving the NCC are unlikely to be tightly coupled with the gamma band responses in implementing the C = L × D type of interaction.

The BAC firing mechanism as related to behavior of alert animals was involved also in the findings of Palmer et al. (2012a,b). They showed how contralateral sensory stimulation inhibited the firing of layer-5 pyramidal neurons in response to a sensory stimulus *in vivo* and that this inhibition acted on the apical dendrites. Somatic input to pyramidal neurons generates predominantly short-lived EPSPs and fragmented cortical neural activity but not a sustained "field of consciousness," unless substantial additional modulatory input to the apical parts of the dendrites arrives. On the other hand, only or predominantly non-specific modulation by the L-system targeting the superficial cortical layers in the absence of sufficient afferentation from sensory signals by the D-system (targeting the perisomatic compartment) should lead to fading or "contentless" conscious experience. For example, this happens with fading of retinally stabilized images (due to massive adaptation of the same sensory units), Ganzfeld effects featuring feelings of sensory emptiness (e.g., the coloring of the Ganzfeld disappears from subjective phenomenal experience), falling asleep in sensory isolation experiments, and with hypnotic effects of mononotonous and predictable stimuli (the responses to which are diminished by adaptation effects).

Whatever the normally functioning consciousness mechanism is, it must not represent mainly unreal or hallucinatory content in the awake state but perform adaptively sound *reality monitoring* (communicating actual environmental input to the subject by the D-system). This is obvious. Indeed, integrative computation by the BAC firing mechanism works mostly when the real presynaptic input via the D-system to the neuron's basal compartment is present (Larkum, 2013). Only when the cell is caused to fire by feedforward presynaptic D-input to the soma it becomes highly sensitive to the apical-dendritic modulation (Larkum, 2013). This means that the mechanism that brings contents to consciousness has its strong effect insofar as the actual input is present, meaning that in consciousness we clearly experience actual sensory input. Conversely, when subjects experience dreams or are engaged in daydreaming, the conscious contents that are hypothetically ignited by the presynaptic input to the dendrite tufts originating from more rostral, higher level cortical areas responsible for top-down effects (ignited by the L-system) cannot be as detailed and vivid as they are in the actual conscious perception of the real environment.

In the absence of top-down facilitation or selection by the L-system (which may be contolled by the brainstem-nonspecific thamocortical axis) the conscious contents are blurred or nonexistent. The necessary integration is missing. It can be presumed, that with increasing depth of anesthesia, from shallow sedation to deeper levels, the L-system and the D-system neural processes are changed in parallel: reduced state means reduced level of data input to the integrative system and vice versa. Even though strong inputs by the D-system may break through in the state of suboptimal L-system activity, they produce strong and focused

sensory activation but in a limited, modality-specific local area (e.g., Laureys et al., 2002; Liu et al., 2013). However, integration with context from higher, supramodal associative processing centers, that are now silent, does not take place. Moreover, there are specific conditions, for example under the influence of hallucinogens, when the normal interaction of the L and D systems can be clearly dissociated.

Many subjective psychophysical phenomena of perception have temporal resolution with time constants varying around 30– 300 ms (see Bachmann et al., 2011, for listing and characterizing the many pertinent phenomena.) Thus, for example metacontrast masking maxima occur typically at the target/mask delays of 30– 60 ms; apparent motion in the form of beta motion integrating two static inputs for perception is optimized with SOA set at about 40–100 ms; iconic (visual sensory) memory (i.e., visible persistence) spans about 100–200 ms; critical flicker fusion temporal values are also specified in the range of dozens of ms; attentional blink is maximized at about 250–350 ms; target stimulus that is backward-masked by a pattern mask or by an other, competing, subsequently presented target becomes fully consciously available when the mask-free time approaches about 150 ms, but is replaced by mask in visual awareness with shorter target/mask time intervals. This suggests that the neural processes fitting to allow effective inter-stimuli interactivity (caused by presynaptic input arriving from different sources in response to briefly inset stimuli) and also for sustained activity enabling inter-stimuli interaction should be unfolding in the range of intervals of about 30–350 ms. In the traditional integrate-and-fire neuronal models, the EPSP decay is usually too short for this. However, in the BAC firing model the plateau-wave time-course corresponds well to this temporal scale. Most significantly, the BAC firing model nicely corroborates the distinction between the L-system effects and D-system effects related, respectively, to the presynaptic input to the tuft dendrite compartment and the basal somatic compartment of the post-synaptic membrane of layer-5 pyramids. Several psychophysical phenomena can be explained by the hypothetical action of the BAC firing mechanism allowing to combine the L-system and D-system effects by a single cellular level mechanism of integration. We believe that this is exactly the locus where the two traditions of NCC research can be mutually harmonized.

In the *visual backward masking by pattern*, a brief target stimulus (e.g., with 40 ms duration) is immediately followed by a masking pattern and as a result, target is not consciously perceived while the masking pattern is explicitly experienced (Bachmann, 1994; Breitmeyer and Ö˘gmen, 2006). Recall that the theoretically necessary condition for sensory input to be integrated up to the level of conscious experience requires that input from feedforward D-channels targeted at the somatic compartment of the neurons that represent the stimulus content and associative (modulating) input targeted at the distal dendritic compartment that modulate content-specific activity up to the necessary level for consciousness must simultaneously coincide in time. Target signals feed synaptic receiving membrane compartment close to cell soma with a short delay (say, 30–50 ms) generating few somatic Na+ spikes. However, because this delay is too short for any associative input to arrive to the tuft region of the dendrite in

response to the target-evoked perturbation, initially there is no target experience. Processing is pre-conscious. After some more time has lapsed, this associative, tuft-area directed presynaptic input arrives (say, with about 100 ms post-target delay), but it coincides with the mask-stimulus evoked Na+ spikes produced by the neurons that encode mask features. Because certain features of target and mask are shared by the target-responsive and maskresponsive cells (e.g., spatial location, some blob- or line-defining features, etc.), mask-responsive cells receive also the associative presynaptic Ca2+activity initiating input to the dendrite's upper compartment (this was evoked by the preceding target) and as this input is coincident with Na+ spikes, a plateau wave is produced primarily for the neurons representing the mask features instead of the neurons representing exclusively the target features. We must remember here that the modulating input through the L-system in response to a perturbation by a stimulus has a longer delay to reach the apical compartments of the dendrites of the neurons compared to the delay it takes for the initial basal input throught the D-system to arrive to the cell. This basic fact is the crucial precondition for BAC firing based binding of the later presented stimulus with conscious representation instead of the first presented, briefly offset stimulus. Thus, the target-evoked dendritic Ca2<sup>+</sup> mediated EPSP appears after a delay, is spread also to the dendrites of other cells (e.g., mask-related neurons), and coincides with the fast Na+ based somatic EPSP/spiking process of the mask-related cells. It is exatly then and there when and where the coincidence detection device sets in, but as a result, masking stimulus is emphasized for awareness. Bachmann (1994, 1997) in his model of modulated EPSPs as the explanatory mechanism of masking contends that temporal dynamics of the target-evoked N1 may be a signature of the L-system effects as they unfold in real time and are applied onto the D-system neural nodes.

In the *mutual masking of the spatially overlapping two successive targets* subjects are asked to report both successive stimuli, S1 and S2 (Bachmann and Allik, 1976; Bachmann, 1994). With very short SOAs (e.g., 0–20 ms) the results typically show neither the perfect identification of both the S1 and S2 nor the perfect masking of S1 and S2. The BAC firing mechanism based explanation is as follows. Because the SOA is so short, the L-system presynaptic input to the dendrites tuft-region compartment arrives with a considerable delay and "finds" S1 and S2 related neurons producing some residual somatically ignited Na+ spikes in response to the presynaptic D-system afferents. As a result, a plateau wave with enhanced spiking in the form of a burst is produced and both stimuli become bound together as an integrated pseudo-object in visual awareness. With intermediate SOAs (e.g., 40–90 ms) usually the following stimulus (S2) is well perceived, but the preceding stimulus (S1) is not perceived or not so well perceived. BAC firing mechanism based explanation of this psychophysical result is essentially the same as was used for explaining pattern masking above. When the time interval between onsets of S1 and S2 (SOA) exceeds about 100–150 ms, S1 becomes well visible also. The strongest dominance of S2 over S1 is typically found with SOAs around 40–60 ms. The BAC-based scenario for explaining this is as follows. Somatic presynaptic inputs from S1 carrying the contents of S1 arrive at the layer-5 neurons that represent S1 features, creating Na+ EPSPs'spiking response, but because there is no dendritic calcium response as yet (this response cannot be produced unless more time-consuming L-system mediated signaling originating from the nonspecific reticulo-thalamic influence on apical dendrites is present), plateau/burst wave is not produced for the neurons carrying S1 sensory contents in this early time window. When this delayed presynaptic input arrives at the distal parts of the apical dendrites, EPSPs of the S1-neurons are somewhat decayed, firing frequency is decreased, and probability of the conspicuos plateau-wave is low. S1 is not explicitly perceived so well as the following S2 which altogether suppresses S1 in conscious perception. This is because the somatic-compartment response of the following stimulus related neuron (S2 neuron) is maximized at the moment temporally coinciding with the slower (S1-induced) calcium wave and because due to the spatial-location and some partial featural similarity between S1 and S2, apical dendrites of the S2 neurons are also targeted and the plateau wave is distinctive and strong for the S2-related neurons. As a result, S2 is consciously perceived and S1 perception suppressed. (It is likely that some additional assumptions with regard to the mechanisms involved are needed. For example, the role of NMDA receptors and finely localized sub-cellular membrane mechanisms of competitive inhibition could be postulated to explain why S2 so robustly overpowers S1. See Larkum et al. (2009), for premises of this kind of mechanisms. Also, more precise understanding of the workings of GABAa and GABAb mediated inhibitory mechanisms may be needed in order to precisely describe the ways the L-system and D-system interact on the basis of the layer-5 pyramidal neurons.)

*Metacontrast masking* can be explained similarly by the L-system and D-system interaction taking place at the basal and apical-dendritic membrane of the cortical neurons representing the contents of stimulation. In this phenomenon, S1 (the target stimulus) is spatially non-overlapping with S2 (the masking stimulus), but is closely spatially adjacent. Typically, when contrasts and durations of S1 and S2 are compatible, with very short SOAs S1 and S2 are both perceived as if they form a single, composite object. The BAC firing based L/D interaction explanation is similar to what was used for mutual masking with shortest SOAs as described above. Simply because in metacontrast S1 and S2 are not spatially superimposed, the common formed perceptual image allows good discrimination of mask and target features spatially. It is easy to envisage a small neural network where the distinct spatial location cues dictate that modulations executed by the associative presynaptic input to the apical dendritic compartment is bifurcated to different neurons in parallel due to the difference in the receptive fields which is an important feature of stimulation. With intermediate SOAs (e.g., 40–70 ms) S1 is again strongly masked while S2 is well perceived – metacontrast masking is effective. The theoretical explanation is similar to the one presented when explaining pattern- and mutual masking. Moreover, as the shapes of S1 perimeter and S2 internal part surrounding S1 shape are identical or very similar in most of the metacontrast stimuli setups, surface quality of S2 (i.e., the metacontrast mask such as a ring for example) becomes preceptually represented combined with S1 shape which lacks its surface quality and therefore is shown as being"emptied"from this quale.

Explaining masking based on L-system and D-system interaction is not a recent "invention." For example, Bachmann (1984, 1997) in his perceptual retouch theory interpreted masking as a result of L-system-mediated modulation of mask-specific signals up to the level of conscious experience by simultaneously depriving the target-specific signals of a sufficient level of this kind of modulation. Because L-system effecs at the cortical level of the D-system nodes are delayed compared to the initial Dsystem effects, the boost of L-system modulation ignited by target presentation is maximized exactly when the freshly ariving D-system signals carrying mask content arrive cortical neurons dedicated to encode contents. As the signal-to-noise ratio of the mask-related specific afferent activity is higher than the specific target-related activity when the boost of L-modulation arrives, mask-related net activity combining the presynaptic D-input and L-input effects wins over the target-related net activity (targetrelated neural activity has decayed already when L-input arrives, but mask-related activity is maximized at that moment). Modulated post-synaptic EPSPs (Bachmann, 1994, 1997) and N1 components of ERPs (Bachmann, 1984) were used to illustrate the hypothetical D-system and L-system interactive effects producing psychophysical effects of visual awareness. Results of more recent studies with directly measuring ERPs and spectral EEG perturbations and using contrastive analysis with invariant physical stimulation conditions also show that NCC in masking includes a relatively late brain activity (Aru and Bachmann, 2009a,b).

A psychophysical paradigm called *perceptual latency priming* (PLP) has helped to demonstrate that there is a relative latency advantage (i.e., earlier perception) of a visual stimulus that is preceded by another, masked stimulus at its location (Bachmann, 1989; Neumann and Scharlau, 2007; Scharlau, 2007). The first stimulus accelerates perception of the second stimulus even if the first stimulus is backward-masked by the second one up to a total invisibility for direct awareness. In the control condition S2 is presented alone and the temporal delay of its perception is measured psychophysically (e.g., temporal order judgment against a reference stimulus). In the main condition, S2 is preceded by a priming stimulus S1 (spatially adjacent or overlapping with S2). As a result of priming, subjective delay is shortened compared to the control condition without priming. (The effect is obtained also when the prime remains unconscious and is masked.) L- and D-systems interaction-based explanation capitalizing on BAC firing neuronal mechanism assumes that S1 induces the processes consisting in presynaptic somatic input to the neurons encoding its features and a temporally delayed associative (collateral and/or cortico-thalamic) presynaptic input to apical tufts of the neurons that represent attributes of the stimuli associated with the prime in an associative network (including S2 related neurons). Spatial location and approximate size are among the important attributes shared by the prime (S1) and the target (S2). This delayed apical presynaptic input sent via the L-system appears to be temporally coincident with S2 related early somatic-compartment activity in the neurons representing S2 (this being brought about via the Dsystem). As a result, a plateau wave with a burst of spiking will be an associative medium for temporal binding of S1-evoked late activity and S2-evoked early activity. Because the apical-compartment input for the S2-representing neurons is unusually early (due to

the preceding S1-evoked associative activity), already the very first somatic presynaptic input to the S2 neurons finds unusually early coincidence with the apical calcium spikes, which means that S2 is consciously perceived unusually fast.

*Object substitution masking* (OSM) can be explained in a similar vein. In OSM, if a backward mask does not cover the target in space and is spatially and form-wise sparse (e.g., four dots surrounding a target image such as a landolt C), the masking effect is absent. However, when the same target and mask are presented among the spatially distributed distractor objects (with the subject not knowing beforehand where the target is located), with the mask specifying which object is the target and when mask offset is considerably delayed after target offset (a simultaneous onset, asynchronous offset display), strong masking occurs (Enns and Di Lollo, 1997; Di Lollo et al., 2000). In OSM, traditionally weak masks have strong effects when attention is not focused on the target before its presentation. The BAC firing mechanism could be used to suggest a similar explanation to what was presented for backward masking and mutual masking. However, it is necessary to explain why masking is absent in the condition without distractors and occurs when distractors are presented. It is natural to expect that the temporally delayed associative top-down modulation targeting presynaptically the layer-5 neurons' dendrites and igniting the BAC firing mechanism is susceptible to the lateral inhibitory influences mediated by the inhibitory synaptic effects in the dendrites' tuft compartment. If the early-onset OSM display contains many competing stimuli presented simultaneously from the several spatially separated locations, the subsequent delayed associative modulation of the target-related neurons will be inhibited by the lateral afferents from the formerly activated neurons representing the spatially remote competing stimuli. Basically, OSM results from the main share of the L-system-mediated modulation being used by the mask D-system representation instead of the target D-system representation.

In the*motion-induced blindness* displays a few static yellow disks otherwise clearly perceptible become extinguished from awareness from time to time, provided that they are viewed on the background of a moving texture or a set of moving random dots (Bonneh et al., 2001). The BAC firing mechanism-based explanation: two higher level representations – static disks versus moving set of small items – compete for being "serviced" by the top-down modulation by the L-system targeted at the tuft compartment of their respective neurons. Although somatic D-system responses to presynaptic input are preserved all the time for both types of stimuli, dendritic apical compartment afference through the L-system is unstable and fails at times. As a result, discs temporarily fade out of awareness. Competition at the higher level between two classes of neurons representing two different actual perceptual events (one showing static objects and the other a dynamic background) does not allow to send sufficiently steady and efficient top-down signals simultaneously to the dendrites of the lower level neurons that represent local cues of these two types of events.

*Ganzfeld effects*. When a subject stares at an empty field with homogeneous coloring (no brightness gradients and contours in it) for a long time, sooner or later an experience of "fading out" or "emptiness" of the visual sense occurs (Avant, 1965; Wackermann et al., 2008). Colorful visual experience disappears. The explanation founded on the L/D systems' interaction implemented by the BAC firing mechanism assumes this: without almost any specific input that would "recruit" layer-5 pyramidal neurons by somatic-compartment presynaptic input (there is simply not enough sensory input) being available, apical-compartment directed modulation by the L-system becomes without its necessary somatic counterpart of presynaptic input and as a result, any sufficient amount of the plateau-waves is absent. Awareness "gets visually empty." This interpretation invites another assumption. The top-down modulating signals from the higher level are generated insofar as there is a certain minumum amount of objectand feature-specific variability in the perceived environment.

(Hypnotic states resulting from sensory input monotony and lack of variability are also a common reality. Conversely, hallucinatory experiences so common to Ganzfeld experiments suggest a dominance of the top-down or associative input mediated by the dendritic calcium sub-mechanism in predetermining the contents of consciousness in comparison with the input from the actual sensory environment.) How exactly this principle is implemented in the brain remains only a matter of speculation for now.

The possible combinations of the D-system and L-system neurons' post-synaptic activity in response to the strength and synchronicity of their presynaptic inputs are schematically illustrated in **Figure 1**. The psychophysical phenomena of variable consciousness can be explained on the basis of the variants of D/L interaction. Variant (i) characterizes conditions for an absence of contentful conscious experience when the D-system input is insufficient or asynchronous with the L-system despite a strong contribution from the latter. Variant (ii) shows how conscious experience is absent when the L-system contribution is sub-par and/or asynchronous despite the normal contribution of the Dsystem. Variant (iii) illustrates the emergence of the interactivity between D- and L systems sufficient for conscious awareness as a result of strong enough and synchronized input.

#### **CONCLUSION**

In this article, we presented a theoretical perspective, supported by variable empirical data, integrating the two traditions of NCC research which have been developing in relative mutual isolation. Tradition-1 has been examining consciousness state systems with their central role in regulating the level of consciousness, but largely ignoring or overlooking, or being unable to study in detail the functioning and expression of the system of specific contents. Tradition-2, on the other hand, has produced a vast amount of information about the system of processing contents of consciousness and its activity signatures, but has largely ignored the involvement of the state/level system. Importantly, there is enough scientific knowledge about the build and expression of function of the neural mechanisms subservig each of these two functions but unfortunately, there has been insufficient research simultaneously investigating the contributions of the mechanisms serving the two functions within the same experiment in the context of contrastive analysis. We suggest an integration of these two paradigmatic perspectives whereby the objective measurements of the processes of the two neural systems – the state/level system and the contents system – are carried out in experiments where the confounds from stimulus variability are avoided and the problems emerging from the need to distinguish NCCpr, NCC, and NCCae are minimized.

We describe the two neural systems – the data-representng D-system and the level-modulating L-system – and characterize their working parameters and regularities. We suggest a heuristic "formula" C = L × D enabling to combine the two necessary subsystems for consciousness jointly sufficient for producing the subjectve experience with its specific content and varyig in the level of expression of this content, spanning from unconscious to fully conscious and stable phenomenal representation. Importantly, several measurable neurobiological, temporal, and spatial

parameters of functioning of the two systems allow us to explain many experimental facts as a result of interaction of the Land D-systems in real time. Our conceptualization is consistent with several well known approaches and accepted principles in consciousness studes such as the importance of top-down, backpropagating information flow in the brain for consciousness, the temporally delayed nature of phenomenal experience compared to unconscious specific information processing and the importance of the long-range connectivity and global integration of information for consciousness. Among other aspects of the present theoretical view, it must be emphasized that because the L-systembased modulation targets superficial cortical layers and thus largely contributes to negativity and because long-range connectivity which is principally significant for the information integration theory of Tononi and associates also implies apical presynaptic input, the SCP/negativity hypothesis, info-integration theory, and Larkum's BAC firing mechanism seem highly mutually consistent. We also think that our approach has merit because the neural subcellular level, cellular-level, local field potential level, local-circuit level and global-connectivity level aspects seem to be harmonized within the same conceptual framework, which at the same time, does not necessarily remain merely speculative but can be tested and falsified by specific experiments combining brain imaging and psychophysical experiments. We believe that C = L × D helps better specify the NCC, especially by objectively measuring and differentiating the relative contribution of the mechanistically distinguishable subcomponents of the brain involved in producing the astonishingly rich and often hearthbreakingly beautiful phenomenal view of the world.

#### **ACKNOWLEDGMENTS**

The research leading to this publication was partly supported by the Estonian Science Agency, institutional financing IUT20-40 (TSHPH14140I) and by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number R01-GM103894. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

#### **REFERENCES**


Hudetz, A. G., and Alkire, M. T. (2011). "Thalamocortical system and anestheticinduced unconsciousness," in *Neuroscientific Foundations of Anesthesiology*, eds G. A. Mashour and R. Lydic (New York: Oxford University Press), 45–61.


activation of functionally diverse neuronal populations. *Neuroimage* 85, 711–720. doi: 10.1016/j.neuroimage.2013.08.070


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 16 June 2014; accepted: 06 August 2014; published online: 22 August 2014. Citation: Bachmann T and Hudetz AG (2014) It is time to combine the two main traditions in the research on the neural correlates of consciousness: C* = *L* × *D. Front. Psychol. 5:940. doi: 10.3389/fpsyg.2014.00940*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Bachmann and Hudetz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Closing in on the constitution of consciousness

## **Steven M. Miller 1,2\***

<sup>1</sup> Monash Alfred Psychiatry Research Centre, Monash University Central Clinical School and The Alfred Hospital, Melbourne, VIC, Australia <sup>2</sup> School of Psychological Sciences, Monash University, Melbourne, VIC, Australia

#### **Edited by:**

Talis Bachmann, University of Tartu, Estonia

#### **Reviewed by:**

Alexander Maier, Vanderbilt University, USA Jaan Aru, University of Tartu, Estonia

**\*Correspondence:**

Steven M. Miller, Monash Alfred Psychiatry Research Centre, Level 4, 607 St Kilda Road, Melbourne, VIC 3004, Australia e-mail: steven.miller@monash.edu The science of consciousness is a nascent and thriving field of research that is founded on identifying the minimally sufficient neural correlates of consciousness. However, I have argued that it is the neural constitution of consciousness that science seeks to understand and that there are no evident strategies for distinguishing the correlates and constitution of (phenomenal) consciousness. Here I review this correlation/constitution distinction problem and challenge the existing foundations of consciousness science. I present the main analyses from a longer paper in press on this issue, focusing on recording, inhibition, stimulation, and combined inhibition/stimulation strategies, including proposal of the Jenga analogy to illustrate why identifying the minimally sufficient neural correlates of consciousness should not be considered the ultimate target of consciousness science. Thereafter I suggest that while combined inhibition and stimulation strategies might identify some constitutive neural activities—indeed minimally sufficient constitutive neural activities—such strategies fail to identify the whole neural constitution of consciousness and thus the correlation/constitution distinction problem is not fully solved. Various clarifications, potential objections and related scientific and philosophical issues are also discussed and I conclude by proposing new foundational claims for consciousness science.

**Keywords: consciousness, neural correlates, constitution, binocular rivalry, philosophy of mind, mechanistic explanation, foundations, philosophy of science**

## **HISTORICAL CONTEXT OF THE PROBLEM**

The science of consciousness is founded on searching for the neural correlates of consciousness (Crick and Koch, 1990, 1998; Crick, 1994; usually referred to as "NCC" but here as "NCrC," unless quoting directly). Specifically, consciousness science seeks to identify the *minimally sufficient* NCrC. Chalmers (2000) provided a foundational work concerning this notion, in a paper entitled, "What is a neural correlate of consciousness?" His paper is widely cited, as is his definition of the NCrC (p. 31):

An NCC is a minimal neural system N such that there is a mapping from states of N to states of consciousness, where a given state of N is sufficient, under conditions C, for the corresponding state of consciousness.

Also widely cited is the shorter definition by Koch (2004, p. 16) in which the NCrC is considered to be the "minimal set of neuronal events and mechanisms jointly sufficient for a specific conscious percept." Although there has been some discussion of Chalmers' definitions and claims, and the methodological assumptions on which they are based (e.g., Noë and Thompson, 2004; Bayne, 2007; Hohwy, 2007, 2009; Neisser, 2012; Bayne and Hohwy, 2013), there has been less scrutiny than might be expected for a foundational work in a nascent scientific discipline.

A few years ago, two groups of scientists (Aru et al., 2012; de Graaf et al., 2012) independently parsed NCrCs into "NCrC substrate" or "NCrC proper" on the one hand (the construct of interest for consciousness science) and "NCrC precursors/prerequisites" and "NCrC consequences" on the other (constructs of lesser interest). Their partition was aimed at developing strategies to distinguish these various NCrCs and this is of course, the topic of this special issue. In proposing this terminology, however, these authors were unaware that the problem of identifying which, from among many, NCrCs<sup>1</sup> are most directly relevant to consciousness had been appreciated, named and analyzed a decade prior.

Both Revonsuo (2000, 2001, 2006, in press) and myself (Miller, 2001, 2007, 2013a, in press-a,b) have been explicitly concerned with just how science will distinguish the neural *correlates* and the neural *constitution* of consciousness (NCnC). This is particularly problematic for *phenomenal* consciousness—the subjective or qualitative nature of our conscious states (Nagel, 1974). Revonsuo's (2000, 2001) concern was that current scientific methodologies—brain recording techniques in particular—are not capable of targeting the right level of organization in the brain (which he termed, "the phenomenal level"). Revonsuo (2006, in press) also developed a highly detailed level-based biological framework for a consciousness research program that seeks to discover the *constitutive mechanisms* of consciousness.

<sup>1</sup>Note that the plural "NCrCs" indicates the many individual neural activities that correlate with a specific state of consciousness. This is different to "the NCrC" which is a collective term commonly used to denote the set of all NCrCs for such a state. More points of clarification and definition are detailed below.

Although I have a great deal of affinity with Revonsuo's concerns and his framework, my development of the issue has been different. I noted (Miller, 2001) that although the problem of identifying the constitution of consciousness had been *alluded to* by investigators in the science of consciousness, such as Francis Crick, Christof Koch, and Nikos Logothetis (Crick, 1994; Crick and Koch, 1998; Logothetis, 1998) 2 , it required much more than such passing reference. I therefore sought to name and examine the *correlation/constitution distinction problem* (herewith, Cr/Cn distinction problem) so that it might be addressed *explicitly*, with a view to its solution or dissolution, or indeed to its acceptance as an ultimate epistemic limit.

I explicated the Cr/Cn distinction problem utilizing the phenomenon of binocular rivalry (Miller, 2001). Unlike Revonsuo, my initial analysis led me to be concerned that perhaps even future scientific methodologies, including those targeting the right level of organization in the brain, might fail to solve the problem. I suggested there were no obvious corollaries regarding the NCnC from studies of the NCrC and yet it was the NCnC that a science of consciousness should ultimately wish to identify. I therefore proposed that consciousness science might require entirely new scientific strategies if it is to move from mere correlation to actual constitution. My initial concerns were re-stated and developed in a subsequent paper (Miller, 2007), in which I noted (p. 161, italics in original):

If we imagine that through the employment of all current and future neuroscientific methods (in all contexts, under all conditions and with all methodological constraints overcome), we were able to obtain a complete, real-time and multimodal description of all the NCC and all observable properties of such, would we be satisfied that we had obtained a comprehensive understanding of the neuroscience of consciousness? I assert not, because *not every neural correlate of a conscious state is necessarily constitutive of that state.*

In both my 2001 and 2007 papers, I discussed how NCrC recording, inhibition (disablement), and stimulation techniques—either alone or in combination—failed to distinguish correlated but non-constitutive neural activities that are *upstream* or *downstream* from correlated constitutive neural activities (Miller, 2001, 2007). (This upstream/downstream terminology is equivalent to the precursors/consequences terminology and is also used by others; e.g., Chalmers, 2000; Hohwy and Bayne, in press.)<sup>3</sup> I considered therefore, that these empirical approaches failed to yield conclusions regarding the NCnC, that entirely new approaches might be needed and that the Cr/Cn distinction problem might need to join other well-known hard problems of consciousness. I also made it clear, however, that there were no grounds, as yet, to proclaim the problem intractable (Miller, 2007).

#### **CURRENT STATUS OF THE PROBLEM**

Discussion of the Cr/Cn distinction problem has since gained momentum, evidenced by this special issue and by a two-volume project in which scientists and philosophers discuss the problem and the conceptual, empirical, and philosophical territory within which it is situated. In the first volume (Miller, 2013a), scientific groundwork was laid with papers on the brain and visual system's constituents, organization and processes, on the current status of binocular rivalry research from multiple empirical perspectives and on current neuroscientific investigative techniques (including various invasive and non-invasive recording, inhibition, and stimulation techniques).

The second volume (Miller, in press-a) addresses scientific and philosophic perspectives on consciousness science and its methods and foundational constructs, the Cr/Cn distinction problem, the philosophical territory of phenomenal consciousness, hard problems of consciousness, the notion of explanation in consciousness science, the relation between brain and mind, and in particular, notions of correlation, constitution, identity, causation, supervenience, emergence, and realization. My paper in that second volume (Miller, in press-b) closely examines Chalmers' (2000) foundational notion of the minimally sufficient NCrC and through that analysis suggests new foundational claims for consciousness science.

In the present paper—given the topic of this special issue— I outline the specifically *methodological* aspects from Miller (in press-b). I start by providing brief reference to foundational issues in consciousness science and then discuss neural inhibition approaches that challenge those issues. Thereafter I discuss neural stimulation and the combined inhibition/stimulation approach, suggesting that although the latter may provide the best evidence for identifying at least some constitutive neural activities, it fails to fully solve the Cr/Cn distinction problem. Various points of clarification are then discussed and six objections to the presented arguments are listed, with three discussed in detail. A brief presentation of related scientific and philosophic issues is then provided and I conclude with the proposed new foundational claims for consciousness science.

#### **A BRIEF SCAN OF THE FOUNDATIONS**

The Cr/Cn distinction problem is neatly exposed when considering NCrCs during binocular rivalry. This visual phenomenon in which *dynamic* perceptual alternations are induced by *static* presentation of a different image to each eye—provides several advantages for the scientific study of consciousness (reviewed in Miller, 2013b). In particular, binocular rivalry allows for dissociation between neural activity correlated with a subject's perceptual alternations and neural activity correlated with image presentation. Thus, *perception-dependent* neural activity is rightly considered a neural correlate of visual consciousness during rivalry. However, the perception-dependent data for rivalry from electrophysiological and brain-imaging *recording* studies (reviewed in many chapters in Miller, 2013a) yield a wide array of such NCrCs. This *makes evident* the Cr/Cn distinction problem. That is, because not every NCrC is necessarily constitutive of that conscious state, we can ask *which* NCrCs from this array are actually constitutive. Moreover, we can ask what methodologies science

<sup>2</sup> I expect the problem to have been alluded to by many other authors as well, as I continue to discover (e.g., Bachmann, 2000, p. 125; Frith et al., 1999, p. 109). 3 It should also be noted that in addition to correlated non-constitutive activities upstream and downstream from correlated constitutive activities, there can be correlated non-constitutive activities that are *parallel* to correlated constitutive activities. For simplicity, in the arguments that follow I do not consider the parallel category further.

might employ to experimentally examine various hypotheses in this regard.

In Chalmers' (2000) examination of the NCrC notion, he discusses the range of cases over which a correlation should be expected to hold. Those discussed include the normal brain, unusual input (such as binocular rivalry), lesion studies, and stimulation studies. Chalmers' analysis is predominantly conceptual rather than a detailed methodological approach to identifying the NCrC, but he does discuss the lesion case in some depth. He notes that such studies should be regarded very cautiously and perhaps abandoned altogether in searching for the NCrC, due to the altered brain architecture they induce (thus suggesting that an NCrC should be architecture-dependent not architecture-independent)<sup>4</sup> . Chalmers discusses in far less detail, the methodological NCrC approach to the normal brain, unusual input, and brain stimulation. He notes nonetheless that there are interpretive complexities with unusual input and brain stimulation too, but he considers these cases and the normal brain case to be those over which an NCrC should be required to hold (with perhaps some "good" lesion approaches also being admitted). The present analyses aim to address in detail, inhibition and stimulation empirical strategies—and their interpretive complexities—that surround the notion of the minimally sufficient NCrC.

There are several other aspects to Chalmers' (2000) formulation of the NCrC construct that are worth mentioning. First, his definition is intended to constrain the discussion to "correlation" terminology, as he considers this to be theoretically neutral rather than theoretically loaded. Second, he notes his definition is constructed in a way that provides a tractable methodology for NCrC identification, and thus a way forward for consciousness science. Third, he accepts that even if the thus-defined NCrC was identified, this would not necessarily *explain* consciousness and may not even be the key to understanding processes underlying consciousness. However, he does not *explicitly* consider the potential for a Cr/Cn distinction problem, perhaps because of his reluctance to shift the discussion beyond notions of correlation. Moreover, although his formulation of the minimally sufficient NCrC construct acknowledges the empirical potential it provides, it does not adequately examine the empirical entailment of the "minimally sufficient" qualifier or the methodological details of such entailment.

As outlined in greater detail in Miller (in press-b), despite the valuable contribution provided by Chalmers' conceptual proposals, there are several problems with accepting his formulation as unchallenged foundations for consciousness science. Indeed, Chalmers himself considered his work to be "conceptual spadework" that would require refinement. If my contention above is accurate—that consciousness science wishes to ultimately identify the NCnC—then it is noteworthy that this construct is nowhere to be seen in Chalmers' analysis. It is a construct that can be conveyed using a wide range of terms such as the neural "basis," "mechanism," or "substrate" of consciousness (see Miller, 2007 for more terms), so concerns over the entailed philosophical commitments of "constitution" terminology should not be reason to avoid such a construct, or something like it<sup>5</sup> . Moreover, once we admit talk of such notions, questions arise regarding: (i) whether by minimally sufficient NCrC, we mean the very same thing as the neural basis, mechanism, substrate, or constitution of consciousness; (ii) whether these constructs pick out the same or different neural activity sets; and (iii) if they could pick out *different* neural activity sets, which should be considered the ultimate empirical target for consciousness science.

### **STEPWISE INHIBITION**

The notion of the minimally sufficient NCrC was created to distinguish it from the *merely* sufficient NCrC (with a *necessity* criterion being considered altogether too strong; Chalmers, 2000). However, the distinction between the merely and minimally sufficient NCrC *entails* an empirical strategy. If we are to arrive at identification of the minimally sufficient NCrC, we will presumably need to *remove* NCrCs one by one to assess whether each has minimally sufficient status. It is critical here to note that when I refer to an NCrC being minimally sufficient, or having minimally sufficient status, I mean to say that it is a *part* of the whole minimally sufficient neural activity set. Similarly, when I refer to an NCrC being constitutive, or having constitutive status, I mean to say that it is a *part* of the whole NCnC. Returning to the empirical approach to reducing the merely sufficient NCrC to the minimally sufficient NCrC, the most obvious means of doing this is by *stepwise inhibition* (disablement) of neurons, neuron types<sup>6</sup> ,

<sup>4</sup>Although Chalmers expresses caution over the lesion analysis case, no one would deny that modern understanding of the brain and its functional specialization owes a great debt to such neuropsychological analysis. Phenomenological dysfunction following destruction of certain brain regions by cerebrovascular insult, brain tumors, traumatic brain injury or surgery has informed mechanistic understanding of normal brain function enormously (Damasio and Damasio, 1989; Howieson et al., 2012), as has direct electrical stimulation of the brain (Borchers et al., 2012; see Stepwise Stimulation and Combined Inhibition/Stimulation).

<sup>5</sup>There are differences between the terms "basis," "mechanism", "substrate," and "constitution," however, in most of what follows, I use these terms synonymously. Hence, by "NCnC" I mean to equally imply the neural "basis," "mechanism," or "substrate" of consciousness. At times I will state these terms in full to remind the reader of this point and indeed, on this broad construal, one could even "define" the NCnC as the neural basis, mechanism, or substrate of consciousness. However, although the Cr/Cn distinction problem is able to be appreciated and analyzed using any of these broadly synonymous terms, use of "constitution" terminology becomes relevant and informative later, as a specific relation within philosophy of mind (with additional implications for science thereafter). It is because of the relational complexities in the philosophical domain that I resist any further simple definition of the NCnC. For example, a definition could be proposed that states the NCnC is the neural activity that simply *is* the conscious state in question, but any such definition then invites debate about contrasting the "is of identity" with the "is of constitution" (see Miller, 2007, and various papers in Miller, in press-a).

<sup>6</sup>On the issue of neuron types, Koch (2004, p. 282) refers to perceptiondependent neural activity high in the visual processing pathway (inferior temporal cortex, IT, and superior temporal sulcus, STS) during binocular rivalry as follows:

It is implausible that all of the storied IT and STS neurons that follow the percept express its phenomenological attributes directly. Some must be involved in the underlying winner-take-all operations; others must relay the winner's identity to the motor centers to initiate behavior or to short-term memory for future recall; some must carry a transient signal indicative of

local or distributed neural circuits, specific brain regions or sets of NCrCs.

Chalmers' definitions would seem to imply something like a stepwise inhibition empirical strategy when he comments (2000, p. 25), "In this way, we pare down any potential NCC to its core: Any irrelevant material will be whittled away, and an NCC will be required to contain only the core processes that suffice for the conscious state in question." There is some conflict in Chalmers' view here, given his willingness to whittle away but not to lesion. We therefore need to qualify the *type* of inhibition to be employed in identifying the minimally sufficient NCrC, and select methods that do not induce gross lesions and consequent architectural disruption (such as that induced by stroke, tumor, injury, or surgery). Although current methods such as inhibitory transcranial magnetic stimulation could be employed—and indeed this method provides striking and immediate perceptual disruption during binocular rivalry (Miller et al., 2000; see also Ngo et al., 2013)—for conceptual clarity we might postulate future highly specific molecular knockout techniques in which this or that NCrC can be selectively and reversibly inhibited. Such techniques were predicted by Crick and Koch (1998; see also the quote further below from Fenno et al., 2011) and are *currently* being developed and refined, with stunning progress, in the field of optogenetics (as detailed further below; Fenno et al., 2011; see also Klink et al., 2013). Although harmless application of such techniques in humans remains a very long way off, we can nonetheless begin to think through how such highly selective inhibitory techniques could, *in principle*, assist with identification of the NCnC.

As I have previously noted (Miller, 2001, 2007), if we inhibit an NCrC and consciousness disappears (or degrades), this might suggest the inhibited NCrC *is* constitutive in the normal case but does not actually prove its constitutive status. This is because such an NCrC might simply be *necessary and supportive* for consciousness without being constitutive of it. Conversely, if we inhibit an NCrC and consciousness does not disappear (or degrade), this might suggest the inhibited NCrC is *non*-constitutive in the normal case but does not prove such non-constitutive status. This is because such an NCrC might simply be *redundantly* constitutive. Because of these uncertainties, I have claimed that stepwise inhibition does not lead to conclusions regarding the NCnC. However, it certainly *does* lead to conclusions regarding the minimally sufficient NCrC.

#### **THE JENGA ANALOGY**

To illustrate how the minimally sufficient NCrC and the NCnC are constructs that can pick out different neural activity sets, and to schematize the stepwise inhibition approach described above, I have proposed the Jenga analogy (Miller, in press-b). This analogy is from the popular game in which blocks are removed from a tower structure, one by one, until eventually a *critical point* is reached and the tower falls. An upright tower in this analogy (**Figure 1**) represents a *specific* conscious state or content being present while the fallen tower represents the absence (or degradation) of that conscious state or content. Each block in the tower represents an NCrC that correlates with that specific conscious state or content. This follows the distinction by Koch (2004) between "specific factors" and "enabling factors," with the former dealing with particular conscious states or content and the latter with the overall state of being conscious. Although this distinction is itself the subject of considerable conceptual and methodological controversy (see Noë and Thompson, 2004; Bayne, 2007; Hohwy, 2007, 2009; Neisser, 2012; Bayne and Hohwy, 2013; Hohwy and Bayne, in press), I sidestep this debate and constrain the ensuing discussion to just specific factors<sup>7</sup> .

As the case of binocular rivalry illustrates (**Figure 2A**; see Miller, 2013a, in press-b), there are many (specific factor) NCrCs for a given conscious state and these can include specific brain regions or specific neural populations within brain regions. In the Jenga analogy, each NCrC block can thus be considered representative of a specific factor NCrC at either local, distributed or regional levels<sup>8</sup> . The first step in the stepwise inhibition strategy is to use recording techniques to create an NCrC specific factor map for a specific (target) conscious state. Several further stipulations for the Jenga analogy are required. First, the entire tower is considered to be the outcome of previous strategies to "screen off " irrelevant *neural* activities (Hohwy, 2009; Hohwy and Bayne, in press). Hence, on this analogy, already screened off would be (i) neural activities that do *not* correlate with specific conscious states or content; (ii) neural activities that correlate with specific conscious states or content, but do so only in a *loose* fashion, in which the correlation can be broken one way or another<sup>9</sup> ; (iii) neural activities that correlate with specific conscious states or content but whose precise *timing* provides clear ascription of either upstream or downstream, rather than constitutive, status (see Miller, in press-b for details;

a perceptual switch; and still others might represent the same information but in a delayed manner.

When considering the temporal profile of cellular responses in these areas, I am struck by their extreme heterogeneity. An entire menagerie of distinct patterns can be observed. Some cells fire in a transient manner whereas others respond in a more sustained fashion. Some fire in bursts, some show a pronounced, rhythmic discharge in the 4–6 Hz range, while others peak early before settling down to a more sedate and sustained pace of firing. Do these reflect discrete cell types with discrete functions and connectivity patterns? This will be important to know.

<sup>7</sup>Bayne and Hohwy (2013; see also Hohwy and Bayne, in press) refer to *differentiating* NCrCs, which are considered neural states that are selectively implicated in particular kinds of conscious states in an already conscious creature.

<sup>8</sup>As neurophysiological understanding develops, so too will the notion of what an NCrC block could be taken to be in this analogy—that is, what appropriate "units" of neurophysiological signaling will turn out to be. For now, however, each block represents individual NCrCs such as those being identified at various levels of constituents, organization and processes in studies of binocular rivalry.

<sup>9</sup> In my view, and without wishing to undermine or dismiss these important conceptual and methodological efforts, I consider the strategies so far proposed to distinguish NCrC precursors/prerequisites, NCrC proper/substrate, and NCrC consequences by de Graaf and Sack (in press) and Aru et al. (in press) generally fall into this category, appealing to ever more refined means of identifying loosely from tightly correlated neural activities. This refinement certainly helps to close in on the constitution of consciousness but the Jenga analogy begins only *after* application of such strategies.

see also Aru et al., in press; de Graaf and Sack, in press; Hohwy and Bayne, in press; Revonsuo, in press) <sup>10</sup>; and (iv) neural activities that correlate with specific conscious states or content but whose known mechanistic functions provide grounds for clear ascription as non-constitutive11. What remains after such a screening off process therefore, is a tower of neural activities that *tightly* correlate with a specific conscious state or content and whose precise timing or known mechanistic functions cannot be used to accurately ascribe upstream, downstream, or constitutive status<sup>12</sup> .

Second, NCrC blocks in the lower third of the tower (colored yellow) represent tightly correlated non-constitutive upstream activities (tightly correlated NCrC precursors or NCrC prerequisites), those in the upper third (colored blue) represent tightly correlated non-constitutive downstream activities (tightly correlated NCrC consequences), and those in the middle third (colored red) represent tightly correlated *constitutive* NCrCs (the NCnC). Third, the upright tower at the critical point beyond which any further block removal will result in its collapse and the disappearance (or degradation) of the target conscious state represents the minimally sufficient NCrC.

Consider now that each of the thus-defined tightly correlated neural activity blocks in the Jenga tower can be described by only *one* of the following conjunctions:


Consider further, the following two possibilities (P), each of which is to be considered in isolation (because complexities arise when they are considered together):

Possibility 1 (P1): Considering all individual NCrCs that have survived the screening off process, the possibility of nonconstitutive minimally sufficient NCrCs means that the minimally sufficient NCrC could be a *larger* set of neural activities than the neural basis, mechanism, substrate, or constitution of consciousness.

Possibility 2 (P2): Imagining just the neural basis, mechanism, substrate, or constitution of consciousness, the possibility of redundancy (the possibility of constitutive non-minimally sufficient NCrCs) means that the minimally sufficient NCrC could be a *smaller* set of neural activities than the neural basis, mechanism, substrate, or constitution of consciousness.

Suppose now, that the first five correlated non-constitutive upstream activity blocks are removed but the conscious state remains and the tower stays upright. Those NCrCs are therefore *non*-minimally sufficient. Further suppose, however, that removal of the sixth correlated non-constitutive upstream activity block does lead to the disappearance (or degradation) of the conscious state and to falling of the tower. That sixth NCrC *is* minimally sufficient. But the Jenga analogy shows that an NCrC can be minimally sufficient *without* necessarily being constitutive. In this way, the minimally sufficient NCrC could be a larger set of neural activities than the NCnC (**Figure 1**). The downstream case is a little more complex (see Aru et al., in press; de Graaf and Sack, in press; Hohwy and Bayne, in press; van Boxtel and Tsuchiya, in press), and it is not always clear what should be regarded as an upstream activity and what a downstream activity, particularly given the unclear neurophysiological role of feedback. However, the difference between upstream and downstream cases is not particularly important for the point I am making and P1 is illustrated clearly with reference to just the upstream case.

Next imagine the set of neural activities that is the NCnC and consider that due to the possibility of redundancy in this neural activity set, five NCrC blocks could be removed without the disappearance (or degradation) of the conscious state and with the tower remaining upright. Those five neural activities are therefore *non*-minimally sufficient despite being actually constitutive. Removal of the sixth correlated constitutive neural activity block, however, takes the tower passed its critical point and the conscious state disappears (or degrades) and the tower falls. This sixth NCrC block then is *both* constitutive and minimally sufficient. In this way, the minimally sufficient NCrC within this (imagined) constitutive neural activity set could be a smaller set of neural activities than the NCnC (**Figure 1**).

The issue of claiming larger versus smaller sets is complicated when conceiving of P1 and P2 *together*, but the key message here is not about the overall size of the different neural activity sets, but rather that the minimally sufficient NCrC and the NCnC *can be different sets of neural activities*. Other important complexities include combinatorial and order complexities (i.e., whether the critical block removal would be critical *whenever* it is removed, or only when removed *after* removal of the previous five blocks, or only after removal of the previous five blocks in *that* specific order). Despite these complexities, what is important here is that while the empirical strategy of stepwise inhibition can identify an NCrC's minimally sufficient status, it cannot identify its constitutive status. As the Jenga analogy shows, and as depicted in **Figure 2** using the case of binocular rivalry, while stepwise inhibition achieves identification of the minimally sufficient NCrC, it cannot identify (i) which of the *remaining* blocks are constitutive of the conscious state and which are not; or (ii) which of the

<sup>10</sup>With such a brief treatment here, I do not wish to downplay the importance and complexity of timing in the brain with respect to correlations between neural and phenomenal states. This issue is itself likely to be a source of interpretive difficulty and hence there will be many cases in which such "clear ascription" will not be possible. Indeed, this fact underscores the problems being discussed in the present paper. Nonetheless, in cases for which clear ascription appears possible, timing can be used to screen off candidates for constitutive neural activity. For a detailed source on the issue of timing and phenomenal consciousness, see Bachmann (2000) and for a more recent example of timing studies, see Liu et al. (2009).

<sup>11</sup>I provided an example of this fourth category in my first explication of the Cr/Cn distinction problem (Miller, 2001) in terms of the expected correlated neural activity that would be found in subcortical nuclei controlling eye movement tracking (optokinetic nystagmus) during rivalry with drifting gratings. Despite being correlated with consciousness during such rivalry, the known mechanistic function of eye movement nuclei suggests a nonconstitutive role for this individual NCrC.

<sup>12</sup>Herewith, I combine points (ii), (iii), and (iv) such that when I refer to *tightly* correlated NCrCs, I mean to also include the precise timing and mechanistic function issues.

**FIGURE 1 | The Jenga analogy schematizes the stepwise inhibition strategy inherent in distinguishing the merely sufficient from the minimally sufficient NCrC and shows how the minimally sufficient NCrC can differ from the NCnC.** In all panels, the upright tower represents the presence of a specific conscious state (or content) while in panels (**A,B**) the fallen tower (rubble) represents the loss (or degradation) of that state. Each block represents a perception-dependent NCrC that correlates with the specific conscious state (i.e., a specific factor NCrC). The whole tower is what remains after a process of screening off non-correlated neural activities and NCrCs that are only loosely correlated with consciousness or whose precise timing or known mechanistic function indicates they are not candidates for the NCnC. Panels **(A,C)** depict real-world situations, while panels **(B,D,E)** depict colored subdivisions that could only be identified if the Cr/Cn distinction problem is solvable. In **(B,D,E)**, yellow blocks represent tightly correlated non-constitutive upstream activities (NCrC precursors/ prerequisites), blue blocks represent tightly correlated non-constitutive downstream activities (NCrC consequences) and red blocks represent correlated constitutive activities (the NCnC). The right tower in **(B)** represents a more distributed depiction of the colored subdivisions than the left tower in **(B)**. In **(C,D)**, the upright tower with NCrC blocks removed is at the critical point beyond which any further block removal will result in the disappearance (or degradation) of the conscious state and collapse of the tower. The upright tower in **(C,D)** thus represents the minimally sufficient NCrC. Blocks in this critical point tower have minimally sufficient status, while those removed from it do not. In P1 (see main text), the possibility of non-constitutive minimally sufficient NCrCs (remaining yellow and blue blocks in **D**) means that

*removed* blocks are constitutive of the conscious state and which are not<sup>13</sup> .

<sup>13</sup>It is important to note that the Jenga analogy is just that—an analogy. It *schematizes* the stepwise inhibition approach to illustrate the points I am making. There is certainly no implied direct equivalence between removing a block from a tower of mechanically related blocks on the one hand, and inhibiting a neuron, neural circuit or set of neural circuits within an exquisitely complex, dynamic and inter-related neural system with the capacity for functional and structural reorganization, on the other. Aside from differences in complexity between a Jenga tower and a neural system, there are further the minimally sufficient NCrC could be a larger set of neural activities than the neural basis, mechanism, substrate, or constitution of consciousness (i.e., the tower in **D** is a larger set of blocks than the red middle third of the left tower in **B**). In P2 (see main text), the possibility of redundancy in the NCnC (i.e., the possibility of constitutive non-minimally sufficient NCrCs; removed red blocks in **E**) means that the minimally sufficient NCrC could be a smaller set of neural activities than the neural basis, mechanism, substrate, or constitution of consciousness (i.e., the red remaining blocks in the tower in **E** is a smaller set of blocks than the red middle third of the left tower in **B**; note P1 and P2 are not meant to be considered together—see main text). The stepwise inhibition strategy can therefore be used to assign minimally sufficient status to each block in the Jenga tower in **(A)** (thus the real-world situation of getting from **A** to **C**). However, the Cr/Cn distinction problem claims that there are no evident strategies to readily assign color status (constitutive red versus non-constitutive yellow/blue) to each of the NCrC blocks (thus it is not clear how to get from **A** to **B**). The analogy shows that although we might get from **(A)** to **(C)** using the stepwise inhibition approach, this will not satisfy consciousness science because we still cannot assign color status to either the remaining or removed blocks in **(C)** (we cannot get from **C** to **D**). Through this analogy it is claimed that the ultimate target of consciousness science is not to identify the minimally sufficient NCrC, but rather to distinguish constitutive (red) NCrCs from non-constitutive (yellow/blue) NCrCs. That is, it is ultimately the neural basis, mechanism, substrate, or constitution of consciousness that we seek to understand and this construct can pick out a different neural activity set from that of the minimally sufficient NCrC. Figure and caption reprinted with permission from Miller (in press-b).

This is not to say that identifying the minimally sufficient NCrC through stepwise inhibition would be an insignificant

points of difference such as the effect that inhibiting a neuron has on the neuron's milieu and on non-neural structures like nearby glia and vasculature (Logothetis, 2010). Because of these issues, problems of interpretation can arise with respect to whether it is the neural inhibition that is (or is not) changing consciousness or it is the secondary physiological effects of that inhibition. The same interpretive problems arise regarding neural stimulation (discussed in the next section, where further caveats are raised).

**FIGURE 2 | The Cr/Cn distinction problem is clearly illustrated with reference to the phenomenon of binocular rivalry. (A)** Research using electrophysiological (single-unit and local field potential), brain-imaging and brain stimulation (and inhibition) approaches has identified a wide array of NCrCs during binocular rivalry and related phenomena such as flash suppression (for details, see Miller, 2013a, in press-b; for even further electrophysiological NCrCs during rivalry and related phenomena, see Figure 6 in Boly et al., 2013). The existence of multiple NCrCs during rivalry raises the issue of which activities can be said to constitute a specific conscious state rather than being merely correlated with it. There are multiple hypotheses possible for which NCrCs are upstream from (i.e., precursors or prerequisites), which are downstream from (i.e., consequences), and which are constitutive of, consciousness. These competing hypotheses are indicated by the color-coding alternatives in the smaller panels (**B–G**, which follow the color coding of the Jenga analogy in **Figure 1**). While this NCrC array could be subjected to the stepwise inhibition strategy underlying the notion of the minimally sufficient NCrC,

the problem of assigning the appropriate color-coding to each NCrC cannot be solved with the same strategy because of the Cr/Cn distinction problem. There are several caveats to interpreting this figure, including issues such as: not all neurons in each area will exhibit perception-dependent firing (e.g., V1/V2, V4, middle temporal area, MT); even when perceptiondependent in V4 and MT, this includes a proportion of neurons with the opposite expected firing pattern (i.e., lower firing rates when their preferred stimulus is perceived and higher firing rates when their preferred stimulus is suppressed); some of the regions correlate with transitions between rivaling states (or with reporting of such states) rather than with the visual states themselves; some regions are targets of rivalry temporal parameter modulation rather than consciousness modulation per se; regions engaged in attentional selection and top-down modulation could be considered either upstream or downstream activities; and there may be important binding mechanisms that physiologically link individual NCrCs (see main text). Figure reprinted and caption adapted with permission from Miller (in press-b).

achievement for consciousness science. On the contrary, it would be a *major* achievement. However, the problem remains of just how science will experimentally distinguish the neural activity sets of the minimally sufficient NCrC and the NCnC (or if these are in fact the very same sets, how this can be shown to be the case). In my view then, the minimally sufficient NCrC construct should not be considered the ultimate empirical target for consciousness science because it could *include* neural activities that are *not* part of the neural basis, mechanism, substrate, or constitution of consciousness (P1) and it could *exclude* neural activities that *are* (P2). The minimally sufficient NCrC notion, for all its worth, subtly shifts the target of consciousness science to an empirically tractable one, while the real target remains elusive due to the Cr/Cn distinction problem.

## **STEPWISE STIMULATION AND COMBINED INHIBITION/STIMULATION**

If recording strategies make evident the Cr/Cn distinction problem and stepwise inhibition can identify an NCrC's minimally sufficient but not constitutive status, what might the strategy of neural stimulation achieve? And indeed, exactly how would such a strategy be applied in the context of studying consciousness? Perhaps the first distinction required here is that between stimulation of currently *inactive* NCrCs and those *already active*. On the one hand, stimulation of already active correlated neurons, neuron types, local or distributed neural circuits, brain regions or sets of NCrCs, would generally not be expected to change a conscious state (irrespective of whether these stimulated NCrCs are constitutive or not). On the other hand, when stimulating currently *inactive* correlated neurons, neuron types, local or distributed neural circuits, brain regions or sets of NCrCs, a changed conscious state might suggest a constitutive role for such NCrCs but such a role is not proven by this strategy. This is because the stimulated NCrC might in fact be *non*-constitutive with its stimulation simply activating downstream NCrCs that *are* constitutive. In this case there may be conclusions possible regarding the *causal chain* of neural processing for that state, but there are no corollaries regarding the NCnC<sup>14</sup> .

What about *combinations* of inhibition and stimulation? I have previously asserted (Miller, 2007, p. 165) that, "by recording from, disabling and stimulating various NCrCs, there do not seem to be any obvious corollaries regarding the NCnC." However, development of the Jenga analogy now leads me to reassess this assertion. Consider the following experimental scenario—the *reverse* Jenga strategy—which might at least *in principle*, and *partially*, address issues of constitution. This strategy again requires a highly specific and powerful inhibitory, and now also stimulatory, technical capacity, i.e., the ability to selectively inhibit, disinhibit, and stimulate specifically tagged neurons, neuron-types, local or distributed neural circuits, specific brain regions and sets of NCrCs. Note here also that stimulation is a *further* physiological step beyond mere disinhibition to resting state activity. It is also to be noted that the arguments above and below concern inhibition and stimulation of *excitatory* rather than inhibitory neurons, though it is acknowledged that (i) physiological inhibitory neural activity is a fundamental feature of cortical microcircuits and is "electrically inseparable from excitation" (Borchers et al., 2012, p. 66); (ii) neurons that correlate with a specific conscious state by *decreasing* their firing rate are not addressed by these arguments; and (iii) *modulation* of neural activity is not addressed by these arguments. All of these issues (and those in footnote 13) make the arguments I wish to present more complex than is required at this stage, but I do not discount their relevance.

Again optogenetics comes to mind when positing a highly specific and powerful inhibitory and stimulatory technique and it is worth quoting in full, the first two paragraphs of a recent review of this technique (Fenno et al., 2011, p. 390, square brackets in original):

In describing unrealized prerequisites for assembling a general theory of mind, Francis Crick observed that the ability to manipulate individual components of the brain would be needed, requiring "a method by which all neurons of just one type could be inactivated, leaving the others more or less unaltered" (Crick 1979, p. 222). Extracellular electrical manipulation does not readily achieve true inactivation, and even electrical excitation, while allowing for temporal precision in stimulating within a given volume, lacks specificity for cell type. However, pharmacological and genetic manipulations can be specific to cells with certain expression profiles (in the best case) but lack temporal precision on the timescale of neural coding and signaling.

Because no prior technique has achieved both high-temporal and cellular precision within intact mammalian neural tissue, there has been strong pressure to develop a new class of technology. As a result of these efforts, neurons now may be controlled with optogenetics for fast, specific excitation or inhibition within systems as complex as freely moving mammals [for example, with microbial opsin methods, light-induced inward cation currents may be used to depolarize the neuronal membrane and positively modulate firing of action potentials, while optical pumping of chloride ions can induce outwards currents and membrane hyperpolarization, thereby inhibiting spiking (Figure 1)]. These optogenetic tools of microbial origin (Figure 1) may be readily targeted to subpopulations of neurons within heterogeneous tissue and function on a temporal scale commensurate with physiological rates of spiking or critical moments in behavioral tests, with fast deactivation upon cessation of light. With these properties, microbe-derived optogenetic tools fulfill the criterion set forth by Crick in 1979 (Deisseroth 2010, 2011).

<sup>14</sup>This is not the place to discuss in detail the difference between the causation and constitution relation. Suffice to say, I believe the causation relation should be restricted to usage in which one neural activity causes another (hence in the context of causal chain processing) or in the context of mechanistic explanation in which a mechanism or operational part has causal influence on other mechanisms or operational parts (see Craver, 2007 for more on the distinction between causal/etiological and constitutive mechanisms). I do not think the causation relation should be used in the context of the brain– mind relation. That is, brain states constitute mental states, or are identical with them, or realize them, or the relation is one of supervenience, but brain states do not cause mental states. These matters are taken up in much greater detail in various papers in Miller (in press-a). For the view that causation is in fact the relevant brain–mind relation, see Neisser (2012). On the issue of mental causation (i.e., whether mental states could cause brain states and behavior), I do not address this at all, but a constitution, identity, realization or supervenience relation would not preclude mental states causing *other* brain states and consequent behavior—these relations would just preclude mental states causing their *own* brain states.

Others have commented on the prospects for consciousness science offered by optogenetics (e.g., Tononi and Koch, 2008) 15 . The technique has recently been applied in mice to examine top-down modulation of visual processing (Zhang et al., 2014) and can be applied in *Drosophila* in the context of visual rivalry (Miller et al., 2012). For examples of brain stimulation techniques currently applicable in humans, including those already applied or capable of being applied to binocular rivalry, such as transcranial magnetic stimulation, vestibular stimulation techniques, transcranial direct current stimulation, and electrical microstimulation, see Been et al. (2007), Borchers et al. (2012), Cohen and Newsome (2004), Histed et al. (2013), Klink et al. (2013), Law et al. (2013), Ngo et al. (2013), Reppas and Newsome (2007), Sengpiel (2013), Sterzer (2013), and Thomson and Fitzgerald (2013). Despite the value of such techniques for stimulating (and in some cases inhibiting) neural activity, they entail various disadvantages and interpretive complexities such as: (i) whether they in fact cause stimulation or inhibition; (ii) their spatial imprecision and hence unintended effects on other local and regional neural targets; (iii) individual variation, regional variation, and neuronal morphological variation in stimulatory and inhibitory thresholds (and consequent perceptual and behavioral effects); and (iv) the ability of such techniques to be detected by the subject. The in principle "pure" inhibition/stimulation methodology on which the arguments in the present paper are grounded would avoid these problems, as far as is physiologically possible. And of course, knowing just how far this is physiologically possible will require a great deal of further neurophysiological understanding (see, for example, the interpretive cautions outlined by Logothetis, 2010, regarding emerging optogenetic studies).

Nonetheless, with an optogenetics-style technique as an example of the type of in principle methodology to which I am referring, consider that the first step in the reverse Jenga scenario is to use recording techniques to create a tightly correlated NCrC specific factor map for a specific (target) conscious state16. Next, leaving enabling factor NCrCs untouched, *all* previously mapped specific factor NCrCs are inhibited<sup>17</sup> . In addition, all non-correlated causal chain components, and all loosely correlated NCrCs, are inhibited. The crucial final intervention then is to stepwise disinhibit *and* activate (stimulate) each previously mapped tightly correlated specific factor NCrC. Under these circumstances, if the target conscious state is reported<sup>18</sup> then this would seem to provide the strongest evidence possible that the disinhibited and stimulated NCrC is actually constitutive (and thus has constitutive status).

Note that in the case of P1 with the stepwise inhibition strategy, the reason a non-constitutive but minimally sufficient NCrC is minimally sufficient is because of its *input* to the NCnC (i.e., its role in causal chain processing, albeit in this case, a correlated rather than non-correlated causal chain role). Without such minimally sufficient non-constitutive NCrC activity, there could not be the required activity in the NCnC and there could not thus be the conscious state. However, in the reverse Jenga case just described, the NCnC is *directly* activated (stimulated) and thus, unlike the stepwise inhibition case, there is no dependency on minimally sufficient non-constitutive NCrCs. So in the reverse Jenga case, with this dependency condition removed, if the target conscious state appears with particular NCrC stimulation—and with other NCrCs inhibited, given the strategy is a *stepwise* combined inhibition/stimulation process this reasonably implies constitutive status of the stimulated NCrC.

With the reverse Jenga strategy, we are thus able to *build* the middle third of the tower to its critical point of minimal sufficiency. Indeed, by identifying the tower at this critical point of just its middle third, it can be stated that what has in fact been identified is the *minimally sufficient NCnC*<sup>19</sup> . The reverse Jenga approach thus, arguably (see later), provides *partial* solution to the Cr/Cn distinction problem because it overcomes the obstacle of P1 and identifies *some* constitutive neural activities. However, to fully solve the Cr/Cn distinction problem, we need to achieve identification of *the whole set* of constitutive neural activities. To do that would require also overcoming the obstacle of P2 which would involve iden-

<sup>15</sup>Indeed, Tononi and Koch (2008, p. 257) note in their update of the NCrC program:

The growing ability of neuroscientists to manipulate in a reversible, transient, deliberate, and delicate manner identified populations of neurons using methods from molecular biology combined with optical stimulation enables the intrepid neuroengineer to move from correlation—observing that a particular conscious state is associated with some neural or hemodynamic activity—to causation.

While agreeing with such a sentiment, I would argue that the intrepid neuroengineer should wish to go beyond not just the correlates of consciousness, but also beyond causation (see footnote 14), to ultimately reach the constitution of consciousness.

<sup>16</sup>The reverse Jenga strategy could theoretically be applied without a specific factor NCrC map first being identified with recording techniques (though the consequent trial and error task would be far more onerous). Such a scenario might be envisaged if developments in highly precise combined inhibition and stimulation technologies occur more rapidly than developments in highly precise recording technologies. However, this is perhaps unlikely given optogenetics, for example, is not just an inhibition and stimulation technique but is also itself a recording technique. Hence developments in its (safe) recording

capacity are likely to occur more or less concurrently with developments in its (safe) inhibitory and stimulatory capacity.

<sup>17</sup>It is not clear what it would be like for a subject to be in a state in which all specific factor NCrCs were inhibited but all enabling factor NCrCs remained active, but this is an issue I have side-stepped and is debated elsewhere (Noë and Thompson, 2004; Hohwy, 2009; Hohwy and Bayne, in press; Neisser, 2012). Confusion could also arise if one were to describe non-constitutive causal chain specific factor processes as themselves enabling factors, so this should be avoided. Enabling factors should be restricted to those neural activities present for *all* conscious states in a conscious creature.

<sup>18</sup>Neural activity for reporting will thus also need to remain untouched; this of course raises issues of access versus phenomenal consciousness (Block, 1995, 1996; discussed in Klink et al., in press) but again this is not my point here. For more on reporting and downstream NCrCs, see Aru et al. (in press); de Graaf and Sack (in press); Hohwy and Bayne (in press), and van Boxtel and Tsuchiya (in press).

<sup>19</sup>In the case of combined inhibition/stimulation (and indeed, only in this case), the minimally sufficient NCrC and the minimally sufficient NCnC can be considered equivalent constructs.

tifying not just the middle third of the tower at its critical point of minimal sufficiency, but the exact *boundaries* of that (whole) middle third (i.e., the exact boundaries of the NCnC; **Figure 3**).

To show why the reverse Jenga approach fails to fully solve the Cr/Cn distinction problem, consider the following further specific combined inhibition/stimulation scenarios (herewith I refer to the combined inhibition/stimulation approach as "*isolated* stimulation"). First, isolated stimulation of non-constitutive NCrCs in the bottom or top third of the tower—whether they are minimally sufficient NCrCs or not—would *not* induce the target conscious state. Consequently, these NCrCs can be reasonably excluded as constitutive neural activities. Second, after the conscious state first appears due to isolated stimulation of the minimally sufficient NCnC, and then as redundantly constitutive NCrCs are *additionally* disinhibited and stimulated, the conscious state will *not* change, so this will not allow a distinction to be made between redundantly constitutive NCrCs and non-constitutive NCrCs. Hence the whole set of constitutive neural activities cannot be identified.

The next isolated stimulation scenario is also informative. Thus, we can ask what would occur with isolated stimulation of redundantly constitutive NCrCs when the minimally sufficient NCnC is not also stimulated (i.e., whether that would induce the conscious state). Here the issue depends on the nature of the redundancy and the notion of *neural multiple realizability* (see Miller, 2007). That is, it may be that the minimally sufficient NCnC is a *fixed* set of neural activities for a particular conscious state, without activation of which there will never be that conscious state. In such a scenario—which invokes the notion of *necessity* rather than sufficiency—the usual case would involve a conscious state being constituted by activation of that fixed minimally sufficient neural set *and* by any additional redundantly constitutive neural activities. But another alternative is that the minimally sufficient NCnC may be a *variable* set of neural activities for a particular conscious state, such that *separate* isolated stimulation of two or more different sets of neural activities could induce (and constitute) the target conscious state, even though in the normal case *both* or *all* of these sets are constitutive. This would be a case of neural multiple realizability, whereby two or more different neural states could nonetheless constitute the same phenomenal state20. Note that this notion of neural multiple realizability, however, is still one in which redundancy is involved.

There is yet another, perhaps extreme, alternative in which neural multiple realizability could occur without any involvement of redundancy. That is, it could be that the whole scenario of isolated stimulation radically changes the NCnC, such that neural activities that are never constitutive in the normal case (perhaps tightly correlated upstream or downstream NCrCs) *become constitutive* in the case of isolated stimulation. We might describe this as *radical* neural multiple realizability, in which isolated stimulation of non-constitutive NCrCs in the bottom or top third of the tower—whether they are minimally sufficient or not *could* conceivably induce (and constitute) the target conscious state. If this extreme scenario were to hold, it would mean that constitutive neural activities would still be identified by the isolated stimulation strategy but that such constitutive activity would bear no relationship to constitutive activity in the normal case. Achieving that identification would be far less relevant to consciousness science than achieving identification of constitutive activity in the normal case.

While radical neural multiple realizability cannot be totally excluded as a possibility, its likelihood can be questioned. That is, while the brain exhibits remarkable capacity for rapid and substantial plastic change—such as reorganization of somatosensory maps following deafferentation (Merzenich et al., 1983;

<sup>20</sup>Note that in philosophy, multiple realizability refers to functional roles, including consciousness, being realized by *any* substrate, not just neural activity, with such roles thus being substrate-independent. My interest in multiple realizability, however, concerns just *neural* multiple realizability. In Miller (2007), I argued that within the bounds of the problem of direct intersubjective exchange and the Cr/Cn distinction problem, the science of consciousness could nonetheless make progress on issues of neural multiple realizability. As it turns out, such progress was already underway, utilizing the phenomenon of rivalry (Maier et al., 2007). These investigators examined the perception-dependency of single-unit responses to a variety of bistable stimuli and showed, amongst other findings, that the same perceptual state can be correlated with different neural activity patterns in macaque middle temporal (MT) area.

Ramachandran et al., 1992; Weiss et al., 2000)—such changes nonetheless take days to weeks to occur. While radical neural multiple realizability for a specific conscious state may be highly probable after days to weeks, for it to confound the isolated stimulation strategy, it would need to involve more or less instantaneous reorganization of the NCnC. Instantaneous reorganization of the NCnC in this way would seem highly *im*probable.

Non-radical neural multiple realizability (**Figure 3**), however, is a far more likely possibility, as is redundancy in the NCnC, certainly in the case of within-region neural activity (see next section and footnote 20). Both non-radical neural multiple realizability and redundancy, given their higher probability, challenge consciousness science and account for the isolated stimulation strategy's failure to identify the exact boundaries of the NCnC. To reiterate what each possibility involves, recall P2 above in the case of redundancy, and in the case of non-radical neural multiple realizability, we can state:

Possibility 3 (P3): Non-radical neural multiple realizability involves the possibility that more than one *subset* of constitutive neural activity can be minimally sufficient for a conscious state when separately subjected to isolated stimulation.

In summary then, with the isolated stimulation empirical strategy, while the obstacle of P1 is overcome (arguably, see later), the obstacles of P2 and P3 are not. And while minimally sufficient constitutive neural activities might be identified by isolated stimulation, redundantly constitutive neural activities cannot be distinguished from non-constitutive neural activities (P2) and hence the whole NCnC cannot be identified. Moreover, initially identified minimally sufficient constitutive neural activities may turn out to not be the *only* minimally sufficient constitutive neural activities (P3). Nonetheless, the reverse Jenga scenario may enable identification of some constitutive neural activities (indeed, minimally sufficient constitutive neural activities) and such identification would represent very significant progress in consciousness science. However, because this strategy fails to identify the exact boundaries of the NCnC, the Cr/Cn distinction problem is not solved and we may therefore only *close in* on the constitution of consciousness21,22 .

## **CLARIFICATIONS**

At this stage, some points of clarification are required before proceeding to discussing potential objections to the presented arguments, and then to proposing new foundational claims for consciousness science.

## **BETWEEN- AND WITHIN-REGION CASES**

When considering the notions of redundancy in the NCnC (P2) and non-radical neural multiple realizability (P3), it should be noted that these possibilities apply both *between* and *within* specific brain regions. In **Figure 2**, the stepwise inhibition strategy and the obstacles of P1 and P2 were illustrated for the case of binocular rivalry. This phenomenon also serves to illustrate P2 and P3 in the between- and within-region cases. Thus, constraining our analysis to perception-dependent neural activities (which the Jenga and reverse Jenga analogies require), and setting aside the fact that rivalry is not normal vision, consider the following.

In the between-region case, a visual state (during rivalry, with motion, color and complex images) is constituted say, by IT/STS (inferior temporal/superior temporal sulcus) neural activity *and* V4/MT (middle temporal) neural activity (this being the whole specific factor NCnC for each rivaling state), but (i) the visual state would not be affected by inhibition of V4/MT activity because V4/MT activity is redundantly constitutive; *and* (ii) isolated stimulation of IT/STS neurons induces the visual state and is thus minimally sufficient constitutive neural activity for it, but isolated stimulation of V4/MT activity does not induce the visual state and thus is not minimally sufficient constitutive neural activity for it; *or* (iii) separate isolated stimulation of either IT/STS or V4/MT induces the visual state and thus both can be considered minimally sufficient constitutive neural activity for it (and nonradical multiple realizations of it).

In the within-region case, a visual state during rivalry is constituted say, by IT/STS neural activity (this being the whole specific factor NCnC for each rivaling state), but (i) the visual state would not be affected by inhibition of *some* IT/STS neural activity because that inhibited activity is redundantly constitutive; *and* (ii) isolated stimulation of some IT/STS activity induces the visual state and is thus minimally sufficient constitutive neural activity for it, but isolated stimulation of the remaining (or some other) IT/STS activity does not induce the visual state and is thus not minimally sufficient constitutive neural activity for it; *or* (iii) separate isolated stimulation of either some IT/STS activity or of remaining (or some other) IT/STS activity induces the visual state and thus both can be considered minimally sufficient constitutive neural activity for it (and non-radical multiple realizations of it).

While the between-region case above is certainly conceivable, the within-region case, especially for the redundancy possibility (P2), must be considered *highly probable*. That is, it is highly probable that thousands, tens of thousands, hundreds of thousands or millions of neurons are specific factor constituents for a conscious state, and it is highly improbable that every one of them would need to be active to constitute that state. Hence it is highly improbable that every one of them would need to be stimulated in the isolated stimulation case, to induce (and constitute) the target state. Within-region redundancy also raises further issues of importance for consciousness science and I address these later when considering an objection to the redundancy argument.

It is important to also be mindful in this context, however, of existing data from electrical microstimulation studies. It is a remarkable fact from such studies that stimulation of relatively few MT neurons (perhaps just hundreds) can *bias* a

<sup>21</sup>Given the redundancy problem (P2), the "closing in" notion may be misleading because isolated stimulation would actually overshoot such closing in, through its exclusion of redundantly constitutive neural activities. Nonetheless, identification of at least some constitutive neural activity is what is meant by "closing in" on the problem.

<sup>22</sup>For a detailed analysis of many of the principles I have been discussing from the perspective of philosophy of science, see Craver (2007). Craver's analysis includes discussion (with specific scientific examples) of notions of mere correlates, interference and stimulation interventions, redundancy, and constitutive relevance (on the latter, see footnote 24). He provides examples of scientific approaches to dealing with redundancy, but not in the context of consciousness.

monkey's perceptual decision regarding motion direction (Britten et al., 1992; Salzman and Newsome, 1994; Ditterich et al., 2003; Cohen and Newsome, 2004, 2009; Shadlen and Kiani, 2013; see Sengpiel, 2013, for discussion of this data in the context of binocular rivalry and for the point that microstimulation has never been applied to rivalry). One cannot necessarily extrapolate from those studies, however, in which stimulation of relatively few neurons can influence discrimination under difficult decisionmaking conditions, to the notion that there will be similarly few neurons constitutive for a specific conscious state. That is, it does not follow that the number of neurons it takes to bias competition within pools of active neurons, and to thus influence a subject's difficult decision, is equivalent to the number of specific factor neurons constituting a conscious state. There may be similarly few neurons determining competition between conscious states (such as during rivalry) but this is a different matter to the number of specific factor neurons constituting each state.

Nonetheless, it may turn out that the activity of fewer neurons than we expect is the (specific factor) neural basis, mechanism, substrate, or constitution of a conscious state. This will, at the end of the day, be an empirical matter that strategies like isolated stimulation, should they come to pass, will determine. It is also noteworthy that Newsome himself is not afraid to examine the issue of just what his and his contemporaries' microstimulation work means for the issue of the neural basis, mechanism, substrate, or constitution of subjective experience (Cohen and Newsome, 2004; Reppas and Newsome, 2007). He concludes (Reppas and Newsome, 2007, p. 7) that, "The development of a non-invasive technique to modify precisely and locally neural activity in humans will probably be necessary to address such questions satisfactorily."

#### **LINKING, BINDING, OR INDEX PROCESSES**

I have previously acknowledged (Miller, 2007) that searching for this or that NCrC or set of NCrCs might be misguided and that it is instead a neurophysiological process *linking* or *binding* multiple NCrCs, such as, for example, recurrent processing (Lamme, 2006, 2010; see also Klink et al., in press) or oscillatory activity and temporal synchrony (Singer, 2001; Fries, 2005; Fries et al., 2005; though see Shadlen and Movshon, 1999; see also Sengpiel, 2013), that *indexes* the neural basis, mechanism, substrate, or constitution of consciousness. It is certainly possible that physiological processes between, or that bind, individual NCrCs could index phenomenally conscious neural activity over and above the tight correlations observed in each of those individual NCrCs. But if so, there will likely be various sets of recurrently, temporally or otherwise bound NCrCs for a conscious state, and we can ask which set should be assigned constitutive status and how will various hypotheses in this regard be tested. Perhaps each bound neural set could be considered an "individual" NCrC and the same obstacles of P1–P3 would apply.

Alternatively, even if there are not various sets of indexed or bound neural activities—because all such sets should rather be considered just one larger set—there can still be various hypotheses proposed regarding whether all individual NCrC components of that one larger set are constitutive, or just some. In other words, it can be claimed that a neurophysiological process linking or binding various individual NCrCs, indexes the neural basis, mechanism, substrate, or constitution of consciousness, but it is a *further claim* to hold that all neurons or all brain regions participating in such a process are constitutive. That further claim is itself a hypothesis, with rival hypotheses being based on the notion that only some neurons, neuron types, local or distributed neural circuits, specific brain regions or sets of NCrCs participating in the index/binding process are constitutive.

To test hypotheses in this regard would require intervening through stepwise inhibition and stepwise isolated stimulation to prevent and reintroduce, respectively, the relevant index or binding process, and observing what happens to the conscious state in each case. In such experiments, it may turn out that in the absence of the index or binding process there is *never* the conscious state and this would certainly inform consciousness science. But it may also turn out that inhibition of one component of an indexed/bound neural set does not lead to the absence of the index/binding process in remaining components of the set and does not lead to a change in the target conscious state. Would that inhibited component therefore be considered nonconstitutive or redundantly constitutive? And could isolated stimulation of various subsets of indexed/bound NCrCs be minimally sufficient constitutive neural activity? The index/binding process case does not seem to enable us to avoid the obstacles of P1 and P2 for stepwise inhibition and P2 and P3 for stepwise isolated stimulation.

#### **A CAVEAT ON ISOLATED STIMULATION**

The final point of clarification relates to earlier reference to isolated stimulation only *arguably* enabling identification of at least some constitutive neural activities. That is, the above discussion of possible index processes that link or bind different neural activities raises an important potential complication for the isolated stimulation strategy. A scenario could be postulated whereby isolated stimulation of the (putative) minimally sufficient NCnC could fail to induce the target conscious state if that neural activity set required for its correct functioning, intact connections to other (disinhibited and active) neural activity sets (such as noncorrelated causal chain components, loosely correlated NCrCs or tightly correlated non-constitutive NCrCs). Such a scenario would potentially then lead to an inability to distinguish the minimally sufficient NCnC and non-constitutive minimally sufficient NCrCs (and even non-correlated causal chain components) and this is precisely the obstacle of P1. In such a scenario therefore, it could be claimed that isolated stimulation does *not* in fact overcome the obstacle of P1. This is a concern addressed further at the end of the next section, where I discuss a potential objection based on a postulated requirement for intact connection even to disinhibited and *inactive* neural sets.

## **OBJECTIONS**

There are at least six potential objections to the arguments presented thus far. Each of these is discussed in detail in Miller (in press-b). Here I list three of these and then discuss in detail only the three most relevant to the presented arguments. The objections not discussed here include: (i) *Definition objections* that Chalmers' (2000) definition of the minimally sufficient NCrC, including his discussion of redundancy technicalities, would exclude the sorts of scenarios I have discussed; (ii) *Specificity objection*—that the Cr/Cn distinction problem may not be specific to consciousness science, but rather applies in many scientific domains; and (iii) *Theoretical loading objection*—that the very notion of constitution is theoretically loaded (regarding the relation between mind and brain) in a way that the notion of correlation is not.

#### **TRIVIALITY OBJECTION**

Another potential objection is that it could be claimed that redundancy (P2) and non-radical neural multiple realizability (P3) are but *trivial* possibilities. This objection would hold that if we had identified the minimally sufficient NCnC using the isolated stimulation strategy, we need not be concerned about failing to go on to achieve identification of the whole NCnC because any differences between these two neural activity sets is trivial. There are two responses to this objection and both draw on the notion of *explanation* in consciousness science (a notion discussed more fully in Chalmers, 1996; Revonsuo, 2000, 2001, 2006, in press; Hohwy and Frith, 2004; Bayne, 2007; Hohwy, 2007, 2009; Seth, 2009; Neisser, 2012; Drayson, in press; Hohwy and Bayne, in press; Keaton, in press; Miller, in press-b; Opie and O'Brien, in press).

Explanation is a somewhat vexed issue in consciousness science because of "the hard problem" discussed by Chalmers (1996) in which even identification of the whole NCnC could still leave unanswered questions such as *how* it is that *that* particular set of neural activities constitutes consciousness and *why* it is that there should be any consciousness at all. Despite the possibility of explanatory gaps in the study of consciousness (Levine, 1983), some degree of explanation *can* still be sought and achieved in the scientific study of consciousness, in particular with respect to the notion of *mechanistic explanation* in the biological sciences (Bechtel, 1994; Machamer et al., 2000; Revonsuo, 2000, 2001, 2006, in press; Craver, 2007; Horst, 2007; Neisser, 2012; Oizumi et al., 2014; Hohwy and Bayne, in press; Mahner, in press; Miller, in press-b; Opie and O'Brien, in press; cf. Irvine, 2013, who appeals to mechanistic explanation to argue why there cannot even be a science of consciousness)<sup>23</sup> .

It could be argued, as Jakob Hohwy has pointed out to me, that we need not worry about identifying redundantly constitutive neural activities because if a conscious state does not disappear (or degrade) when a redundantly constitutive NCrC is inhibited, then the difference between the minimally sufficient NCnC and the whole NCnC, is a *difference-without-a-difference* (and any such difference is therefore trivial). As such, the argument would hold that, being redundant, the unidentified constitutive NCrCs could do no *explanatory work* even if they were identifiable. My reply, however, is that a difference-without-a-difference for consciousness does not amount to a difference-without-a-difference for consciousness *science*.

That is, we can reasonably wish to answer the following questions: (i) is there in fact any redundancy in the NCnC? (ii) if so, why should there be such redundancy? (iii) is such redundancy based on a critical *size* of neural activities (a critical number of involved neurons, neuron types and neural circuits)? (iv) is such redundancy based on a critical *location* of neural activities? (v) is such redundancy based on a critical *combination* of stepwise inhibition when applying that empirical strategy? (vi) is such redundancy based on a critical *order* of stepwise inhibition when applying that empirical strategy? (vii) is there non-radical neural multiple realizability within the NCnC (which draws on the notion of redundancy) when applying the isolated stimulation empirical strategy? (viii) and how does such redundancy relate to index or binding processes? Answers to such questions would seem, in my view, far from devoid of explanatory power (just as understanding many such issues would help to *mechanistically explain* when and why a Jenga tower might fall).

The second response to the triviality objection rests on the fact that *the* major element of the scientific study of consciousness, at least currently, is the search not for explanations as such, but rather for identification of *which* neural activities are the basis, mechanism, substrate, or constitution of consciousness. This is fundamentally a process of identifying the relevant neural activities, not of explaining how and why *those* activities do the constituting. That is, the issue with which I have been concerned in this paper (and with which consciousness science appears most concerned) is one of determining inclusion and exclusion into the constitutive neural activity set. Understanding explanatory mechanisms may of course help with this identification process, but consciousness science could conceivably identify the NCnC even without understanding some explanatory mechanistic principles relevant to that neural activity set.

Hence in summary, appeals to the absence of explanatory power in redundantly constitutive NCrCs do not support their identification as trivial, because (i) issues regarding redundantly constitutive NCrCs can in fact do explanatory work; and (ii) despite their redundancy, redundantly constitutive NCrCs are nonetheless constitutive activities and are thus legitimately part of the neural basis, mechanism, substrate, or constitution of consciousness. As such, we should seek their identification and we should consider what scientific strategies might achieve this goal.

#### **WAIT-AND-SEE OBJECTION**

A further potential objection concerns Crick and Koch's (1998) suggestion that hard problems of consciousness be set aside until science makes more progress. It is not clear if such a caution should include the Cr/Cn distinction problem given its clear scientific relevance, but obviously in my view, it should not. That said, there is certainly an important message in the wait-andsee approach and indeed I have previously noted (Miller, 2007, p. 165) that, "… future scientific work may show that the notion of the Cr/Cn distinction is somehow fundamentally misguided (in a way that cannot yet be appreciated because the science is not yet done)." Similarly, Revonsuo (2006, p. 292) rightly cautions us over imaginary neuroscience scenarios when he says: "In the absence of the relevant empirical facts, we simply cannot imagine or foresee the perfect future science regarding *any* phenomenon."

<sup>23</sup>The arguments I have presented thus far have considered the terms neural "basis," "mechanism," "substrate," and "constitution" of consciousness as synonymous, but in the context of mechanistic explanation, the term "mechanism" has its own implications (just as the "constitution" term has its own implications also; see later).

I am entirely accepting of the fact that as neuroscience progresses, particularly with respect to dynamic mechanistic multilevel explanation of neural processing within *and* outside of consciousness science, any or all of the scenarios and claims made in this paper (and its longer version) may require refinement, revision or indeed rejection. For example, the technique of optogenetics as applied in the context of the current paper would also be capable of being applied in many other neuroscientific contexts (e.g., sensory coding, voluntary movement and various executive functions). As such application advances understanding of neural coding, signaling, dynamics and function, and understanding of the relationship between neural processing and the phenomena being studied, so too such understanding will have flow-on implications for consciousness science. In addition to optogenetics, developments will occur (as discussed in several papers in Miller, 2013a) with other recording, inhibition, and stimulation methods, and signal analysis approaches therein, to improve the spatial and temporal resolution of each technique, to enable combined methodological approaches, to accurately map structural and functional neural connections, to improve neural spike detection and sorting and neural population, source and network modeling, to facilitate integration of knowledge across the various levels targeted by each technique, and in general to develop a more detailed understanding of subcellular, neural, neural circuit, and large-scale/regional/systems processing.

Conversely, as neuroscientific understanding progresses, the presented scenarios and claims may rather become strengthened (for example, the claim that the isolated stimulation strategy identifies some constitutive neural activities may not be questionable after all). Moreover, the presented scenarios and claims may go beyond the realm of "in principle" and instead become *directly testable*. Until there is development of a safe, reversible and "pure" recording, inhibitory and stimulatory technique, it would be unwise to not accept likely revision to the presented scenarios and claims. Just as this and other neuroscientific progress will help to disentangle the complexities of the microstimulation and decision-making literature, and the broader neural coding and processing issues within which those complexities lie (see references in previous section; reviewed in Shadlen and Kiani, 2013; see also Sengpiel, 2013), so too it will help to disentangle the complexities of recording, inhibition, and stimulation scenarios within consciousness science. We may wish to wait and see on such issues before getting too conceptually entangled, but in my view it would be unwise to hold that the conceptual analyses and proposals presented here should not be further discussed and debated, or should not even have been embarked upon.

#### **INTEGRATED INFORMATION THEORY OBJECTION**

The final potential objection is that stemming from Integrated Information Theory (IIT; Oizumi et al., 2014; see also Klink et al., in press). IIT is a highly complex and developed theory of consciousness, with its own detailed conceptual definitions and tools, that starts with phenomenological axioms and proceeds to formalize these into, "postulates that prescribe how physical mechanisms, such as neurons or logic gates, must be configured to generate experience (phenomenology)" (Oizumi et al., 2014, p. 1). IIT is based heavily on mechanistic causal roles—though specifically in IIT, and importantly in the present context, only differences that make a difference—and involves perturbation (again important in the present context) of the elements of candidate sets into all possible states, and identification of maximally irreducible cause–effect repertoires and structure. IIT does not permit for P1, P2, or P3 because it takes the *inactive* elements in maximally irreducible cause–effect structure to be just as critical as the active elements. Additionally, through an exclusion postulate, among overlapping candidate sets of mechanistic elements, only one forms a complex—that with the maximum quantity of integrated conceptual information—and hence no subsets or supersets of those mechanistic elements can form a complex.

It will take someone better versed than myself in IIT to properly set out the detailed objections to P1, P2, and P3 that stem from the theory, but from the brief description above, some clues to the objections should be visible. The arguments presented in this paper are based on an excitation approach to constitutive neural activity, albeit with an acknowledgment of the physiological inseparability of excitatory and inhibitory processes (and other caveats). However, IIT considers inactive (i.e., resting state, or disinhibited and not stimulated) neural states to be just as critical to the constitution (structure) of a conscious state as stimulated neurons, and on this construal, the isolated stimulation scenario I have been discussing would be considered inaccurate. Note though, that this way of thinking would also lead to objecting to the notion of the minimally sufficient NCrC, insofar as that construct also relies on excitatory correlates rather than the set of relevant active *and* inactive neural elements.

Instead, IIT would argue that the constitution of consciousness is identifiable by identifying the (neural) complex with the maximum quantity of integrated conceptual information and that once this complex has been identified, adding anything to it or subtracting anything from it must change the conscious state, however minimal that change may be. On this construal and on the axioms and postulates of IIT, P1, P2, and P3 are not possibilities at all. This IIT objection to the present arguments is testable, in principle, and if the identified complex with the maximum quantity of integrated conceptual information can in fact be added to or subtracted from without a concomitant change in the conscious state, it will then be IIT that is found wanting.

#### **RELATED SCIENTIFIC AND PHILOSOPHIC ISSUES**

There are many additional issues associated with the notion of constitution and the Cr/Cn distinction problem, with both explanatory and other theoretical implications (discussed in detail in Miller, 2007, in press-a,b). For example, additional scientific questions include: (i) what do the Cr/Cn distinction problem, the Jenga analogy and the reverse Jenga analogy look like for *enabling* rather than specific factors? (ii) what is the relationship of the Cr/Cn distinction problem to other scientific and philosophic consciousness problems—such as identifying where in phylogeny and where in ontogeny phenomenal consciousness exists, knowing what it is like to be another subject (the other minds problem or problem of direct intersubjective exchange), and the hard problem? (iii) if science meets an epistemic limit with the Cr/Cn distinction problem, what is it about scientific method that gives rise to such a limit?

Another important scientific issue concerns at what *level* of neural processing consciousness is constituted (Revonsuo, 2000, 2001, 2006, in press; Miller, 2007, in press-b; Hohwy and Bayne, in press; Opie and O'Brien, in press)? This is another constitution problem that both the science *and* philosophy of consciousness will need to address. Thus, should the NCnC be considered most relevant at the level only of constitutive action potentials? Or perhaps at the level only of the electrophysiological processes of distributed constitutive neural circuits and networks (bound as they may or may not be)? Or perhaps both levels are constitutively relevant? And what is the relation of NCnC *micro*constituents (subcellular molecular constituents) to consciousness? Should the NCnC be considered to include all microconstituent processes, or only some? And if just some, which, and how will we test hypotheses in this regard? In this context, the term "constitution" takes on additional relevance over and above "basis," "mechanism" and "substrate" and specifically, different *grains* of constitution can be appreciated—i.e., at coarse (systems), fine (individual neurons and microcircuits), and very fine (subcellular) scales (see Miller, in press-b).

Some of the additional philosophic issues have been alluded to above in terms of other philosophic problems of consciousness. Also, the issue of various grains of constitution suggests, in my view, a *mereology of phenomenal consciousness*. Mereology is the branch of philosophy that deals with the relationship between parts and wholes. It has mostly concerned itself with analysis of part-whole relations for static objects (see several papers in Miller, in press-a) rather than *process*-based physiological systems. It is my contention that the philosophy of consciousness needs to (i) engage in mereological analysis of processes to better understand part-whole relations of physiological and specifically neural systems; and (ii) thereafter, focus on what additionally may be relevant in considering a mereology of the NCnC. A mereology of phenomenal consciousness will require concerted interdisciplinary interaction amongst philosophers and scientists, with updating of the analyses as neuroscientific knowledge itself progresses. Extension of the classical mereological focus on objects to (neurophysiological) processes should be undertaken to *complement* mechanistic explanation approaches (which also deal with component parts, and specifically their causal operations, activities, and organization)—particularly *dynamic* mechanistic explanation; see Bechtel and Abrahamsen, 2010, 2013)—rather than as an alternative to it<sup>24</sup> .

Other relevant philosophic issues surrounding the notion of constitution concern the perennial issue in philosophy of mind of the relation between brain and mind. Thus, (i) what are the similarities and differences between the constitution relation and others that purport to describe the brain–mind relation, such as identity, supervenience, realization, emergence, and causation? (ii) is the constitution relation in the case of phenomenal consciousness a unique constitution relation, and if so, unique how? (iii) if indeed unique, does it simply look nothing like a constitution relation in the usual parts-whole sense, even for processes? (iv) does the uniqueness of the constitution relation in this particular case tell us something about the uniqueness of consciousness and the place of consciousness in nature? Finally, another perennial issue in philosophy of mind—the ontological issue of whether consciousness and mind are entirely physically (materially) composed—is also relevant in the context of the Cr/Cn distinction problem. If with the Cr/Cn distinction problem, science meets an epistemic limit, what does this mean for these ontological matters? For more detailed discussion of these related scientific and philosophic issues, see Miller (2007, in press-a,b).

### **NEW FOUNDATIONS FOR THE SCIENCE OF CONSCIOUSNESS**

In this paper (and in Miller, in press-b), we have seen through the Jenga analogy that the minimally sufficient NCrC construct can pick out a different neural activity set to that of the NCnC. We have also seen that while application of the stepwise inhibition empirical strategy can distinguish the merely sufficient from the minimally sufficient NCrC, it fails to make any progress on identifying constitutive neural activities (due to P1 and P2). In addition, while application of the isolated stimulation empirical strategy can (arguably) identify some constitutive neural activities, it fails to make any progress on identifying the whole NCnC (due to P2 and P3).

Although the Cr/Cn distinction problem remains unsolved, as are other (mereological) consciousness constitution problems, the analyses I have presented suggest new conceptual foundations for consciousness science, depicted by the following claims (C). These claims are proposed for further discussion and notably, they include reference to specific empirical strategies and to possibilities P1–P3. There are many aspects of consciousness science

<sup>24</sup>The notion of a mereology of processes differs from the mechanistic approach to activities in a subtle but potentially important way. Craver (2007, p. 136) notes, "There are no mechanisms without active organisation, and no mechanistic explanation is complete or correct if it does not capture correctly the mechanism's active organisation." Craver's notion of activities (Machamer et al., 2000; Craver, 2007) is one with a decidedly mechanistic flavor. In addition, Craver (2007) outlines the importance of attending to the notion of *constitutive relevance* in mechanistic explanation (i.e., identifying relevant components and their interactions and excluding irrelevant components and their interactions). While this is no doubt indeed important for mechanistic explanation—and for finding constitutive mechanisms of consciousness (as sought by Revonsuo, 2006)—understanding the constitution of consciousness may need to take more than just a mechanistic explanation form. Indeed, this is why the title of this paper—"Closing in on the Constitution of Consciousness"—stands in contrast to the title of a chapter in Revonsuo

<sup>(2006)—&</sup>quot;Closing in on the Mechanisms of Consciousness." Although the mechanistic explanation form may turn out to be the best way by which to understand the constitution of consciousness, mereological approaches, in my view, also need developing. This will not, however, be easy territory. Craver's (2007) use of the notion of "constitutive explanation" (used in the mechanistic context) is not intended to have metaphysical relational implications. Thus he notes (p. 20), "Metaphysicians reserve the term "constitutive" for a specific relation that has more entanglements than I intend." A metaphysical mereological notion of "constitutive" may well, I suggest, have different implications for the notion of constitutive relevance than the mechanistic notion of "constitutive." Finally, in answering how we might test hypotheses regarding which microconstituents can be considered part of the NCnC and which not (hence which are constitutively relevant), microconstituent stepwise inhibition could be applied, with assessment of whether a conscious state changes or does not change accordingly. However, drawing conclusions from such interventions, and determining whether a Cr/Cn distinction problem also exists at this level (see Miller, in press-b), would seem to require the combination of mereological and mechanistic analyses that I am advocating.

not dealt with by these claims, so this is not intended to be an exhaustive list. It is merely a list of claims arising from the main analyses I have undertaken. The proposed claims are:

C1: Setting aside notions of the hard problem of consciousness, the explanatory gap, hard phylogeny and ontogeny problems of consciousness, and the problem of direct intersubjective exchange, the ultimate aim of consciousness science is not to identify the minimally sufficient neural correlates of consciousness, but rather to identify the (whole) neural basis, mechanism, substrate, or constitution of consciousness.

C2: The current foundational construct of consciousness—the minimally sufficient neural correlates of consciousness—can pick out a different neural activity set to that picked out by the neural basis, mechanism, substrate, or constitution of consciousness.

C3: If by "under conditions C" in Chalmers' (2000) definition of the minimally sufficient neural correlates of consciousness, we mean stepwise inhibition—the most obvious empirical approach to distinguishing the merely sufficient from the minimally sufficient neural correlates of consciousness—then the minimally sufficient neural correlates of consciousness construct is limited by its inclusion of neural activities that are not in fact part of the neural basis, mechanism, substrate, or constitution of consciousness, and by its exclusion of neural activities that are. Consciousness science can nonetheless continue to work toward identifying the minimally sufficient neural correlates of consciousness using recording strategies and the stepwise inhibition strategy.

C4: If by "under conditions C" in the definition of the minimally sufficient neural correlates of consciousness we mean isolated stimulation, then the minimally sufficient neural correlates of consciousness construct is equivalent to the minimally sufficient neural constitution of consciousness construct and at least some constitutive neural activities will (arguably) be identifiable. Whichever term is preferred, this neural activity set can be different to that of the *whole* neural basis, mechanism, substrate, or constitution of consciousness and hence neither construct with the minimally sufficient qualifier should be considered the ultimate target of consciousness science.

C5: Because of the possibilities of redundancy in the neural constitution of consciousness (i.e., of constitutive non-minimally sufficient neural correlates of consciousness) and of non-radical neural multiple realizability, there is not yet evident an empirical strategy to identify the whole neural constitution of consciousness. Empirical approaches to this problem need to be developed, and may require entirely new scientific strategies<sup>25</sup> .

The science of consciousness is young and thriving. There is a great deal of empirical and conceptual work to be done in this field and the foundational map charted by Chalmers (2000) needs to be reassessed and built upon, with a focus on empirical strategies. The analyses and new foundational claims presented here (and in Miller, in press-b) are an attempt in this direction. Consciousness remains, as ever, an intriguing subject of intellectual discourse. It is today also rightly situated at the frontier of scientific endeavor.

### **ACKNOWLEDGMENTS**

I am grateful to Jakob Hohwy, Trung Ngo, Anthony Hannan, Nao Tsuchiya, Jeroen van Boxtel, Tom de Graaf, Talis Bachmann, Giulio Tononi and the two reviewers for helpful comments and discussions regarding this work. The paper has also benefited from my interactions with many other authors in the volume in which the longer version appears. Thanks also to Phillip Law for assistance with the figures.

### **REFERENCES**


<sup>25</sup>An abbreviated, conjoined and simpler set of claims, without specific reference to empirical strategies and to P1–P3, would be:

C1: The ultimate aim of consciousness science is not to identify the minimally sufficient neural correlates of consciousness, but rather to identify the (whole) neural basis, mechanism, substrate, or constitution of consciousness.

C2: The current foundational construct of consciousness—the minimally sufficient neural correlates of consciousness—can pick out a different neural activity set to that picked out by the neural basis, mechanism, substrate, or constitution of consciousness. Consciousness science can nonetheless continue to work toward identifying the minimally sufficient neural correlates of consciousness.

C3: Although there may be empirical strategies to identify some constitutive neural activity, there is not yet evident a strategy to identify the whole neural

constitution of consciousness. Empirical approaches to this problem need to be developed, and may require entirely new scientific strategies.


*Consciousness Research*, ed. S. M. Miller (Amsterdam: John Benjamins Publishing Company), 167–186. doi: 10.1075/aicr.90


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 15 August 2014; accepted: 24 October 2014; published online: 17 November 2014.*

*Citation: Miller SM (2014) Closing in on the constitution of consciousness. Front. Psychol. 5:1293. doi: 10.3389/fpsyg.2014.01293*

*This article was submitted to Consciousness Research, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Miller. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## ADVANTAGES OF PUBLISHING IN FRONTIERS

FAST PUBLICATION Average 90 days from submission to publication

COLLABORATIVE PEER-REVIEW

Designed to be rigorous – yet also collaborative, fair and constructive

RESEARCH NETWORK Our network increases readership for your article

## OPEN ACCESS

Articles are free to read, for greatest visibility

#### TRANSPARENT

Editors and reviewers acknowledged by name on published articles

GLOBAL SPREAD Six million monthly page views worldwide

## COPYRIGHT TO AUTHORS

No limit to article distribution and re-use

IMPACT METRICS Advanced metrics track your article's impact

SUPPORT By our Swiss-based editorial team

EPFL Innovation Park · Building I · 1015 Lausanne · Switzerland T +41 21 510 17 00 · info@frontiersin.org · frontiersin.org