# THE LONG AND SHORT OF MENTAL TIME TRAVEL— SELF-PROJECTION OVER TIME-SCALES LARGE AND SMALL

EDITED BY: James M. Broadway, Claire M. Zedelius, Jonathan W. Schooler, and Simon Grondin PUBLISHED IN: Frontiers in Psychology

### *Frontiers Copyright Statement*

*© Copyright 2007-2015 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

*All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88919-583-1 DOI 10.3389/978-2-88919-583-1

## About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

## Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

## Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

## What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

## **THE LONG AND SHORT OF MENTAL TIME TRAVEL– SELF-PROJECTION OVER TIME-SCALES LARGE AND SMALL**

Topic Editors:

**James M. Broadway,** University of California Santa Barbara, USA **Claire M. Zedelius,** University of California Santa Barbara, USA **Jonathan W. Schooler,** University of California Santa Barbara, USA **Simon Grondin,** Université Laval, Canada

Researchers working in many fields of psychology and neuroscience are interested in the temporal structure of experience, as well as the experience of time, at scales of a few milliseconds up to a few seconds as well as days, months, years, and beyond. This Research Topic supposes that broadly speaking, the field of "time psychology" can be organized by distinguishing between "perceptual" and "conceptual" time-scales. Dealing with conceptual time: "mental time travel," also called mental simulation, self-projection, episodic-semantic memory, prospection/foresight, allows humans (and perhaps other animals) to imagine and plan events and experiences in their personal futures, based in large part on memories of their personal pasts, as well as general knowledge. Moreover, contents of human language and thought are fundamentally organized by a temporal dimension, enmeshed with it so thoroughly that it is usually expressible only through spatial metaphors. But what might such notions have to do with experienced durations of events lasting milliseconds up to a few seconds, during the so-called "present moment" of perception-action cycle time? This Research Topic is organized around the general premise that, by considering how mental time travel might "scale down" to time perception (and vice-versa, no less), progress and integrative synthesis within- and across- scientific domains might be facilitated. Bipolar configurations of future- and past-orientations of the self may be repeated in parallel across conceptual and perceptual time-scales, subsumed by a general "Janus-like" feedforwardfeedback system for goal-pursuit. As an example, it is notable that the duality of "prospection" and semantic-episodic memory operating at conceptual time-scales has an analogue in perception-action cycle time, namely the interplay of anticipatory attention and working memory. Authors from all areas of psychology and neuroscience are encouraged to submit articles of any format accepted by the journal (Original Research, Methods, Hypothesis & Theory, Reviews, etc.), which might speak to questions about time and temporal phenomena at long and/or short time-scales.

**Citation:** Broadway, J. M., Zedelius, C. M., Schooler, J. W., Grondin, S., eds. (2015). The Long and Short of Mental Time Travel–Self-Projection Over Time-Scales Large and Small. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-583-1

# Table of Contents


Takako Mitsudo, Yoshitaka Nakajima, Hiroshige Takeichi and Shozo Tobimatsu

*122 Interval discrimination across different duration ranges with a look at spatial compatibility and context effects*

Giovanna Mioni, Franca Stablum and Simon Grondin


Mark A. Elliott


Damian Scarf, Christopher Smith and Michael Stuart


Michael S. Franklin, Stephen L. Baumgart and Jonathan W. Schooler

## Editorial: The long and short of mental time travel—self-projection over time-scales large and small

James M. Broadway <sup>1</sup> \*, Claire M. Zedelius <sup>1</sup> , Jonathan W. Schooler <sup>1</sup> and Simon Grondin<sup>2</sup>

<sup>1</sup> Psychological and Brain Sciences, University of California Santa Barbara, Santa Barbara, CA, USA, <sup>2</sup> École de psychologie, Université Laval, Québec, QC, Canada

Keywords: time perception, consciousness, mental time travel, oscillatory activity, memory, attention, rhythm perception, internal clock

This Research Topic is about time-experience broadly construed, as it manifests at perceptual and conceptual time-scales (milliseconds-to-seconds vs. longer times, respectively). Authors representing a broad spectrum of psychology and neuroscience have contributed, introducing novel theories, empirical findings, and methodological innovations. In this Editorial we give a thematic overview of the exciting and diverse contents of this Research Topic. (Abbreviations: O, Opinion; H&T, Hypothesis and Theory; OR, Original Research; P, Perspective; MR, Mini-Review; GC, General Commentary; M, Methods).

## Consciousness

Edited and reviewed by: Philippe G. Schyns, University of Glasgow, UK

\*Correspondence: James M. Broadway, broadway@psych.ucsb.edu

### Specialty section:

This article was submitted to Perception Science, a section of the journal Frontiers in Psychology

Received: 25 April 2015 Accepted: 07 May 2015 Published: 20 May 2015

### Citation:

Broadway JM, Zedelius CM, Schooler JW and Grondin S (2015) Editorial: The long and short of mental time travel—self-projection over time-scales large and small. Front. Psychol. 6:668. doi: 10.3389/fpsyg.2015.00668 Many contributors acknowledged that the experience of time is an irreducible part of conscious experience, strongly related to the experience of being a self (Wittmann et al., 2015). Zhou, Pöppel, and Bao (P) integrate different aspects of temporal experience into a unified biologicallygrounded framework held together by the concepts of identity and homeostasis. Berkovich-Ohana and colleagues (H&T) locate representations of minimal self and extended self within a threedimensional consciousness state-space defined by time, awareness, and emotion. Fingelkurts and Fingelkurts (O) link phenomenology to neural activation patterns, which in turn are constrained by bodily space and the larger context, in order to construct the temporal dimensions of past, future, and present. Martin and colleagues (H&T) consider how disturbances of the minimal self in patients with schizophrenia are possibly related to alterations in temporal processing typically associated with this disorder. In order to better illuminate how creative insight is subjectively experienced, with special attention to its temporal aspects, Cosmelli and colleagues (P) recommend neurophenomenological methodologies, integrating cognitive neuroscience with descriptive, first-person data.

## Multiple-Scales

Many contributors addressed how time is represented at multiple scales. Wackermann (O) grounds the experience of time in a knowing, willing subject, exploring how subjective horizons determined by human biological constraints impose various preconditions on the "notion and knowledge of time" across multiple scales. Singularly, Bonato and colleagues (O) propose that temporal cognitions respecting events occurring at vastly different time-scales, such as interval timing vs. mental time travel, are nonetheless represented by a common spatial metric (Bonato et al., 2012).

### Broadway et al. Perceptual and conceptual time

## Mechanisms

Many contributors addressed theoretical mechanisms of timing and time perception (Grondin, 2010; Matthews and Meck, 2014). Block and Grondin (GC) argue that the time perception field is too preoccupied theoretically with internal-clock models and empirically with interval timing. Alternatives to internal-clock models are well-represented in the Research Topic. For example, Marchetti (H&T) discusses time-experience as resulting from a process of construction performed by sampling and binding of the contents of working memory by attention. Schweickert and colleagues (OR) similarly emphasize the roles of attention and working memory, introducing to the field a novel complex span and timing hypothesis, which predicts that only secondary tasks that are sensitive to individual differences in working memory capacity will interfere with performance on the primary task of time perception in a dual-task setting. Perhaps of some importance for this notion, working memory capacity predicts timing abilities, even when these are tested without dual-task interference (Troche and Rammsayer, 2009; Broadway and Engle, 2011) Gupta (H&T) proposes a new theory explaining suband supra-second timing performance by calibration of neural oscillators by sensory, motor, and feedback processes.

## Perceptual Time

Experimental and theoretical work on time perception has led to increased understanding of how people keep track of time through multimodal perceptual processes (Grondin, 2001). Mitsudo and colleagues (OR) identify event-related brain potentials specifically associated with equality/inequality judgments of successive time intervals marked by auditory cues. Mioni and colleagues (OR) present evidence in favor of distinct timing mechanisms above and below a threshold of 1300 ms based on effects of context and space-time compatibility. Grondin (O) proposes that cross-modality influences on perceiving time raises questions about the realism of a unitary clock hypothesis. Van Rijn (O) demonstrates how abstract mechanisms underlying interval timing are affected by perceived speed, in a simulated driving task. Schaefer (O) speculates that shared components between movement and musical imagery may offer a window into timing and temporal skills, of

## References


relevance to cognition beyond movement or auditory functions. Elliott (OR) reports effects on perception due to specific phase relationships among stimuli in backward masking experiments. Zakay (H&T) looks at time perception in relation to information processing load, focusing his analysis on the experience of boredom.

## Conceptual Time

Experimental and theoretical work on mental time travel has led to better understanding of how time is represented conceptually (Roberts, 2008; Wittmann, 2013). Ye and Song (OR) report that in contrast to adults, the prospective bias in children seems to be unrelated to current task demands. Roy (O) suggests that so-called "optimistic biases," causing people to underestimate how much time a particular task will take, are overstated in the literature. Scarf and colleagues (MR) argue that if "spoon test" performances of children and great apes were assessed by identical criteria, the evidence suggests the ability for mental time travel can be observed among great apes. Logan (O) advocates similarly for the existence of mental time travel among nonhuman animals, focusing on certain bird species, and proposes new research approaches to further advance this question. Friedman and colleagues (M) describe a new immersive virtual reality technique in which individuals can virtually travel back in time to re-live a previous life-or-death moral dilemma, with the opportunity to act differently the second time around. Franklin and colleagues (O) challenge traditional concepts of time with notions of precognition and retro-causality.

## Conclusion

To sum up, the collection of articles in this Research Topic showcases the diverse fecundity of theoretical and empirical developments across a wide spectrum of contemporary research into time-experience broadly construed. It should be acknowledged that the organizing distinction between "perceptual" and "conceptual" time-experience is really only a heuristic; but it has enabled this Research Topic to bring into a single view, a wide panorama of ways to think about and investigate time-experience.


Wittmann, M., Otten, S., Schötz, E., Sarikaya, A., Lehnen, H., Jo, H.-G., et al. (2015). Subjective expansion of extended time-spans in experienced meditators. Front. Psychol. 5:1586. doi: 10.3389/fpsyg.2014.01586

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Broadway, Zedelius, Schooler and Grondin. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

## In the jungle of time: the concept of identity as a way out

## *Bin Zhou1, Ernst Pöppel 1,2,3,4,5 and Yan Bao2,3,4,5,6 \**


### *Edited by:*

James M. Broadway, University of California, Santa Barbara, USA

#### *Reviewed by:*

James M. Broadway, University of California, Santa Barbara, USA Yoshihiro Miyake, Tokyo Institute of Technology, Japan

#### *\*Correspondence:*

Yan Bao, Department of Psychology, Peking University, 5 Yiheyuan Road, Beijing 100871, China e-mail: baoyan@pku.edu.cn

What could be a unifying principle for the manifold of temporal experiences: the simultaneity or temporal order of events, the subjective present, the duration of experiences, or the impression of a continuity of time? Furthermore, we time travel to the past visiting in imagination previous experiences in episodic memory, and we also time travel to the future anticipating actions or plans. For such time traveling we divide time into three domains: past, present, and future.What could be an escape out of this "jungle of time" characterized by many different perceptual and conceptual phenomena? The key concept we want to propose is "identity" which is derived from homeostasis as a fundamental biological principle. Within this conceptual frame two modes of identity are distinguished: individual or self-identity required because of homeostatic demands, and object-related identity necessary for the reliability and efficiency of neuro-cognitive processing. With this concept of self- and object-identity, the different temporal experiences can be conceptualized within a common frame.Thus, we propose a fundamental biological principle to conceptually unify temporal phenomena on the psychological level.

**Keywords: time, identity, homeostasis, circadian rhythm, subjective present, simultaneity, episodic memory**

### **DIVERSITY OF TEMPORAL EXPERIENCES AND CONCEPTS**

When thinking about the different temporal experiences one gets the impression that one is dealing with very different topics which are not related to each other (Pöppel, 1978; Pöppel and Bao, 2014); temporal experiences appear to be qualitatively distinct. The question that has to be addressed first, is, what is meant with "experience": do we have "temporal experiences" at all, or are we not usually referring to theoretical concepts? We take the position that we often confuse experiences as "direct experiences" (in German "Erleben") and abstract notions (in German "Erfahrung"), without being aware of a categorical error; the English term "experience" apparently does not differentiate between these two meanings. We should remember what Gibson once said (Gibson, 1973): "Events are perceivable but time is not." And in a similar way Woodrow (1951) states: "Time is not a thing that, like an apple, may be perceived."

What are some of the problems that make it difficult to find a unifying principle of temporal experiences? A good example is the one when we refer to the "present." Do we think of the present as a border with no temporal extension between past and future, or does the present have a temporal extension? If the present is just a border without temporal extension between past and future, it cannot have any experiential quality. In this case the present is a theoretical notion, and temporal experiences are reduced on the phenomenal level to what has happened and what might happen. But what could be the experiential quality of what is going to happen or what has happened? Temporal experiences are then either anticipations, hopes, plans or memories of events past. But if the present has a temporal extension, as is implicitly assumed by Augustinus (397/8, 1993), one of the founding fathers of "temporal philosophy," or James (1890), then the empirical question comes up how long such a present might be, and this question has indeed been addressed with different experimental paradigms (Pöppel, 1978; Pöppel and Bao, 2014; Wittmann, 2014). And then the question comes up: does this temporal interval of finite duration move continuously along the arrow of time, or does it move in discrete steps along an abstract temporal axis?

There is another problem which is usually neglected: does the English word "present" cover equivalent connotations in Chinese, German or any other language? Can we disregard the language we are using in such discourses? Cultural neuroscience is dealing exactly with this problem of cultural specifics contrasting them with anthropological universals (Bao and Pöppel, 2012). Thus, we might even move around in a conceptual jungle defined by different languages, in which specific terms only superficially appear to be equivalent. The English "present" evokes different associations compared to the German "Gegenwart" or the Chinese "xian zai." "Present" is associated with sensory re*present*ations, whereas "Gegenwart" has a more active flavor; the component "warten" refers either to "take care of something" or "to wait for something," and it is thus also past and future oriented. "Xian zai" is associated with the experience of existence in which something is accessible by its perceptual identity, it implies a spatial reference indicating the "here" as the locus of experience, and it is also action oriented. Although the different semantic connections are usually not thought of explicitly, they still may create a bias within an implicit frame of reference (Pöppel and Bao, 2011).

Being already in the jungle of time with reference to the present only, the jungle gets deeper if further temporal phenomena are included. Is the experience of simultaneity as obvious as it appears at first sight? Simultaneity as a perceptual experience is for instance indicated by the philosopher Kant (1781/1787) in his "Critique of Pure Reason" when he analyses time as an *a priori* form of knowledge. We claim that there is nothing like such an experience. It would require that (at least) two independent experiences on a perceptual or conscious level at exactly the same time take place; this we consider as impossible. On the basis of a retrospective evaluation of perceived events we may conclude that they have been simultaneous; on this basis of reasoning, simultaneity is a theoretical construct with no experiential quality.

The conclusion that temporal experiences are often retrospectively constructed and not representing "experiences" at all, can also be drawnfrom experiments measuring temporal order threshold (Bao et al., 2013a, 2014a). If for auditory stimuli their temporal sequence has to be indicated, subjects may use a holistic strategy by perceptually fusing the two sequential stimuli with different frequency into one percept with tones going up or going down; on the basis of the direction of the perceptual movement, subjects reconstruct the sequence of events retrospectively. The same applies to the case when one refers to spoken Chinese as a tonal language: the second, third and fourth tones are characterized by frequency changes with immediate experiential quality defining the meaning of a word within a semantic frame; only retrospectively and on an abstract level the tones may be defined as auditory stimuli with typical changes of frequency and the changing order, but in the moment of hearing they are not distinguished on an analytical level.

There are more temporal phenomena which appear to be unrelated to others: why do experiences with the same objective duration may have different subjective durations? What happens when we are bored in which case our attention is drawn to the passage of time and time appears to slow down? But is it "time" that slows down? One can argue that attention is drawn to a reduced availability of information, and that only secondarily this mental state is interpreted as a slowing down of time itself. Thus, one is again confronted with a potential confusion of categories, one being experiential and the other being conceptual. Furthermore, what gives us the impression of a continuity of time (Pöppel, 2009)? Also in this case we have to address the question whether we deal with an experiential quality of continuous time, or whether we derive a concept of temporal continuity on the basis of what is represented in consciousness.

To overcome the apparent diversity of temporal experiences or concepts, time as defined in classical physics might provide a unifying principle; Newton writes (Newton, 1686): "Absolute, true, and mathematical time, of itself, and from its own nature, flows equably without relation to anything external." In this definition time is considered to be a medium or one-dimensional "container" within which temporal experiences are implemented. Such a mapping of subjective time onto Newtonian time has been dominant in psychological research (e.g., Pöppel, 1978); but then one is confronted with another problem: in modern physics one has to deal with different concepts of time as formulated for instance in the theories of relativity, cosmology, the second law of thermodynamics or dissipative structures (Ruhnau, 1994). Why should one select the classical concept of time as a unifying principle if others and more advanced ones exist? Furthermore, it is important to stress that there is no concept of a "now" in physics (Ruhnau and Pöppel, 1991); thus, a useful reference for a better understanding of psychological phenomena does not exist in physics.

Moving to another path in this jungle of time makes it even more unlikely to find a solution in physics: we can mentally travel to the past visiting previous experiences in our episodic memory; when we do so we discover that our pictorial memories are always related to specific places, that they have been imprinted by a strong emotion, and that we are pictorially confronted with ourselves in these images becoming our own doppelganger. Thus, these internal images are no longer copies of objective events from the past, because in reality we are physically never present in an image we have in front of our eyes.

There are even more temporal phenomena which have to be dealt with on the psychological level if one is looking for a unifying principle. Human behavior is embedded within geophysical cycles like the diurnal or annual rhythms which give rise to the experience of a day or a year which defines a subjective and objective frame for the temporal organization of life. These rhythms are controlled by biological clocks (Aschoff, 1965; Roenneberg and Aschoff, 1990), and they carry an evolutionary heritage in our "temporal genes." They are responsible for feeling embedded in the temporal structure of the natural environment.

## **IDENTITY AS NECESSARY CONDITION FOR HOMEOSTASIS IN TIME**

What could be an escape out of this jungle of diverse temporal phenomena? What might be a unifying principle that conceptually binds these phenomena together? We suggest leaving behind "time" as a conceptual frame for such an integrating enterprise. Instead, we want to propose as a unifying frame the concept of "identity" (Pöppel, 2010), i.e., the defining characteristics by which a thing or person is recognized as a persisting entity over time. The argument is based on the biological principles of homeostasis (Bernard, 1856/1957; Gross, 1998) and allostasis (Sterling, 2004) and their consequences for behavioral control. Homeostasis as originally suggested by Bernard (1856/1957) is essential for all biological processes; physiological systems like the regulation of body temperature in homoiothermic organisms have to be kept stable to guarantee the maintenance of life. Allostasis (Sterling, 2004) expands this concept by stressing the anticipative and adaptive regulation of bodily functions which unfold over time; bodies are designed for efficiency, and the prediction of future states is required. A similar concept of including future states has been suggested in a generalization of the reafference principle (von Holst and Mittelstaedt, 1950; Bao et al., 2014b); efference copies are compared with sensory reafferences to allow a continuous self-monitoring of behavior and anticipate future environmental constellations.

Taking this biological perspective every organism including every human being has to establish and maintain a homeostatic state throughout time. This necessarily implies a "self" (Pöppel, 2010), i.e., the identity of the organism to achieve this individual goal. As the homeostatic state may be violated because of perturbations by unexpected stimuli or changes of the metabolic state expressed for instance in the feeling of hunger, thirst or sexual desire, the organism constantly monitors its internal state in order to maintain homeostasis. What are the operating mechanisms for these activities to keep equilibrium? The answer to this question is based on a taxonomy of functions (Pöppel, 1989) which distinguishes between content and logistical functions; temporal processing represents a logistical function, whereas percepts or memories refer to content functions.

What an organism has to do for the creation and maintenance of homeostasis is to determine a functional state of the organism using sensory systems. This requires a time interval of finite duration within which sensory information is integrated. The challenges for the brain in creating identity become obvious if one considers the neuronal machinery for perceptual processing. Taking visual and auditory perception, one has to deal with the problem that the time required for the transduction of optic and acoustic information in the two modalities is different, being much shorter in the auditory modality (Pöppel et al., 1990). Furthermore, transduction in the retina is dependent on flux such that the necessary information to define a visual object with different areas of brightness arrives at different times in cortical areas. Thus, to create for instance the perceptual identity of another person whom we see and who talks, our brain has to overcome temporal uncertainty of the neuronal information. It is suggested that this logistical problem is solved by the use of neuronal oscillations (Pöppel, 2009; Pöppel and Bao, 2014). Stimulus-entrained oscillations with a period of some tens of milliseconds are used to integrate the neuronal information. Experimental evidence indicates that information within such temporal window is a-temporal in nature; the before and after relationship of events can not be extracted within such intervals. All information within one period of the oscillation is treated as co-temporal, thus, allowing the definition of a functional state. (This neuronal operation also implies that the brain steps out of a continuity of time as defined in Newtonian physics.)

Support for this notion comes from experiments on temporal order threshold in different sense modalities (Hirsh and Sherrick, 1961; Bao et al., 2013a, 2014a); only if some tens of milliseconds have passed, it is possible to indicate their correct sequence. Evidence for such a temporal mechanism is also provided by measurements of choice reaction time or pursuit eye movements (Harter and White, 1968; Pöppel, 1970, 1972; Pöppel and Logothetis, 1986); response histograms show multimodalities with modal distances of 30–40 ms, which are explained by an underlying process of discrete time sampling. These observations implying stimulus-triggered neuronal oscillations have also been made visible in studies on auditory evoked potentials (Galambos et al., 1981; Madler and Pöppel, 1987; Schwender et al., 1994).

To program a successful movement trajectory toward an anticipated goal in order to maintain or reach homeostasis, at least two such functional states have to be defined for reasons of comparison. A difference between functional states indicating different physical or psychological distances toward the goal suggests the direction of a movement trajectory, i.e., an action or more generally speaking a plan. If no difference is determined, no action is required. The necessary comparison between functional states is only possible if the distinct states keep their identity. On the basis of a comparison, a choice and a decision can be made for an action that is characterized by anticipating the consequences of such a decision. Anticipation implies the maintained identity of both actor and the goal of action. Thus, all sequential operations require the identity of functional states, the results of the comparisons, the content of the decisions, and the anticipated goals to be reached. If temporal processing is distorted, as can be observed in certain brain diseases (Teixeira et al., 2013), the consistency and coherence of our mental life may break down; pathological changes which may occur at different levels of temporal processing support the outlined mechanisms of cognitive control.

As suggested in a model of hierarchical temporal processing (Pöppel, 1997), a comparison between successive functional states has to happen within a"temporal window" (Pöppel and Bao, 2014) of a few seconds; if the delay between the states to be compared is too long, the representation of the first one may have faded away. We suggest that the "subjective present" in humans with the duration of a few seconds serves the purpose of meaningful comparisons. Experimental evidence suggests that a specific neuronal mechanism creates a temporal window with the duration of ∼3 s. A pre-semantic temporal integration mechanism provides a temporal stage on which conscious activity is represented. For every time window of up to 3 s, the identity of what we perceive, what we remember or what we think of is maintained.

Examples for such a temporal integration within a few seconds providing perceptual identity of what is represented come from the duration of intentional acts in humans (Schleidt et al., 1987; Nagy, 2011) and other higher mammals (Gerstner and Fazio, 1995), sensorimotor synchronization (Mates et al., 1994), and spontaneous speech (Vollrath et al., 1992), as well as observations of disrupted temporal integration in patients having suffered speech deficits (Szelag et al., 1997) or in autistic children (Szelag et al., 2004). Most importantly, this temporal window of some 3 s is also used to temporally integrate spatial attention. Using the paradigm of inhibition of return (IOR) it has been discovered that the perifoveal region of the visual field is characterized by a different attentional control mechanisn compared to the periphery (Bao and Pöppel, 2007); although the decay functions of IOR are different for the two attentional fields, they share the same time window (Bao et al., 2013b).

With the hierarchically connected neurocognitive machineries to define the temporal order and an integration interval for optimal comparisons (Pöppel, 1997), the brain also owns a mechanism to create experiences of what we interpret as different subjective durations. If in a pre-semantically defined temporal window of a few seconds (Pöppel, 2009) more or less information is integrated on the basis of defined functional states, subjective durations are retrospectively appreciated as having been long or short. However, to refer to such different durations is only possible if the functional states maintain their identity within the temporal window of 2–3 s. The derived impression of continuity of time on a presumably higher level of processing requires the semantic connection of what is represented in

successive temporal windows which again entails the identity of such successive contents.

The circadian clock (Merrow et al., 2005) creates a particular challenge with respect to individual identity as different physiological and psychological functions show different diurnal patterns (e.g., Lotze et al., 1999), and the same can be suspected for circannual cycles. The "phase-map" is such that an identity of the constellation of all psychological and physiological functions is only observed in intervals of 24 h; we are not "the same" throughout a day and possibly also throughout the year, and everybody returns to his or her psychophysiological self only at regular intervals defined by geophysical cycles. However, these regular fluctuations are usually masked and we can refer to our self effortlessly (Han and Northoff, 2009; Pöppel and Bao, 2011; Zaytseva et al., 2014). The creation of individual identity over time is made possible by functional memory systems operating both on an implicit and explicit level (Pöppel and Bao, 2011). The importance of memory systems for the creation of individual identity is documented also by time travels to the past when we visit images in our episodic memory. As indicated above, one is confronted in such time travels with the representation of oneself, with the own doppelganger, and it is suggested that this doubling of oneself is essential for the construction of personal identity: we double our self and in doing so we can refer to our self. Taken together, we suggest that the concept of identity provides a common frame for different temporal phenomena and, thus, allow an escape out of "the jungle of time."

### **ACKNOWLEDGMENTS**

The authors were supported by the Chinese Academy of Sciences (CAS Visiting Professorships for Senior International Scientists, 2013T1S0029) and the National Natural Science Foundation of China (Projects 31371018, 31100735, and 91120004).

### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 06 May 2014; accepted: 15 July 2014; published online: 29 July 2014. Citation: Zhou B, Pöppel E and Bao Y (2014) In the jungle of time: the concept of identity as a way out. Front. Psychol. 5:844. doi: 10.3389/fpsyg.2014.00844 This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Zhou, Pöppel and Bao. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## The consciousness state space (CSS)—a unifying model for consciousness and self

#### *Aviva Berkovich-Ohana1 \* and Joseph Glicksohn2,3*

<sup>1</sup> Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel

<sup>2</sup> Department of Criminology, Bar-Ilan University, Ramat Gan, Israel

<sup>3</sup> The Leslie and Susan Gonda (Goldschmied) Multidisciplinary Brain Research Center, Bar-Ilan University, Ramat Gan, Israel

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

#### *Reviewed by:*

Yuka Sasaki, Brown University, USA James M. Broadway, University of California Santa Barbara, USA Diego J. Cosmelli, Pontificia Universidad Católica de Chile, Chile Claire Zedelius, University of California Santa Barbara, USA

#### *\*Correspondence:*

Aviva Berkovich-Ohana, Department of Neurobiology, Weizmann Institute of Science, 234 Herzl St., Rehovot 76100,Israel e-mail: aviva.berkovich-ohana@ weizmann.ac.il

Every experience, those we are aware of and those we are not, is embedded in a subjective timeline, is tinged with emotion, and inevitably evokes a certain sense of self. Here, we present a phenomenological model for consciousness and selfhood which relates time, awareness, and emotion within one framework. The consciousness state space (CSS) model is a theoretical one. It relies on a broad range of literature, hence has high explanatory and integrative strength, and helps in visualizing the relationship between different aspects of experience. Briefly, it is suggested that all phenomenological states fall into two categories of consciousness, core and extended (CC and EC, respectively). CC supports minimal selfhood that is short of temporal extension, its scope being the here and now. EC supports narrative selfhood, which involves personal identity and continuity across time, as well as memory, imagination and conceptual thought. The CSS is a phenomenological space, created by three dimensions: time, awareness and emotion. Each of the three dimensions is shown to have a dual phenomenological composition, falling within CC and EC. The neural spaces supporting each of these dimensions, as well as CC and EC, are laid out based on the neuroscientific literature. The CSS dynamics include two simultaneous trajectories, one in CC and one in EC, typically antagonistic in normal experiences. However, this characteristic behavior is altered in states in which a person experiences an altered sense of self. Two examples are laid out, flow and meditation. The CSS model creates a broad theoretical framework with explanatory and unificatory power. It constructs a detailed map of the consciousness and selfhood phenomenology, which offers constraints for the science of consciousness. We conclude by outlining several testable predictions raised by the CSS model.

**Keywords: consciousness, time, awareness, emotion, self, default mode network, flow experience, meditation**

### **INTRODUCTION**

Every human experience, those we are aware of and those we are not, is embedded in a subjective timeline, and is tinged with emotion, be it the subtlest. At the same time, each experience inevitably evokes a certain sense of self, either minimal (i.e., nonconceptual first-person content, without personal identity) or expanded and autobiographic (i.e., personal identity and continuity across time). Human experiences which are devoid of a sense of time, phenomenal awareness, and emotional tone would largely fall either into a category of neuropathology, or of an altered state of consciousness. Certainly, time, awareness, and emotion are all necessary ingredients of consciousness and selfhood. But how are these all related to each other? Undoubtedly, an endeavor relating these concepts within one framework, which bridges phenomenology and neuroscience, is a presumptuous attempt. However, this is what we will cautiously try to propose here, a model named the consciousness state space (CSS), building on current formulations of consciousness and self, supported by neuroscientific evidence.

The model is rooted in a view of the embodied mind, held by both philosophers and cognitive neuroscientists (Varela et al., 1991; Damasio, 1999; Lakoff and Johnson, 1999; Cosmelli and Thompson, 2010), suggesting that consciousness behaves like a complex non-linear dynamical system (Varela et al., 1991; Thompson and Varela, 2001; Smith, 2005; Cosmelli et al., 2007) created by a state-space (Fell, 2004; Werner, 2009). As CSS is informed by both empirical evidence from cognitive neuroscience and phenomenological accounts, it is essentially a neurophenomenological model. Importantly, CSS is a theoretical model. Yet, it relies on a broad range of literature, hence has high explanatory and integrative strength, and helps in visualizing the relationship between different aspects of experience. This is in alignment with Revonsuo's (2003) proposition: "The science of consciousness should direct considerable resources to the systematic study of phenomenological issues, in order to first construct a detailed map of the phenomenal level of description. ... for the features of the phenomenal level (how it is structured, how it dynamically changes across time, and so on) offer top-down constraints for the science of consciousness in the search for potential explanatory mechanisms in the brain" (p. 3).

Briefly, CSS suggests that three dimensions, time, awareness, and emotion, create a state-space encompassing all possible total system behaviors, i.e., a repository of all potentially accessible phenomenological states. These, in turn, fall into two large categories of consciousness, each with its respective sense of self. Section A Dual Organization of the CSS describes the dual organization of the CSS, as well as its neural space. Section The Three Dimensions of the CSS describes the three dimensions of the CSS. Section The Dynamics within the CSS describes the typical antagonistic dynamic behavior of the system, as well as atypical behavior of the CSS, when the typical antagonistic relationship between the two categories is reduced, for example during the experience of flow and in meditation. In section A Comparison to Other Models of Consciousness we compare CSS to other theories of consciousness, to highlight its unique contribution. We conclude in section The Limitations, Predictions and Contribution of the CSS Model by outlining the model's limitations, as well as its contribution by providing examples of testable predictions.

### **A DUAL ORGANIZATION OF THE CSS**

### **A DUAL ORGANIZATION OF CONSCIOUSNESS, SELF, AND BRAIN ACTIVITY**

While avoiding philosophical definitions of consciousness (e.g., James, 1890/1950; Searle, 1994), which are beyond the scope of this paper, the term *consciousness* here generally denotes, as in previous neuroscientific approaches (Edelman, 2006; Boly et al., 2009), an experienced property of mental states and processes, which is lost during a dreamless deep sleep, deep anesthesia or coma. Consciousness and self-consciousness are tightly related, based on both philosophical accounts and cognitive theories (e.g., Gennaro, 1996; Natsoulas, 1998; Kriegel, 2004; Morin, 2006; Gallagher and Zahavi, 2008; Damasio, 2012). We are aware that the concept of self has many definitions and that there is no consensual framework for conceptualizing the various aspects of the self. Yet, we adopt here an increasingly accepted framework for the self, grounded in James' (1890/1950) differentiation between the self as "I," the subjective knower, a momentary enduring presence, and the self as "me," the object that is known, the self-concept and autobiographical identity. This framework distinguishes between the "minimal self" (MS), a self that is short of temporal extension, which is endowed with a sense of agency, ownership, and nonconceptual first-person content, and the "narrative self" (NS), which involves personal identity and continuity across time, as well as conceptual thought (Gallagher, 2000). Consciousness can also be divided into a simpler and a more complex form, each one of them supporting one type of self-experience. The first is coreconsciousness (CC), which supports the MS, its scope being the here and now. The second is extended-consciousness (EC), which supports the NS, and involves memory of past, imagination of future, and verbal thought (Damasio, 1999, 2012). Importantly, while CC is independent of the EC, and relies only on its exchange with the body (and environment), the EC is always dependent on CC (Damasio, 1999). Hence, the NS is dependent on the MS, but not vice versa.

In cognitive neuroscience the MS and NS have been attributed to various different neural processes. Following, we will refer to these neural spaces as Nms and Nns, respectively. While these neural spaces cannot yet be fully identified, there is accumulated knowledge suggesting main brain regions involved. NS has been conceptualized as self-referential processing, such as assessing one's personality, appearance or feelings and recognizing one's own face or name. The neural regions supporting self-referential processing are mainly the midline regions, including the medial prefrontal cortex (mPFC), posterior cingulate cortex (PCC) and precuneus (Gusnard et al., 2001; Northoff and Bermpohl, 2004; Northoff et al., 2006), as well as the temporoparietal junction (TPJ) and temporal pole (Christoff et al., 2011). The MS has been attributed to self-specifying processing, experiencing oneself as the agent of perception, action, cognition and emotion. The cortical regions suggested to be involved include those related to sensorimotor integration (such as motor and supplementary motor area—SMA) and proprioception (the insula), as well as higher-level regulatory regions, including dorsal anterior cingulate cortex (dACC) and dorsolateral PFC (DLPFC) (Legrand, 2006; Legrand and Ruby, 2009; Christoff et al., 2011). Other regions involved in the sense of agency include the TPJ and inferior parietal lobule (IPL) (Chaminade and Decety, 2002; Blanke and Metzinger, 2009).

Intriguingly, Nms and Nns can be related to a dual organization of the cortex. Accumulating evidence supports this notion, showing that thalamo-cortical networks can be divided into two, often antagonistic, global systems (Fox et al., 2005; Golland et al., 2007; Tian et al., 2007; Soddu et al., 2009): (i) a system of inwardoriented networks (the "intrinsic" or default mode network - DMN); and (ii) a system of externally-oriented, sensory-motor networks (the "extrinsic" system). Resting-state activity involves the DMN (Raichle et al., 2001; Greicius et al., 2003), a taskinhibited network related to self-reference and mind-wandering. The DMN includes a consistent set of five regions that comprise the mPFC, PCC, IPL, medial temporal lobe (MTL) including the hippocampus, and lateral temporal cortex (LTC) (Buckner et al., 2008). Task-induced neural activity is related to the dorsal attention network (DAN), which includes regions in the frontal eye fields, ventral premotor cortex, the supplementary motor area (SMA), superior parietal lobule, intraparietal sulcus, and motion-sensitive middle temporal area (Corbetta et al., 2008). Interposed between them is suggested to be the frontoparietal network (FPN), which includes the anterior PFC, DLPFC, dorsomedial superior frontal, ACC, anterior IPL, and anterior insular cortex (Vincent et al., 2008). The FPN cooperates with either one of these typically antagonistic systems, possibly integrating information from, and adjudicating between, these two potentially competing brain systems (Vincent et al., 2008; Spreng et al., 2010; Smallwood et al., 2012). The FPN can be broken down into two sub-networks (Seeley et al., 2007), the "executive control network" (DLPFC and IPL) and the "salience system" (anterior insula and ACC), with the latter also being specifically attributed the role of switching between the intrinsic and extrinsic systems (Menon and Uddin, 2010). Following, we will refer to this interposed network generally as Ni.

### **THE CSS CONTAINS TWO CONCENTRIC SPHERES**

Based on the dual organization of consciousness, self, and underlying neural activity, CSS is organized into two concentric spheres around the body. The concentric organization depicts the reliance of each sphere on the previous level: CC relies on the body, while EC relies on CC. The inner sphere of CC/MS is phenomenologically related to the body, experiencing agency and momentary sensory experiences. It is embodied. The inner sphere is surrounded by the EC/NS sphere, which is phenomenologically further away from the body, in the realm of conceptual thought, language, memories and imagination, and relies more on mental representation, rather than actual sensory experiencing (**Figure 1A**).

We refer more generally to the neural space, mainly building on current neuroscientific knowledge. As to the neural space of the CC/MS (Nms), this is the point where the physiological condition of the body, which is in constant exchange with the environment, becomes available to the brain. This mainly relies on sensori-motor integration, i.e., convergence of action and perception, allowing one to perceive the sensory consequences of one's action through action monitoring, and proprioception perceiving the body state. The proposed regions of the brain involved in sensori-motor integration are SMA and pre SMA (Legrand, 2006; Ferri et al., 2012), while proprioception involves the somatosensory and insular cortex (Craig, 2002, 2009) as well as a deep portion of the posterior cingulate (Parvizi et al., 2006; Damasio and Meyer, 2009). The neural reference space of the EC/NS (Nns) is largely suggested to involve the DMN. It should be noted, however, that some posterior regions of the DMN, including the IPL and precuneus, are argued to be involved in both NS and MS due to their roles in agency (Chaminade and Decety, 2002) and CC (Damasio and Meyer, 2009), respectively. These regions can be viewed as a mutual reference space for both spheres. Between the Nms and Nns there is the interposed Ni, largely related to the control FPN system, which shifts between collaboration with both (**Figure 1B**).

**FIGURE 1 | (A)** The concentric organization of the Consciousness State Space (CSS). The central point denotes the body. Around it is the core-consciousness (CC) and minimal-self (MS) sphere in white, surrounded by the extended-consciousness (EC) and the narrative-self (NS) sphere in gray. This circular organization depicts a shorter and longer psychological distance from the body for the CC/MS and EC/NS, respectively. **(B)** The suggested neural space, depicted as an oval. Within it we identify three main networks, two of them supporting the MS and NS (Nms and Nns, respectively), and the third interposed between them (Ni). The body's condition becomes available to the neural space through Nms. Ni switches between synchronization (marked by circular arrows) with the two other networks. Dashed lines denote alternation between the neuronal states. The top state corresponds phenomenologically to CC/MS, and the bottom to EC/NS. For the suggested brain regions for each neural space see section A Dual Organization of Consciousness, Self, and Brain Activity.

Another important feature of the CSS is the existence of two simultaneous trajectories of experience, each in one of the spheres. This will be further developed in sections Second Dimension of the CSS—Awareness and The Dynamics within the CSS.

## **THE THREE DIMENSIONS OF THE CSS**

A number of theories have described aspects of consciousness within a state-space paradigm, choosing as dimensions different parameters describing the system's behavior. For example, Allan Hobson and colleagues (Hobson et al., 2000) have introduced a three-dimensional state-space model for the classification of mental states during sleeping, dreaming and wakefulness. The dimensions of this model are activation (the information processing capacity of the system), information flow (the degree to which the information processed comes from the outside world and is or is not reflected in behavior), and mode of information processing (the way in which the information in the system is processed). This model shows how alertness, drowsiness and sensory restriction are located within the state-space in relation to each other, and suggests the underlying brain structures and chemistry. Another example was proposed by Wackermann (1999) who developed a three-dimensional global approach to representing the electrical activity of the brain. This model's dimensions are: - (a measure of global field strength, reflected by the data cloud in the state space, in µV), (a measure of global frequency of field changes, reflected by the density of distribution of the momentary states along the trajectory speed of field change, in Hz), and (a measure of spatial complexity, implying the simplicity/complexity of the data cloud, dimensionless). Using this three-dimensional complexity description, the electrical signature of a brain's macro-state may be represented by a trajectory in a three-dimensional space, which facilitates electrophysiological data reduction. More closely related to the current model, Fell (2004) suggested a three-dimensional statespace that characterizes states of phenomenal awareness, with the three state dimensions being the amount of synchronized electrophysiological activity within different frequencies. In this model, phenomenal awareness is related to decreased delta and alpha and increased gamma activity.

We argue that a CSS is created by merely three phenomenological dimensions: (i) subjective time—at one end past and at the other future; (ii) awareness—at one end high and at the other low phenomenal access; and (iii) emotion—at one end pleasant and at the other unpleasant (**Figure 2**). Whereas the other state-space models are quantifiable—that is, the dimensions are continuous and lend themselves to quantification—the CSS model employs dimensions, which refer to psychological distance from the body. Further, the two concentric spheres of CC/EC and MS/NS, are both structured by this same 3D coordinate system, as outlined in length for each dimension separately. Pointing out these particular dimensions might raise the question as to whether, and how, other important phenomena can be manifested in terms of these three particular dimensions. We answer these questions by three arguments.

Firstly, we argue that these three dimensions are phenomenologically distinct. This is not to say that we think of these

dimensions as being self-sufficient for consciousness. We fully acknowledge the crucial role of interconnectivity among these phenomenological aspects to create the unity of consciousness (Searle, 1994; Dainton, 2006). Furthermore, occasionally these dimensions rely on similar, non-specialized, brain regions (Pessoa, 2008). Yet, these dimensions can be characterized as being phenomenologically distinct, while other phenomena could be considered to result from the interplay among these three dimensions. A major example is the general construct of cognition, which includes learning, memory, thinking and language (Mayer et al., 1997), and which is also closely linked to emotion (Pessoa, 2008), hence its different facets are spread over the entire CSS.

Secondly, there are afferent and efferent functions, which are tightly related to, but occur without, consciousness. This includes action (motor output) and language, as speech (either mental or executed), which can also be considered as an action (Jones and Fernyhough, 2007). Since one becomes aware of one's action only after it has been initiated (Libet, 1985), and has sensory consequences (Frith et al., 2000; Legrand, 2006; Carruthers, 2009), these functions are excluded from CSS.

Thirdly, some key mental functions might "overlap" with, or be closely related to, the dimensions presented here. For example, attention is closely related to the awareness dimension, and is captured by it, as will be explained in the section devoted to awareness. Another example is spatial cognition: while there is much evidence which tightly links this to temporal cognition, both phenomenologically and in the neural space (Barsalou, 1999; Glicksohn, 2001; Boroditsky and Ramscar, 2002; Walsh, 2003; Glicksohn and Myslobodsky, 2006; Casasanto, 2008; Srinivasan and Carey, 2010), there is also substantial research arguing that spatial perception is actually the more fundamental dimension (e.g., Srinivasan and Carey, 2010). However, CSS depicts human experience, wherein subjective "time traveling" is far more frequent compared to "space traveling." Our life memories are ordered along a time-line, and not a space-line (Wheeler et al., 1997; Markowitsch, 2003). For that reason, CSS includes time as one of its dimensions, and not space.

Next we describe the three dimensions of CSS in detail. Importantly, each dimension behaves differently in the two spheres, in both phenomenology and its neural space, as subsequently outlined.

### **FIRST DIMENSION OF THE CSS—TIME**

Consciousness would be inconceivable without temporality, as time is an omni-present structural feature of consciousness (James, 1890/1950). As James wrote: "The knowledge of some other part of the stream [of consciousness], past or future, near or remote, is always mixed in with our knowledge of the present thing. . . . These lingerings of old objects, these incomings of new, are the germs of memory and expectation, the retrospective and the prospective sense of time. They give that continuity to consciousness" (p. 606–607). Past or future events can be activated in experience voluntarily, and this constant mental time travel aids one in understanding the meaning of present happenings: "I don't simply exist in the present and happen to have the capacity to envisage the future and remember the past. Rather, human reality is characterized by a kind of temporal stretch. The past continually serves as the horizon and background of our present experience, and when absorbed in action, our focus, the center of our concern, is not on the present, but on the future goals that we intend or project" (Gallagher and Zahavi, 2008, p. 86).

We claim that this time dimension is sufficient, and can also account for the spatial aspect of experience. First, at any time point, one sees the world from only one spatial perspective (Revonsuo, 2003). Second, each episodic memory has one (and only one) spatiotemporal content (Russell and Davies, 2012). Third, the phenomenal fields of different modalities are spatially and temporally integrated, so that different features belonging to the same object are realized in the same location and time (Fingelkurts et al., 2010). Thus, spatial experience is fully integrated with temporal experience in three important ways.

Several cognitive models have been proposed for time-consciousness, most of them variants of a pacemaker– accumulator clock, where experienced duration is represented by a pacemaker which produces a series of pulses recorded over a given time span (Zakay and Block, 1997; Glicksohn, 2001). Yet, competing cognitive models propose that memory decay processes are involved in time perception (Wackermann and Ehm, 2006). A contrasting view to the cognitive models is Varela's (1999) dynamic model of the experience of time, according to which three different scales of duration contribute to a cognitive act: the elementary (10–100 ms), integration (0.5–3 s) and narrative (>10 s) scales. Neurophysiologically, the first two correspond to neuron-level electrophysiology and synchronization. These further correspond to the experienced present. Here, we adopt a dynamic view, akin to Varela's view. However, we differentiate between two time scales, combining Varela's elementary and integration scales into one: the immediate perception of the present moment (<3 s; see Pöppel, 1997), contrasted with the longer time scale. The longer time scale refers to the re-presentation of experience, while the second refers to the immediate perception of the present moment (**Figure 3A**). By "re-presentation" we mean that experience, with its full-blown, present-moment, multi-dimensional vividness, is being "projected," or re-presented (and not represented) into another subjective time, either past or future. The fact that we usually cannot both recall and be "here and now" simultaneously is manifested by placing each of the two phenomenological categories into the two different spheres of CSS. Following, we describe in more detail these two categories along the time continuum.

The first temporal category refers to the longer time scale, and involves the re-presentation of experience in the past and in the future. Unlike immediate perception, this is psychologically further away from the body: when one's conscious awareness re-lives the past or the future, one's conscious awareness is decoupled from the body (which experiences the now). It encompasses mental re-presentations of other "nows," relived or imagined. The intriguing ability of the human mind to mentally travel through time, enabling one to relive past experiences through memory, or project oneself into the future by generating a prediction based on memory, has been also referred to as autonoetic (self-knowing) consciousness (Wheeler et al., 1997; Stuss et al., 2001; Markowitsch, 2003). Hence, this experience pertains to the NS, and is within the EC/NS sphere. At the past end, we find re-presentation of the far or near past, by means of retrospective memory retrieval. This includes either true or false memories, which are experienced as being phenomenologically similar (Lampinen et al., 1997). Here, we adopt Conway's phenomenological description of autobiographic memory (Conway and Pleydell-Pearce, 2000; Conway, 2009), as well as Tulving's (1985) conceptualization of long-term memory. Conway relates autobiographic memory with the NS (Conway, 2005). Autobiographic memory consists of recollected episodes from one's life, which are based on both episodic as well as semantic memory (Tulving, 1985; Conway and Pleydell-Pearce, 2000; Conway, 2005). Autobiographic knowledge comprises several levels of categorization. First is event-specific knowledge, which is a summary record of sensory-perceptual–conceptual-affective processing derived from working memory, which is predominately re-presented in the form of visual images (for example, a specific restaurant). Event-specific knowledge, in turn, is contextualized

within a general event (e.g., during the vacation in Greece). The general event, in turn, is associated with one or more lifetime periods that locate the more specific knowledge within an individual's autobiographical memory as a whole (e.g., it was just after our marriage) (Conway and Pleydell-Pearce, 2000; Conway, 2005). What is common to all these levels of autobiographic memory is the phenomenal feeling of remembering: "the feeling signals the state in an experiential way. Recollective experience, the sense of the self in the past and the episodic imagery that accompanies that sense, indicate to the rememberer that they are in fact remembering and not daydreaming, fantasying, or in some other non-memory state" (Conway, 2005, p. 614). At the other end of the time continuum, the future, we find prospective memory, which is memory for future intentions (Glicksohn and Myslobodsky, 2006), and refers to the functions that enables a person to carry out an intended act after a delay (Burgess et al., 2001). Another ubiquitous phenomenon is the generation of predictions and future simulations based on previous autobiographic knowledge (Schacter and Addis, 2007; Schacter et al., 2007), sometimes termed "proaction" (Bar, 2007, 2009). Mental time-travelling to the future and the past can be actually "experienced" in the "here and now" (Gilbert and Wilson, 2007): if the mental simulation is strong enough, the imagined or recollected images can evoke bodily reactions. At the same time, actual sensory stimulation from the environment is blocked, and one experiences sensory decoupling from the environment (Smallwood et al., 2007).

We now turn to the second category, the immediate perception of the present moment (<3 s), which pertains to the CC/MS sphere. Various models for the phenomenology of immediate time perception have been proposed (Dainton, 2008). One major category is retentional models: our experiencing of change occurs within episodes of consciousness which themselves lack temporal extension, but whose contents coordinate with past and future by virtue of their place in the temporal structure (Dainton, 2008). Specifically, we adopt the retentionalist model for consciousness of time outlined by Gallagher and Zahavi (2008), which assumes a Husserlian view: the immediate sensation, or the "primal impression" is combined with retention (being aware of the "just-passed" slice of the experience) and protention (being aware of the "just-about-to-be"). A perception cannot merely be a perception of what is now, but must include a retention of the just-passed and a protention of what is about to occur. Importantly, retention and protention are not memory, or imagination, which re-present the experience. Rather, they are actual experience. Unlike long-term memory and expectation, they are involuntary and automatic processes, and they could be argued to be working memory (Vogeley and Kupke, 2007; Gallagher and Zahavi, 2008). This experience is related in our model to the MS.

Turning to the neural space, the first category of autonoetic consciousness, within the EC/NS sphere, should involve the Nns. In contrast, CSS predicts that the immediate perception of the present, within the CC/MS sphere, is related to the Nms and bodily processing. As subsequently presented, these predictions are confirmed by neuroscientific evidence.

The most important neural structure for memory is the hippocampus, the locus of interaction between working memory and long-term memory (Fell and Axmacher, 2011). It has been proposed that memory initially depends on the hippocampus. However, with increasing time, the hippocampus becomes less important, and the involvement of multiple cortical regions increases, including the medial frontal gyrus and precuneus (Smith and Squire, 2009). Yet, new findings confirm the important role of the hippocampus even in retrieval of long-term, established memories, in collaboration with the ACC (Suzuki and Naya, 2011). In a nutshell, the hippocampal complex is essential for encoding, retaining, and recovering experiences, enabling the immediate subjective and vivid experience. Other regions, mainly the prefrontal cortex, select, organize, help retrieve, monitor, and verify the hippocampal recollection (Moscovitch, 2008). Though both recent and remote memories are associated with hippocampal activation, it was found that activations associated with more recent memories cluster at the anterior hippocampus, whereas those associated with more remote memories are distributed across its length (Gilboa et al., 2004). Not only memory, but also planning involves the hippocampus, as well as frontal and parietal structures. Strikingly, there is an overlap between memory systems and the network involved in foresight, and these two overlapping regions also overlap with the DMN (key component in Nns), including the hippocampus, mPFC, precuneus and lateral parietal cortex (Bar, 2007, 2010; Schacter and Addis, 2007; reviewed by Schacter et al., 2007). Another line of research, on mental time traveling and "self-projection," revealed the involvement of the IPL (Nyberg et al., 2010), and the temporo-parietal junction (Arzy et al., 2009), which are key regions for selfreferential processing and which are considered components of the DMN.

Turning now to the cortical regions, which have been suggested to be involved in the immediate perception of the present moment, Rubia and Smith (2004) emphasize the DLPFC, ACC, SMA, and IPL in their review of the literature. The IPL is also strongly suggested by others who review the literature (Walsh, 2003; Oliveri et al., 2009). Another suggested region is the insula (Craig, 2002, 2009; Wittmann, 2009), relating cognition of duration with proprioception. This proposition was supported by a functional magnetic resonance imaging (fMRI) study, showing a linear build-up of neuronal activation in the insula during a time reproduction task (Wittmann et al., 2010) and by an anatomical study (Gilaie-Dotan et al., 2011), showing that the gray matter volume of the right sensory cortex is correlated with the ability to discriminate time intervals. In addition to the literature on time perception, we consider Baddeley's (1992, 2003) influential model of working memory. Here, again, we find that the DLPFC plays an important role, as an executive control system, assisted by two subsidiary storage systems: the phonological loop and the visuospatial sketchpad (including right and left IPL and premotor cortex, respectively), both of which store perceptual information. Indeed, the DLPFC is believed to provide a buffer to hold information in mind, and to order it in space-time (Dehaene and Naccache, 2001). To conclude, as hypothesized, all the neural regions that are related to momentary experience of time, as well as to working memory, are within the FPN and DAN, considered as key elements in Nms and Ni, and in contrast, autonoetic consciousness and prospective memory involves the DMN, as a key element in Nns.

### **SECOND DIMENSION OF THE CSS—AWARENESS**

Awareness is a primary feature of consciousness, being the subjective experience of internal phenomena, a perception of the field of inner and outer events that encompasses one's reality at any given moment, the state of perceiving (Laureys, 2005; Cohen and Dennett, 2011). Awareness can be largely categorized into two types, following the influential conceptualization of Block (1995, 2007), represented in CSS as the two sides of the awareness dimension. The first is access awareness, which corresponds to states that can be reported on, by virtue of high-level cognitive functions such as memory and decision making, and which necessitates attention. The second is phenomenal awareness, related to private first-person experience, and occurs without—or with very little—attention (Kouider et al., 2010). In contrast to this division, awareness could also be conceived of as a graded phenomenon (Kouider et al., 2010): at one end expanded awareness, when all levels of relevant processing are accessible, and at the other end complete non-awareness, when all levels of processing are not accessible. Intermediately, there is partial awareness, combining awareness at some level and unawareness at another level of processing.

A possible solution to the current debate whether top-down attention is necessary for consciousness (Cohen et al., 2012a,b) or if these are two independent processes (Koch and Tsuchiya, 2007a,b; Tsuchiya et al., 2012) could be settled if we consider that phenomenal awareness can emerge without top-down attention (Aru and Bachmann, 2013), in contrast to access awareness. This would support the notion that attention allows information to be more fully transmitted across cortical regions than unattended information (Cohen et al., 2012b), hence is required for access awareness. This argument supports our proposition that the awareness dimension stands for attention as well.

In the following, we describe various phenomenal states along the awareness continuum, moving from no phenomenal access to high phenomenal access (**Figure 3B**). In doing so, we largely rely on Gallagher and Zahavi's (2008) account of prereflective and reflective consciousness, as well as on Morin's (2006) social/personality model, describing degrees of consciousness based on several theories (including Brown, 1977; Natsoulas, 1998; Schooler, 2002). Starting at no phenomenal access, we find outside CSS states of consciousness in which there is no phenomenal awareness to either external or internal input. These include the dreamless portion of deep sleep, coma, anesthesia, vegetative state, epileptic loss of consciousness, and somnambulism. When one regains awareness, there is still no access or memory as to external or internal happenings during that state. Further along the continuum, there are preconscious states and subliminal experiences. A division should be drawn between inaccessible internal and external input. Processing of internal input is conducted within the EC/NS sphere, and is referred to here as "subliminal awareness." This includes normal states of day dreaming, as well as pathological states such as dissociation. Processing of external input is conducted within the CC/MS sphere, and is referred to as "subliminal sensory awareness." These states include the natural decoupling of attention from sensory processing (Smallwood et al., 2007), as can occur during driving. All of these states are not accompanied by top-down attention.

Next, we describe the access states along the awareness axis, which can be conceptual or non-conceptual (Kapitan, 2006), as subsequently detailed. First, within the CC/MS sphere, there is first-order awareness, also called pre-reflective awareness (Gallagher and Zahavi, 2008). This is an implicit and direct awareness to experience, prior to any reflection on the experience. In this state, according to Morin (2006), one will directly be attentive and process external input from the environment, without conceptual elaboration of the mental events that are taking place. Hence, the organism will be totally immersed in experience. These states are accompanied by top-down attention to external input (Chun et al., 2011). States along the access awareness within the EC/NS sphere involve second-order awareness, also called reflective awareness (Gallagher and Zahavi, 2008). This is an explicit, conceptual, and objectifying awareness, which is accompanied by focal attention to internally generated input (Chun et al., 2011). In this state, one attends directly to the cognitive experience itself. It is described by Morin (2006) as the capacity to become the object of one's own attention, a process that occurs when an organism focuses not on the external environment, but on the internal milieu. In its extreme form, it becomes meta-awareness, being aware that one is aware (Morin, 2006).

We rely on neuronal global workspace theory (Dehaene and Naccache, 2001; Dehaene et al., 2006) to describe the corresponding neural space, suggesting that conscious access is produced through the interaction between specialized neural subsystems and a multimodal limited capacity global workspace (Baars, 2002, 2005), the FPN (the key component of Ni). During states of phenomenal awareness, including subliminal and preconscious states, activation due to internal or external input does not involve the FPN. In contrast, during states of access awareness, synchronized activity increases in the FPN, which becomes capable of guiding intentional actions including the production of verbal reports. The transition between phenomenal and access awareness is sharp, as expected in non-linear dynamic systems (Dehaene et al., 2006). Furthermore, CSS posits two simultaneous trajectories, one in the EC and the other in the CC. As these two trajectories are usually antagonistic, habitually one has access awareness only to the phenomenology related with one of the trajectories, while the phenomenology related with the second trajectory continues its activity at a sub-threshold level, within the phenomenal awareness continuum (**Figure 4**). Hence, the brain alternates dynamically, shifting awareness from internal to external processing. The actual phenomenal experience is constantly dictated by the neural space that is more active at the moment, and which is synchronized with the Ni. When one is immersed in "intrinsic" (i.e., Nns) activity, one is "decoupled" from extrinsic processing (i.e., Nms), which nevertheless continues (Smallwood et al., 2007; Vanhaudenhuyse et al., 2011). And vice versa, while externally engaged, the intrinsic system is continuously activated in a "sub-threshold for awareness" manner. Indeed this is what Singer has been proposing from the early 60s, in his exposition of daydreaming (Singer, 1966). As he has more recently suggested (Singer, 2009, p. 196), "Many years ago I proposed that what we

**FIGURE 4 | (A)** The two trajectories in the CSS are shown (red denotes the CC/MS trajectory, and orange the EC/NS trajectory), with their dynamic behavior. A full line denotes access awareness, while a dashed line denotes phenomenal awareness. The figure shows a quarter of the CSS, depicting the future/unpleasant space. Gray denotes the CC/Ms 3D space. The three time points demonstrate the typical antagonistic behavior of the two trajectories. At t0, the CC/MS trajectory is under the threshold for access awareness; hence one has access awareness to the EC/NS trajectory. For example, one is driving, decoupled from sensory awareness (present moment and relatively neutral emotionally) while mind wandering, imagining a future unpleasant experience. At t1 there is a threat on the road, resulting in a phase transition: the CC/MS trajectory arrives at the threshold for access awareness, and at the same time, the EC/NS trajectory shifts under the threshold for access awareness. At t2, the CC/MS trajectory expresses negative arousal due to the danger, while the EC/NS trajectory becomes even more unpleasant due to self-criticism for lack of awareness to the driving, continuing at a subliminal level of awareness; **(B)** a schematic diagram of the neural space behavior over the same specific three time points. N1 corresponds to the neural space of the CC/MS sphere, and N2 to the neural space of the EC/NS sphere. At t0, FPN is collaborating with N2 (one phenomenally experiences thoughts). At the short transition at t1, all networks are at threshold level for access awareness (one phenomenally experiences "no thoughts"), and at t2 the FPN collaborates with N1 (one phenomenally experiences high concentration while driving).

now call the brain's default network may be almost continuously active at a subthreshold level." The FPN alternates between cooperation with the intrinsic or the extrinsic systems. This notion is supported by the findings that a systematic impairment of FPN was found in altered states of consciousness, such as sleep, anesthesia, coma, vegetative state, epileptic loss of consciousness, and somnambulism (summarized by Boly et al., 2009), all of these being states which are considered outside the model's awareness dimension. This suggests that an intact FPN, enabling access awareness, is a prerequisite for a normally functioning CSS.

In relation to this neural space, both second-order awareness (within the EC/NS sphere, upper side of continuum) and mind wandering, i.e., subliminal awareness (within the EC/NS sphere, lower side of continuum) have been related to DMN activity (Nns). However, what differentiates between states has been shown to be DLPFC activation, which characterizes thoughts that occur with access awareness (Smith et al., 2006; Christoff et al., 2009). The suggested role of FPN as switching collaboration between the two neural spaces which support the two phenomenological spheres along the access awareness is illustrated by the following studies. In relation to the inner sphere of CC/MS, an fMRI study (Ferri et al., 2012) showed that the "bodily-self" (self rooted in bodily motor experience) recruits pre SMA, SMA and insular regions, belonging both to the FPN and DAN systems (Ni and Nms, respectively). In relation to the outer sphere of EC/NS, an fMRI study emphasized the importance of the DLPFC (Ni), as participants learned to regulate its activation by turning their attention toward and away from the contents of their own thoughts, or their DMN-intrinsic (Nns) system (McCaig et al., 2011).

### **THIRD DIMENSION OF THE CSS—EMOTION**

Emotion and consciousness are considered by many to be inseparable (e.g., Damasio, 1999; Lambie and Marcel, 2002; Panksepp, 2005; Barrett et al., 2007; Tsuchiya and Adolphs, 2007), as each conscious state is endowed with some form of emotion, to the point that even the perceptual representation of everyday objects carries subtle affective tone (Lebrecht et al., 2012). Put in the words of Searle (1994, p. 7), "part of every normal conscious experience is the mood that pervades the experience. It need not be a mood that has a particular name to it, like depression or elation; but there is always what one might call a flavor or tone to any normal set of conscious states."

Emotion states are generally agreed to bear two important phenomenal features, the one is mental and the other bodily, grossly speaking. Hedonicity is both intrinsic to bodily states, and depends on the interpretation placed on them. A similar differentiation has been done with many, albeit calling it by various names. Schachter and Singer (1962) described emotional experience as a combination of general arousal, and the cognitive attribution of the cause of this arousal. Similarly, Mandler (1984) distinguished between non-specific arousal, awareness of which provides the intensity of the emotional experience, and the evaluative structure (cognitive interpretation of the situation), which provides the particular content and quality of emotional experience. Damasio (1999) views emotional experience as consisting of sensory changes that occur in the viscera and internal milieu (which he calls emotions) and the mental image of these sensory patterns (which he calls feelings). According to Lambie and Marcel (2002), any emotional state is defined by a combination of two things: the bodily action readiness (and its representation) and the evaluative description (a mental representation). The first includes certain bodily and brain systems which are activated in response to stimuli (chiefly the limbic, autonomic, hormonal and aspects of the skeletal nervous system). The second, the evaluative description, is an appraisal leaving a record, a description of how one's concerns or one's self have been affected by the event. This is a mental representation. Barrett's (Barrett, 2006; Barrett et al., 2007) view of emotion involves two operations. The first is called "coreaffect," which includes bodily fluctuations that are represented in the brain. The second is "conceptualization," a process by which stored representations of prior experiences (i.e., memories, knowledge) are used to make meaning out of sensations in the moment. To summarize, there is general agreement that emotional states bear two important phenomenal features, one mental and the other bodily. Following, we will refer to these two components of emotion as valence (a subjective feeling of pleasantness or unpleasantness) and arousal (extent of bodily excitation), respectively.

We suggest that the manifestation in CSS of these two phenomenological qualities of the emotional experience, namely arousal and valence, is within the CC/MS and EC/NS spheres, respectively (**Figure 3C**). Within the CC sphere, arousal increases in two directions stemming from a minimal degree of arousal at the center of the emotion continuum. Valence, manifested within the EC sphere, increases toward the pleasant and unpleasant ends of the continuum. Each emotional experience has an arousal component, namely a bodily and sensory element, and valence, or a mental representation, in line with the above formulations of the phenomenology of emotions, and with the two CSS trajectories. The relationship between valence and arousal has been subject to a long debate, and various models have been proposed. A classical description is Russell's (1980) circumplex, suggesting an orthogonal relationship. In another framework, Lewis et al. (2007) suggest that aspects of valence generate a U-shaped curve with arousal. According to such a framework, strong positive valence is accompanied by strong positive arousal, and similarly for the negative aspects. Yet, others suggest that the dissociation between valence and arousal might be an issue of measurement more than reflecting distinct qualia underlying emotional experience (Larsen et al., 2003; Kron et al., 2013). For example, Russell (1989) has explicitly opposed the idea that arousal is only a physiological concept, writing (p. 106), "... there is no more reason to speak of arousal as strictly physiological and pleasure-displeasure as strictly mental then there is to express it the other way around." In comparison to the ongoing debate, the CSS model accounts for both valence and arousal, and allows for an orthogonal relationship between them. This means that each point on the arousal continuum could be in principle accompanied by any simultaneous point on the valence continuum, as it depends on the interpretation placed on it, which might be multiple for any given bodily state (as accounted by Schachter and Singer, 1962). As for the Lewis et al. U-shaped proposed relationship, we propose this should be rejected because negative arousal can sometimes be accompanied by positive mental evaluation, as in Schachter and Singer (1962), or as in the example of watching safely a horror movie. Further, the interplay between the emotion and awareness dimensions within CSS predicts a novel relationship between arousal and valence, namely antagonism: emotional states involve both evaluation and the products of arousal, but both of these need not be simultaneously present in experience or awareness, and certainly are not always experienced as such. This means that one can have access awareness to either the arousal or valence aspect at each specific moment and state, and that access awareness alternates between them. As a trait, one can be more prone to emphasize either the arousal or the valence, as shown by Barrett (1998). In addition, the interplay between the emotion and awareness dimensions creates a wide spectrum wherein emotional experience can be classified in terms of accessibility, as suggested by Lambie and Marcel (2002), Damasio (1999) and Frijda (2009), from non-conscious emotions, through phenomenal emotions (1st-order emotional experience), to awareness of emotional experience (2nd-order, emotional experience).

CSS predicts that the neural space for the arousal component of emotion should be related to the Ni and Nms (their key components being DAN and FPN, respectively), while the valence component should be related to the Nns (i.e., DMN). The evidence for that is based on a meta-analysis conducted by Barrett and colleagues (Barrett et al., 2007; Kober et al., 2008; Lindquist et al., 2012), stemming from a constructionist approach to emotion, where the assumption is that emotional mental states result from an interplay of more basic psychological processes that may not, themselves, be specific to emotion (Barrett et al., 2007; Kober et al., 2008; Lindquist et al., 2012). According to this meta-analysis, the distributed network involved in realizing core-affect includes several subcortical, as well as cortical regions: the amygdala, which signals whether exteroceptive sensory information is motivationally salient; the anterior insula, which plays a key role in representing core affective feelings in awareness based on its role in the awareness of bodily sensations and affective feelings; portions of the orbitofrontal cortex, as a site that integrates exteroceptive and interoceptive sensory information to guide behavior; ACC, and more specifically subgenual ACC, regulating somatovisceral states, pregenual ACC, as a visceromotor (i.e., autonomic) control area involved in resolving which sensory input influences the body when there are multiple sources of sensory input, and anterior midcingulate cortex (a part of the FPN), delivering sources of exteroceptive and interoceptive sensory information to direct attention and motor response. The regions suggested by this constructionist approach fall largely within the interoceptive/exteroceptive processing network, as well as the FPN, as suggested by CSS.

In Barrett's (Barrett et al., 2007; Lindquist et al., 2012) view, the mental representation (valence) is specifically related to the DMN: "In our model, categorization in the form of situated conceptualization is realized in a set of brain regions that reconstitutes prior experiences for use in the present. This set of brain regions has also been called the 'episodic memory network' or the 'default network'... this psychological operation makes a prediction about what caused core affective changes within one's own body or what caused the affective cues (e.g., facial actions, body postures, or vocal acoustics) in another person, and this prediction occurs in a context-sensitive way (with the result that core affect in context is categorized as an instance of anger, disgust, or fear" (Lindquist et al., 2012, p. 129). Another important network for categorization and emotional perception includes the anterior temporal lobe (ATL) and ventrolateral prefrontal cortex (VLPFC). Further, the DLPFC is postulated as being involved in mental states of attending to emotional feelings or perceptions, and holding affective information in mind in order to categorize it (Lindquist et al., 2012). Hence, the constructionist approach to emotion provides support for CSS, both regarding the categorization of emotions as being coupled with the DMN, as well as attributing to the FPN the role of mediating the activity within the two CSS spheres.

## **THE DYNAMICS WITHIN THE CSS**

### **TYPICAL CSS DYNAMICS**

The CSS model is a dynamic system, with rich self-organizing properties. One novel aspect of CSS is the suggestion that each sphere functions as a separate dynamic system, with its own trajectory over time (**Figure 4A**). The two trajectories are simultaneously present, one within the inner sphere of CC/MS, and the other in the surrounding sphere of the EC/NS. These trajectories are usually antagonistic, and phenomenal awareness switches between them, as elaborated in section Second Dimension of the CSS—Awareness. In the neural space, this is manifested by the collaboration, through synchronization, of the Ni with either Nms or Nns (**Figure 4B**). This antagonistic behavior, however, should not be seen in early childhood.

While the detailed ontogenetic development of CSS is presently beyond the scope of this paper, we nevertheless outline in short its development, based on Heinz Werner's (1978, p. 108–109) orthogenetic principle of development, that "wherever development occurs it proceeds from a state of relative globality and lack of differentiation to a state of increasing differentiation, articulation, and hierarchic integration." This orthogenetic principle has been shown to be consistent with the genetic organization of the cortex (Chen et al., 2012). Akin to Werner's (1978) notion of increasing differentiation and hierarchic integration, CSS is proposed to manifest with development as a successively more complex structure. In support, Anokhin et al. (2000) report that EEG dimensional complexity increases with age between 7 and 17. Moreover, the two trajectories in the CSS should, early on in development, be indistinguishable (Werner's "relative globality"), and the corresponding CSS space should comprise one global sphere (and not two). Support for this proposition was given by a recent fMRI study showing that it is only from around 2 years that the antagonistic behavior between the cortical networks is first observed (Gao et al., 2013).

There could also be states where both trajectories are under the threshold for access awareness, for example dreaming, and states where both trajectories enable access awareness, where one attends to the activity of the intrinsic system, without being immersed in it, as an observer such as in meditation. Neuroscientific studies of the neural space support this intuition: during dreaming, most of the DMN deactivates, as well as the extrinsic system (Nir and Tononi, 2010). Similarly, activity in the intrinsic system may persist in parallel to extrinsic stimulation if external stimulation is not sufficiently challenging (Greicius and Menon, 2004; Wilson et al., 2008), or when one attends to the activity of the intrinsic system (Christoff et al., 2009), as is the case during meditation (Travis and Shear, 2010). Next, we describe cases of alteration in typical CSS dynamics. All these states, we suggest, involve an alteration in the regular sense of NS, as is the case in early childhood (Oatley, 2007).

## **ALTERATIONS IN TYPICAL CSS DYNAMICS**

We suggest that alterations in typical CSS dynamics occur when the regular sense of NS is modified. While the typically antagonistic behavior of the trajectories essentially indicates differentiation, here we discuss those conditions wherein hierarchic integration is achieved (in Werner's terms). These states have been termed "no-self," "transpersonal," or "transcendent" states, and they can occur in a spontaneous or training-induced manner (Alexander and Langer, 1990; Pascual-Leone, 2000; Hartman and Zimberoff, 2008). Such states have been related to enhanced performance (Csikszentmihalyi, 1988, 1990; Leary et al., 2006) and heightened happiness (Dambrun and Ricard, 2011). We predict that these situations can be seen phenomenologically (first two predictions) as well as being translated into neural space (last two predictions), resulting in:


To support these propositions, we bring evidence from two distinct states, both described as involving alterations in the sense of self: optimal experience, also called flow (Csikszentmihalyi, 1988, 1990), and meditation (Hölzel et al., 2011; Fell, 2012; Vago and Silbersweig, 2012). These two states have been sometimes considered to be largely similar (e.g., Kristeller and Rikhye, 2008; Bermant et al., 2011). However, others consider flow and meditation to diverge in their phenomenology and ultimate aim, as flow fosters development through higher challenges and skill refinement while meditation mainly points toward self-transcendence (Delle Fave et al., 2011). Moreover, flow is largely spontaneous and transitory, whereas the meditative state is training-induced and can become an enduring condition. While we do not expect isomorphism between flow and meditation in the phenomenological and neural spaces, we bring both as examples of an alteration in the sense of self and increased hierarchic integration, demonstrating how we can put the CSS to work.

### *The state of flow*

Flow is a state in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment in the process of the activity. In essence, flow is characterized by complete absorption in what one does. Conceptualized by Csikszentmihalyi (1988, 1990), flow is an optimal experience of maximum enjoyment, and a good balance between the *perceived* challenges of the task and one's own *perceived* skills. Importantly, it is a state characterized by an altered sense of self. In fact, Csikszentmihalyi (1990, p. 85) describes "excessive self-consciousness" as being the major internal obstacle to experiencing flow, as it "lacks the attentional fluidity needed to relate to activities for their own sake; too much psychic energy is wrapped up in the self, and free attention is rigidly guided by its needs." According to Csikszentmihalyi (1988), flow is defined by eight characteristics:

(1) Flow occurs when we confront challenges where we have a chance of achievement;


Now, we consider the arguments that support a transition toward the CC/MS trajectory being more available to access awareness (prediction 1). First, at the center of the awareness continuum, at the sharp switch from non-awareness to awareness, a person is fully immersed, concentrated and completely absorbed in an activity. This resembles the third and fourth dimensions of flow. Moreover, flow is considered to occur at a subliminal level of awareness, making the experience difficult to distinguish by recollection (Csikszentmihalyi, 1988, 1990). Similarly, Dietrich (2004) positioned flow in between maximal implicit processing and minimal explicit processing. Dietrich (2003, 2004) also related the flow experience to low DLPFC activity, calling it a state of "hypofrontality." This is in accord with our suggestion of low Ni activity from the center toward the lower end of the awareness dimension.

Second, at the center of the time continuum one experiences fully the present moment. This resembles characteristics four and five, describing flow as being totally in the present moment, to the point that one experiences a distortion of time.

Third, at the center of the emotion continuum we expect to find emotional tranquility, due to a balance between negative and positive valence. This resembles characteristic number six, describing flow as being meaningful, and the general equation between flow and enjoyment. This might be a little counterintuitive at first, as flow could be supposed to be pleasurable, hence might be placed toward the pleasant side of the emotional continuum. However, there is a distinction between pleasure and enjoyment, as emphasized by Csikszentmihalyi: "any piece of work well done is enjoyable. None of these experiences may be particularly pleasurable at the time they are taking place. Experiences that give pleasure can also give enjoyment, but the two sensations are quite different" (Csikszentmihalyi, 1990, p. 46). A recent fMRI study (Ulrich et al., 2013) investigated the neural correlates of "flow" (challenging task difficulty was dynamically adjusted to participants' individual level of skill). Comparing "flow" to "boredom" and "overload" conditions (very low and very high task demands, respectively), decreased activity was reported for the amygdala during the flow condition. Furthermore, amygdala activity was negatively correlated with subjective rating of flow. This was interpreted as indicating reduced negative arousal during the flow state, which is in accord with the CSS prediction.

Fourth, being at the center of CSS means experiencing the agentic MS, and being further away from the self-conscious NS. This is in agreement with the last two dimensions of flow, and is supported by the study of Ulrich et al. (2013), showing significantly lower mPFC activity during the flow state.

Fifth, being at the center of CSS predicts a highly embodied state. Indeed, Csikszentmihalyi (1990) describes flow states as being intimately related to the body: "It is through the body that we are related to one another and to the rest of the world. While this connection itself might be quite obvious, what we tend to forget is how enjoyable it can be. Our physical apparatus has evolved so that whenever we use its sensing devices they produce a positive sensation, and the whole organism resonates in harmony" (p. 115–116).

Up until now, we have provided evidence that the state of flow, as an example of an altered experience of the self, supports proposition 1, namely a transition toward the CC/MS trajectory being more available to the access awareness. The third characteristic of flow—"There is a merging of action and awareness," in itself supports proposition 2 of an "integration" of the CC/MS and EC/NS trajectories, phenomenologically. As to neural space, the neuroscientific research of flow is scarce. However, the fourth proposition—namely less antagonism between the intrinsic and extrinsic networks—is given initial support by Ulrich et al. (2013), who reported three DMN regions (angular gyrus, supramarginal gyrus, and parahippocampus) to show U-shaped neural activity with increasing task difficulty, indicating lowest DMN activity during the "flow" condition, as opposed to "boredom" and "overload" conditions. Intriguingly, and counter-intuitively to the common "push-pull" antagonism notion, DMN activity is not minimal with "overload." Taken together, the state of flow supports three of the four proposed changes in CSS and its neural space.

### *Meditation—state and trait*

The word "meditation" is used to describe self-regulating practices that focus on training attention (Cahn and Polich, 2006). Meditation is expected to alter self-referential processing, as the major aim of practice is the realization, by direct experience, of the lack of any essential "self" (Dreyfus and Thompson, 2007). This has been supported by ample phenomenological studies (Austin, 2000; Leary et al., 2006; Dambrun and Ricard, 2011). Findings from meditation studies indicate training-induced neuroplasticity, both in function and in structure, (Cahn and Polich, 2006; Ivanovski and Malhi, 2007; Davidson and Lutz, 2008; Rubia, 2009).

One form of meditation that has been extensively studied is mindfulness meditation (MM), stemming from the Buddhist Theravada tradition, defined in a Western context as "the awareness that emerges through paying attention on purpose, in the present moment, and non-judgmentally to the unfolding of experience moment by moment" (Kabat-Zinn, 2003, p. 145). Looking closely at this definition, we see that it embeds training for all three CSS dimensions: awareness (awareness that emerges through paying attention on purpose), time (experience moment by moment) and emotion (non-judgmentally). We subsequently show that MM training induces a transition toward the CC/MS trajectory being more available to access awareness (prediction 1).

First, in relation to the time dimension, it was previously suggested that meditation induces a change in subjective temporal experience toward emphasizing the "now," or being less aware of the passage of time (Brown et al., 1984). Being at the center of the time continuum in an absorbed manner can be measured as longer and longer time production (indicative of a slower rate of functioning of the internal timer, demonstrating that time seems to be moving slower; Glicksohn, 2001). In agreement with that, longer time production was shown in MM practitioners compared to control participants (Berkovich-Ohana et al., 2011), and a slower internal timer was indicated in another study (Kramer et al., 2013). Second, subjective reports on the effects of meditation have included heightened perceptual awareness (Brown, 1977; Baruss, 2003; Carter et al., 2005). This is supported by physiological studies showing MM practice to increase bodily awareness (Farb et al., 2007; Kerr et al., 2013). Third, various meditative practices were shown to lower the intensity of emotional arousal (Aftanas and Golosheykin, 2005; Nielsen and Kaszniak, 2006), to result in trait reduction in anxiety and negative affect, and an increase in positive affect (Davidson et al., 2003), and to entail lower amygdala reactivity during focused attention meditation (Brefczynski-Lewis et al., 2007). MM was also shown to increase tolerance of negative affect (Chambers et al., 2009; Farb et al., 2010), possibly by restoring balance between affective and sensory neural networks—supporting conceptual and embodied representations of emotion (Farb et al., 2012). Together, these data support prediction 1 of a transition toward the CC/MS.

In addition, accumulating evidence supports prediction 3, namely higher Nms activity and lower Nns. For example, evidence shows that MM practice lowers the DMN (key network in Nns) activity (Farb et al., 2007; Pagnoni et al., 2008; Brewer et al., 2011; Berkovich-Ohana et al., 2012, 2013; Dor-Ziderman et al., 2013; summarized by Fell, 2012; Jerath et al., 2012). More specifically, various meditative techniques showed decreased activation in several areas of the DMN during practice, including the precuneus (Tang et al., 2009; Ives-Deliperi et al., 2011), mPFC (Farb et al., 2007; Brewer et al., 2011; Ives-Deliperi et al., 2011), PCC (Pagnoni et al., 2008; Tang et al., 2009; Brewer et al., 2011), ACC (Pagnoni et al., 2008; Ives-Deliperi et al., 2011;) and LTC (Pagnoni et al., 2008). A similar result was shown using EEG, where lower frontalmidline gamma power (Berkovich-Ohana et al., 2012), or lower gamma functional connectivity (Berkovich-Ohana et al., 2013) were indicative of lower trait DMN activity.

Finally, several recent fMRI studies support prediction 4, namely increased synchronization between the typically antagonistic networks. An fMRI study showed a stronger coupling between the intrinsic and extrinsic systems during non-dual meditation (Josipovic et al., 2011). Brewer et al. (2011) reported that the correlation between areas involved in cognitive control (dACC, DLPFC), which are part of Ni, and the PCC area in the Nns were higher for experienced meditators than controls, both at rest and during meditation. Additionally, it was shown that during MM practice, as compared to rest, functional connectivity is strengthened between the DAN (comprising Nms) and DMN (Nns) (Froeliger et al., 2012). To summarize this section, we have provided support from meditation research for three predictions.

## **A COMPARISON TO OTHER MODELS OF CONSCIOUSNESS**

This section posits CSS in the wider context of different approaches to consciousness. Obviously, other theories are mentioned here briefly, as their elaboration is beyond the scope of this paper. CSS provides a phenomenological map which includes all possible consciousness states. It suggests that all possible states of consciousness are a combination of three dimensions, and that each consciousness state involves a specific sense of selfhood. It also describes the system's dynamic behavior. Furthermore, it provides a tentative neural space. Such a description is totally missing in the literature. However, it builds on previous theories of consciousness, which describe some of these dimensions at a time, as outlined below.

Various theories of consciousness suggesting functional descriptions have emerged in the last decade (reviewed by Lau and Rosenthal, 2011; and by Seth et al., 2008). These include neurodynamical approaches to consciousness (Varela et al., 1991, 2001; Tononi and Edelman, 1998; Dehaene and Naccache, 2001; Dehaene and Changeux, 2005; Tononi, 2008). Despite their differences, these various models agree that the constitution of dynamic spatiotemporal patterns of neural activity, namely neuronal synchrony, plays a central role in the emergence of consciousness (reviewed by Cosmelli et al., 2007). Specifically, these theories explain what differentiates conscious experience from subliminal or un-conscious experience (Dehaene et al., 2006), why thalamocortical anatomy suits conscious experience as opposed to different neural architectures such as seen in the cerebellum, or afferent and efferent pathways (Tononi, 2008), and how the proposed gap between qualia and brain activity (Chalmers, 1995) can be reduced. CSS builds on Tononi's (2008) integrated information theory, accepting that the neural space is mainly attributed to thalamocortical loops, that consciousness arises from integrated informational relationships generated by a complex of elements in the neural space (Tononi's "main complex"), and that the larger the complex, the greater the information the system can generate. CSS sees consciousness as essentially embodied, as previously emphasized by others (Varela et al., 1991; Thompson and Varela, 2001; Cosmelli and Thompson, 2010). CSS also incorporates workspace theory (Dehaene et al., 2006), as elaborated in the section on awareness. However, the CSS model departs from these theories, by proposing to explain phenomenologically all possible states of consciousness, as well as suggesting a possible neural space.

Other phenomenological accounts describe states of consciousness along a continuum of experience of the self. Some theories emphasize the awareness dimension, including those presented by Schooler (2002), Brown (1976), and Natsoulas (1998), which are then integrated into a unifying social/personality model describing degrees of consciousness and selfhood (Morin, 2006). Other theories emphasized the importance of time (including Neisser, 1988; Stuss et al., 2001; Newen and Vogeley, 2003; Zelazo, 2004) or emotions (including Lambie and Marcel, 2002; Panksepp, 2005; Barrett et al., 2007; Tsuchiya and Adolphs, 2007) in relation to selfhood. In relation to these models, CSS extends the experience of selfhood to encompass both the time and emotion dimensions in one coherent framework.

Importantly, Damasio (1999, 2012) provides a neuroscientific account for consciousness and self, which has inspired our model, although we departed from it substantially, as subsequently elaborated. Damasio outlines and combines two theories (Dolan, 1999); the first concerns the propagation of consciousness and self from body along a continuum: from (1) an unconscious bodily self (proto-self), which deals with the state of the internal milieu and creates a first-order representation of current body states in the brain, to (2) the core-self (CS), which gives rise to coreconsciousness (CC). CC is a complex of second-order mental maps based in the feeling state, which arises when the proto-self interacts with the first-order sensory maps that represent objects; and finally, (3) Extended consciousness (EC), which depends on CC, deals with holding in mind, over time, a multiplicity of neural patterns that describe the autobiographical-self (AS). AS is heavily dependent on the formation of enduring experiential memories, attention and language, and its inevitable concomitant is personal identity. Damasio's second theory concerns affect, and has been referred to briefly before (section Third Dimension of the CSS— Emotion). To enable a careful comparison of CSS with Damasio's view, we refer the reader to **Table 1**.

Damasio's view is largely adopted in CSS concerning the CC and EC, and their respective type of selfhood. A major difference is positioning both types of consciousness and self on an awareness continuum, where both can be either conscious or nonconscious, as opposed to the gradual propagation of conscious experience described by Damasio. A second major departure from Damasio's view concerns emotions, his three levels of emotion being replaced by the two dimensions of valence and arousal. As a result of these differences, when the linear view of Damasio is replaced by the 3D view of the CSS, more flexibility is available to the system, and explanatory power increases. For example, Damasio's view could not explain unaware emotional mental evaluation (Winkielman and Berridge, 2004; Sato and Aoki, 2006), unaware mental representation of numbers (Greenwald et al., 2003; Ric and Muller, 2012) or unaware semantic priming (Dehaene et al., 1998; Naccache and Dehaene, 2001).

To conclude this section, the relationship of CSS to other models of consciousness has been elaborated, especially with regard to Damasio's view. This comparison was intended to highlight the integrative, unifying and explanatory power of the CSS model.

### **THE LIMITATIONS, PREDICTIONS AND CONTRIBUTION OF THE CSS MODEL**

The CSS model has several limitations, which warrant delineation. First, CSS describes a phenomenological space, and also attempts to suggest its neural space. The neural space described here should be regarded as a coarse attempt based on current understanding. Second, the resolution between states in phenomenal space is much higher than current resolution in neural space (as already discussed by Fell, 2004), rendering a full translation from the phenomenological to the neural space impossible at the moment. Third, the important issue of how core cognitive functions, such as emotion regulation and reward, are created by the interplay between the three dimensions was left out. Finally, the presentation of the CSS model in this article leaves some important issues untouched, including: (1) Individual differences in the size and shape of CSS; (2) Pathologies of consciousness and selfhood; (3) Elaboration on the neural reference space to include electrophysiological activity; and (4) The location of altered states of consciousness within the CSS, an issue strongly related to the "breakdown" of the time dimension. These issues would hopefully be the topic of further developments of the model.

We conclude by providing examples of several predictions of the model which relate to current debates in the literature, and which are scientifically testable. First, given that an experience can fall into any coordinate in CSS, the model predicts some experiences which are still being debated. For example, albeit the controversy about the very existence of unconscious emotions (e.g., Clore, 1994; Winkielman and Berridge, 2004), the model predicts that full-blown emotions (including hedonic


**Table 1 | A comparison between Damasio's (1999) theory of consciousness and the CSS model, describing points of departure.**

CC, core consciousness; EC, extended consciousness; CS, core self; AS, autobiographic self.

feeling and appraisal) could be experienced without access awareness. Second, based on the dual phenomenological composition of each of the three dimensions, CSS predicts that their interaction in experience (access awareness) takes place only in the same sphere. For example, during emotion evaluation (outer sphere), one cannot simultaneously process external output (inner sphere). Or if one experiences core emotions/arousal (inner sphere), one cannot experience simultaneously mental time traveling (outer sphere). Obviously, this bears consequences for the neural spaces' interaction, requiring antagonism between the corresponding structures (as described in this paper), which could be readily addressed empirically by blood oxygenated level dependent (BOLD) fMRI studies. Third, CSS predicts that any well-reasoned condition in which one would expect hierarchic integration and alteration in the regular sense of NS, such as flow and meditation, will exhibit the four predictions laid out in section Typical CSS Dynamics, including that the two neural spaces should be positively correlated. Fourth, a prediction concerning complexity can be derived from CSS development (section The Dynamics within the CSS). As CSS manifests with development as a successively more complex structure, it predicts not only increasing complexity with age, but decreasing complexity in conditions of hierarchic integration, such as flow and meditation (for a similar view, see Sharp, 2011). For example, children should have a CSS which is more "global" in a Wernerian sense (hence, less complex), and meditators should have a CSS which is more hierarchically integrated (hence, also less complex). Partial support for this prediction was given by a finding of lower dimensional complexity during meditation, compared to rest (Aftanas and Golocheikine, 2002).

The model presented here creates a broad theoretical framework with explanatory and unificatory power, that attempts to make sense of a wide range of otherwise unrelated phenomenological and neuroscientific observations. Importantly, the model provides a new framework for understanding the relationship between core aspects of consciousness, hence lays a theoretical basis for the study of consciousness. We hope this model will inform future studies, and raise further testable predictions.

## **AUTHOR CONTRIBUTIONS**

Aviva Berkovich-Ohana conceptualized the model and wrote the paper. Joseph Glicksohn contributed to the conception of the work and revised all its versions critically.

### **ACKNOWLEDGMENTS**

The model presented here was inspired by a series of lectures given by Partizio Paoletti (http://www.fondazionepatriziopaoletti.org/ areediintevento/25/research.html), which the first author attended in 2000–2005 in Israel and Italy. We thank Tal Balaban Dotan Ben-Soussan, Yochai Ataria, Yair Dor-Ziderman and Zoran Josipovic, for reading and helping improve previous versions of this paper. This work was supported by the Mind and Life Institute, Francisco J. Varela Research Award 2012-Varela-Berkovich, by a grant from the Bial Foundation (27/10), and by a European Cooperation in Science and Technology (COST) action on Time in Mental Activity (TIMELY; TD0904).

## **REFERENCES**


L. Filipovic and K. M. Jaszczolt (Amsterdam: John Benjamins Publishing Company), 283–304.


*Proc. Natl. Acad. Sci. U.S.A.* 106, 8865–8871. doi: 10.1073/pnas.09040 31106


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 20 November 2013; accepted: 01 April 2014; published online: 29 April 2014. Citation: Berkovich-Ohana A and Glicksohn J (2014) The consciousness state space (CSS)—a unifying model for consciousness and self. Front. Psychol. 5:341. doi: 10.3389/fpsyg.2014.00341*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Berkovich-Ohana and Glicksohn. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Present moment, past, and future: mental kaleidoscope

## *Andrew A. Fingelkurts\* and Alexander A. Fingelkurts*

BM-Science – Brain and Mind Technologies Research Centre, Espoo, Finland \*Correspondence: andrew.fingelkurts@bm-science.com

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

*Reviewed by:*

Valdas Noreika, Medical Research Council, UK

**Keywords: subjective present moment, operation, subjective time flow, brain operational architectonics, EEG**

It is the every person's daily phenomenal experience that conscious states represent their contents as occurring *now*. Following Droege (2009) we could state that consciousness has a peculiar affinity for *presence*. Some researchers even argue that conscious awareness necessarily demands that mental content is somehow held "frozen" within a discrete *progressive present moment* (James, 1890; Lynds, 2003). Thus, phenomenal content seems to be minimally conscious if it is integrated into a single and coherent model of reality during a *"virtual window" of presence* (Metzinger, 2003; see also Brown, 1998; Varela, 1999; Smythies, 2003).

In order to explain such features of consciousness as phenomenal unity and continuity within the *current present* along with a succession of discrete thoughts that give rise to feeling of the past and future, a reference to mechanisms outside the phenomenal realm is necessary (Revonsuo, 2003). Thus, the question of what could be the neurophysiological mechanisms responsible for these experiences should be addressed.

In this Opinion Article we shall build our argument based on the biological realism approach to consciousness proposed by Revonsuo (2006). According to this approach, subjective consciousness is a real phenomenon that is tightly anchored to a biological reality within the human brain. Broadly speaking, the human brain is the specific physical "location," where the subjective mental reality and the objective neurobiological reality are intimately connected along a unified metastable continuum (Fingelkurts et al., 2009, 2013).

We have argued previously (Fingelkurts et al., 2010) that phenomenal consciousness refers to a higher level of organization in the brain and captures all *immediate* and undeniable (from the first-person perspective) phenomena of subjective experiences (hearing, seeing, touching, feeling, embodiment, moving, and thinking) that present to any person *right now* (subjective present) and *right here* (subjective space). By this definition even remembering the past images and planning the future events can't be performed other than in the *present moment* and in relation to current state of affairs (see also Lynds, 2003; Droege, 2009). This is so because someone possesses phenomenal consciousness if there is any type of subjective experiences that is *currently present* for him/her (Fingelkurts et al., 2010).

In this context what is presented as *now* is not simply whatever sensory or other representations occur in the brain at any given moment but rather the spatialtemporal hierarchy of selected and nested metastable states of neuronal assemblies that serve in real time as a basis for the subjective experiences of the "present moment." Among many theories, the Operational Architectonics (OA) theory of brain and mind functioning (Fingelkurts and Fingelkurts, 2001, 2008; Fingelkurts et al., 2010, 2013) explicitly utilizes the hierarchy of nested metastable states of neuronal assemblies. In short, OA theory is centered on the notion of *operation*. Operation is broadly defined as the process or state of being in effect and it has a beginning and an end (Collins Essential English Dictionary, 2006). In fact, everything which can be represented by a process is an operation. The notion of operation plays a central role in bridging the brain-mind gap and makes it possible to identify what at the same time belongs to the mental level and to the neurophysiological level of brain activity organization, and acts as a mediator between the two (Fingelkurts and Fingelkurts, 2001, 2008; Benedetti et al., 2010). Understanding of the operation as a process and considering its combinatorial nature, seems especially well suited for describing and studying the mechanisms of how information about the objective physical entities of the external world can be integrated within the *present moment* in the internal subjective domain by means of entities of distributed neuronal assemblies (Fingelkurts et al., 2010, 2013). In line with this conceptualization, simple cognitive operations that present some partial aspect of the whole object/scene/concept are presented in the brain by local 3D-fields produced by discrete and transient neuronal assemblies, which can be recorded by an electroencephalogram (EEG) (**Figures 1A,B**). More complex operations that constitute the whole object or scene are brought into existence by joint (synchronized) simple operations in the form of coupled 3D-fields—so called operational modules (OMs) of varied complexity (Fingelkurts and Fingelkurts, in press). Further synchronization of several OMs (complex field spatial-temporal patterns; **Figure 1A**) forms even more coarse scales of nested functional hierarchy (Feinberg, 2000) that is now able to present and hold highly complex sensorial inputs as coherent perceptions of the world, create internal complex images and form conscious decisions (Fingelkurts et al., 2010, 2013). The recombination of neuronal assemblies and their operational modules into new configurations gives rise to a nearly inexhaustible source of presenting different qualities, patterns, objects, scenes, concepts and decisions.

In the following we will discuss how the OA framework could implement the subjective present and some other temporal phenomena. We argue that at the phenomenological level, the lasting OM would be experienced as the "phenomenal present" of consciousness (**Figure 1C**). This hypothesis remains to be proven experimentally, however some empirical evidence already exists. For example, the mean duration of OMs (derived from an EEG with a frequency band of 0.3– 30 Hz) usually varies from 80–100 ms for large OMs spanning the cortex to 30 s for small local OMs. These accounts, including duration variation, are consistent with known estimates for the *frame of a specious present*, which varies from ∼100 ms to several seconds depending on circumstances (Pöppel, 1988).

However, if the brain could implement only a complex but static OM, then such a brain would only experience the presence of one unified world frozen into an internal *now* (Metzinger, 2003). Neither the complex texture of subjective time flow, nor true perspectivalness that goes along with a first-person point of view would exist in such situation (Fingelkurts et al., 2010). Therefore, a dynamic succession of phenomenal moments that are integrated into the flow of subjective time is needed. Indeed, as it is evident from the first-person perspective, the actualization of full-fledged phenomenal objects, images or scenes is realized on a "one-at-a-time" basis, moving serially from one phenomenal pattern within a specious present to another (Revonsuo, 2006). This process gives rise to a stream of consciousness that is best conceptualized in the James' metaphor of a *kaleidoscope* (James, 1890). Using this metaphor James illustrates the ever-changing stream of thoughts like a rotating kaleidoscope where each *momentarily* stable pattern constructed from multiple pieces (local fields in our interpretation) is a speciously presented thought (OM in our interpretation). Thus, the succession of phenomenal images or thoughts is neurophysiologically presented by the succession of discrete and relatively stable OMs, which are separated by rapid transitive processes (RTP), i.e., abrupt changes of OMs (**Figure 1C**). As it has been shown experimentally, at the critical point of transition in mental states, the OM undergoes a profound reconfiguration which is expressed through the following process (Fingelkurts et al., 2000, 2003; Fingelkurts and Fingelkurts, in press): The OM, which is comprised from a set of local bioelectrical fields produced by transient neuronal assemblies across several brain areas, rapidly loses functional couplings and establishes new couplings within another set of local bioelectrical fields, thus demarcating a new OM in the volumetric operational spacetime continuum of the brain (**Figure 1C**).

Thus, the presented model for OM mediated succession of phenomenal images or thoughts is one way of understanding how *subjective time flow* is mentally (re)constructed beyond the phenomenal horizons of "presence." Subjective time flow is not actually experienced or "perceived," rather it emerges as the product of cognitive higher-order processes operating on the OMs (Fingelkurts et al., 2010). Such higher-order processes are also expressed in the form of complex OMs, that not only superceedes lower-level OMs, but also execute memory consolidation and retrieval operations (Fingelkurts et al., 2003). Given such a mechanism, the variation in *subjectively experienced speed of time* could be also explained. When the OMs' average duration decreases, there are many more OMs managing to sequence each other within a given time unit. We suggest that this overflow of OMs would be commonly experienced as an acceleration of the subjective time. Conversely, if the average duration of OMs was to increase, then the subjective experience of time would slow down. Below, we review some experimental evidence in support of our theorizing.

It is well known that certain psychoactive agents create subjective time distortions when administered. For example, opioids can be used to prolong the subjectively perceived duration of thought (Galski et al., 2000). In agreement with OA framework, it has been shown that opioids do indeed increase the duration of the lifespan of neuronal assemblies (indexed by EEG quasi-stationary segments) and limit the synchronization between their operations, thus reducing the possible number of OMs while increasing their life-span (Fingelkurts et al., 2006).

Another important model, where subjective experience could be easily manipulated is hypnosis. In a neutral hypnotic state the subject experiences an altered background state of consciousness different from the normal baseline state of consciousness without the need of suggestion (Kallio and Revonsuo, 2003). This subjective state is characterized by "emptiness" or "absorption" brought about by dissociations in the cognitive modules that are temporarily incapable of normal communication with each other (Gruzelier, 2000). Additionally, it has been shown that the subjective sensation for the passage of time is stretched during hypnosis, because internal events are subjectively slowed (Von Kirchenheim and Persinger, 1991; Naish, 2001). Adhering to the tenets of OA framework, these subjective experiences should be reflected in the operational architectonics of the electromagnetic brain field. It was indeed shown that the functional life-span of neuronal assemblies (indexed by the EEG quasi-stationary segments) was significantly *longer* during hypnosis when compared with the normal/baseline conscious condition (Fingelkurts et al., 2007). It was further found that the number and strength of synchronized operations among different neuronal assemblies were significantly *lower* during hypnosis than during the baseline, thus limiting the possibility for any OMs to emerge. As a result they were absent (Fingelkurts et al., 2007). Since OMs represent the formation of integrated conscious experiences, their absence may explain such unusual subjective experiences during hypnosis as amnesia, timelessness, detachment from the self, a "willingness" to accept distortions of logic or reality, and the lack of initiative or willful movement (Dietrich, 2003).

Dreaming is a special case where the phenomenal world is realized in the brain in its "pure form," because it is nearly completely isolated from the external physical world and the rest of the body. Dreams can appear in REM as well as in the nonREM sleep (Nir and Tononi, 2010). However, the nature of dreams in REM and non-REM sleep is different: during REM the dreams are complex, organized, *temporally evolving*, multimodal, and often bizarre (Hobson et al., 2000), while in nonREM the dreams are characterized by simple, *static* or isolated image(s) or though(s), usually of one modality (Noreika et al., 2009). The OA prediction is that nonREM dreams should be accompanied by shortlived small neuronal assemblies and longlived large neuronal assemblies, and by the significant increase of operational synchrony (poor set of OMs) among different neuronal assemblies in order to subjectively present static images or thoughts. In a pilot nonREM sleep study (Fingelkurts and Fingelkurts, in press) we found that nonREM dreams were indeed accompanied by the small short-lived and large long-lived neuronal assemblies, as well as significant operational synchrony increase in the OA organization of the brain. Future research should establish the OA data for REM sleep dreams.

This brief review of results supports the suggested neurophysiological mechanism (within the operational architectonics of the human brain field) responsible for the experiences of the "present moment," past, and future.

### **ACKNOWLEDGMENTS**

The authors would like to thank Dmitry Skarin for English editing.

### **REFERENCES**


M. (2003). Structural (operational) synchrony of EEG alpha activity during an auditory memory task. *Neuroimage* 20, 529–542. doi: 10.1016/S1053- 8119(03)00305-7


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 18 March 2014; accepted: 15 April 2014; published online: 01 May 2014.*

*Citation: Fingelkurts AA and Fingelkurts AA (2014) Present moment, past, and future: mental kaleidoscope. Front. Psychol. 5:395. doi: 10.3389/fpsyg.2014.00395*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Fingelkurts and Fingelkurts. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Temporal structure of consciousness and minimal self in schizophrenia

#### **Brice Martin<sup>1</sup> , Marc Wittmann<sup>2</sup> , Nicolas Franck<sup>1</sup> , Michel Cermolacce3,4,5 , Fabrice Berna<sup>6</sup> and Anne Giersch<sup>6</sup>\***

<sup>1</sup> Centre Référent Lyonnais en Réhabilitation et en Remédiation Cognitive - Service Universitaire de Réhabilitation, Hôpital du Vinatier, Université Lyon 1 and UMR 5229 (Centre National de la Recherche Scientifique), Lyon, France

<sup>3</sup> Département Universitaire de Psychiatrie, Centre Hospitalier Universitaire Sainte Marguerite and Aix-Marseille Université, Marseille, France

<sup>4</sup> Unité de Neurophysiologie, Psychophysiologie et Neurophénoménologie, UF 4817, Centre Hospitalier Universitaire Sainte Marguerite, Marseille, France

<sup>5</sup> Laboratoire de Neurosciences Cognitives, UMR CNRS 7291 and Aix-Marseille Université, Fédération 3C, Marseille, France

6 INSERM U1114, Department of Psychiatry, Fédération de Médecine Translationnelle de Strasbourg, University Hospital of Strasbourg, University of Strasbourg, Strasbourg, France

### **Edited by:**

Simon Grondin, Université Laval, Canada

### **Reviewed by:**

Mark Elliott, National University of Ireland Galway, Ireland Giovanna Mioni, Université Laval, Canada

### **\*Correspondence:**

Anne Giersch, INSERM U1114, Department of Psychiatry, Fédération de Médecine Translationnelle de Strasbourg, University Hospital of Strasbourg, University of Strasbourg, 1 place de l'Hôpital, 67000 Strasbourg, France e-mail: giersch@unistra.fr

The concept of the minimal self refers to the consciousness of oneself as an immediate subject of experience. According to recent studies, disturbances of the minimal self may be a core feature of schizophrenia. They are emphasized in classical psychiatry literature and in phenomenological work. Impaired minimal self-experience may be defined as a distortion of one's first-person experiential perspective as, for example, an "altered presence" during which the sense of the experienced self ("mineness") is subtly affected, or "altered sense of demarcation," i.e., a difficulty discriminating the self from the non-self. Little is known, however, about the cognitive basis of these disturbances. In fact, recent work indicates that disorders of the self are not correlated with cognitive impairments commonly found in schizophrenia such as working-memory and attention disorders. In addition, a major difficulty with exploring the minimal self experimentally lies in its definition as being non-self-reflexive, and distinct from the verbalized, explicit awareness of an "I." In this paper, we shall discuss the possibility that disturbances of the minimal self observed in patients with schizophrenia are related to alterations in time processing. We shall review the literature on schizophrenia and time processing that lends support to this possibility. In particular we shall discuss the involvement of temporal integration windows on different time scales (implicit time processing) as well as duration perception disturbances (explicit time processing) in disorders of the minimal self. We argue that a better understanding of the relationship between time and the minimal self as well of issues of embodiment require research that looks more specifically at implicit time processing. Some methodological issues will be discussed.

**Keywords: schizophrenia, time perception, self-concept, consciousness, psychology of the self, psychopathology, experimental psychology**

### **DISTURBANCES OF MINIMAL SELF AND SCHIZOPHRENIA**

There is a consensus that the self is disordered in patients with schizophrenia (for review, Lysaker and Lysaker, 2010; Mishara et al., 2014). Several studies suggest these disorders include a disturbance of the most elementary component of self, i.e., minimal self-disorders (SDs; Gallese and Ferri, 2013; Nelson et al., 2013; Hur et al., 2014). However, characterizing these disorders and understanding the mechanisms involved remain a challenge. In the present work, we bring together ideas and concepts from two different domains, phenomenology and experimental psychology. Combining these two fields is not straightforward, but in the present case such an integrative approach helps us to see the recent phenomenology and timing literature in a new perspective and to discuss the possible impact of timing characteristics on the self. Previously (e.g., Mishara, 2007) it has been proposed there may be a relationship between timing disorders and SDs, but recent data in this field may help to shed new light on this possible link.

### **THE MINIMAL SELF**

From the point of view of phenomenology, the minimal self is the most basic level of the self. Gallagher (2000, p. 15) defines the minimal self as "a consciousness of oneself as an immediate subject of experience" and as "the pre-reflexive point of origin for action, experience and thought."

Thus, the minimal self refers to the tacit and pre-reflexive selfhood (Nelson et al., 2013). As explained by Stanghellini (2009) "I experience myself as the perspectival origin of my experiences (i.e., perceptions or emotions), actions and thoughts." Consequently, the minimal self can be seen separately from more elaborate aspects of the self (Gallagher, 2000; Parnas and Handest,

<sup>2</sup> Institute for Frontier Areas of Psychology and Mental Health, Department of Empirical and Analytical Psychophysics, Freiburg, Germany

2003; Nelson et al., 2013) such as the reflexive self (e.g., the explicit awareness of an "I") and the narrative self (e.g., experiencing the self as having special characteristics, like a personality, and a personal history that we tell about ourselves; Haug et al., 2012).

It is especially important to distinguish between the reflexive and pre-reflexive self. We base our approach on earlier phenomenological work (de Warren, 2009). Brentano and Husserl in particular distinguish between the object of a perception and the conscious perception itself. de Warren provides the following example as inspired by Brentano and Husserl: "when looking at this tree in my backyard, my consciousness is directed toward the tree and not toward my own act of perception. I am, however, aware of myself as perceiving this tree, yet this self-awareness (or self-consciousness) is not itself thematic", e.g. not reflexive. (de Warren, 2009, p. 19). According to Brentano, when an object of the outer world is present in our mind we simultaneously apprehend the mental phenomenon itself, but this does not lead to explicit reflection: "every act of consciousness is also implicitly conscious of itself, but only to the extent that it is a consciousness of something other than itself " (de Warren, 2009, p. 77). Put another way, when we say "I perceive a tree," we are usually focusing on the object of perception, i.e., the tree. The "I," although not unconscious, is only present implicitly. This selfawareness does not compete with the representation of the tree as long as it remains implicit. It is part of the act of consciousness. As we have already seen above, this pre-reflexive "mineness" of conscious experience is a central characteristic of the minimal self.

In contrast, a reflexive act of consciousness is described as reflexive when intentionality is directed toward the act of consciousness itself. As regards de Warren's example of the tree, "I can, through a further act of reflexion, make my perceptual act into theme, or object, of my consciousness, in which case I am no longer immersed in my directedness towards the tree, but redirected toward myself as perceiving the tree" (de Warren, 2009, p. 20). In this case, self-consciousness becomes reflexive, i.e., an explicit theme through an act of self-observation, which is distinct from a pre-reflexive perspective. Contrary to the minimal self, the explicit representation of self competes with other representations. We either perceive the tree or think about ourselves.

Although these distinctions have important limitations (see Zahavi, 1999), they will be sufficient for the minimal SD issues addressed in this paper. In particular, philosophers have struggled with the Brentano's proposal according to which the act of consciousness may entail different objects simultaneously: "the presentation of the sound and the presentation of the presentation of the sound (i.e., hearing) form a single mental phenomenon; it is only by considering it in its relation to two different objects, one of which is a physical phenomenon and the other a mental phenomenon, that we divide it conceptually into two presentations" (de Warren, 2009, pp. 78–79). Here, we interpret this proposal like de Warren by considering self-awareness to be present during an act of perception without being the focus of attention, i.e., in an implicit way. It is clear that the concept of self as we discuss it here is not the purest and most abstract form, which would be independent of time and would precede our interaction with the outer world (see Mishara, 2007; Mishara et al., 2014 for a critic of the latter approach), but rather an embodied self. It is during our experience of the world that we implicitly experience ourselves. Consequently, if our experience of the world is distorted, then our implicit experience of ourselves is bound to be disturbed as well. The following clinical examples help further define what we mean by minimal SDs.

## **DISTURBANCES OF THE MINIMAL SELF AS A TRAIT MARKER OF SCHIZOPHRENIA**

Evidence from empirical research and clinical analysis suggest that a disturbance of the minimal self (minimal-SDs) can be considered a core feature of schizophrenia. Minimal SDs have been described in prodromal, early, and more chronic stages of schizophrenia (Møller and Husby, 2000; Parnas et al., 2003, 2005; Raballo et al., 2011). They are described through a broad range of experiences, as listed in detail in the Exploration of Anomalous Self Experience (EASE), a phenomenology-oriented instrument centered on the exploration of minimal SDs (Parnas et al., 2005). This EASE sets out to list recurrent experiences of a distortion of one's first-person experiential perspective. The following examples (translated from French by Brice Martin) illustrate patients' responses (Parnas et al., 2005):

*Alterations of the 'stream of consciousness'* can be understood as there being a gap between one's own thoughts and the self, leading to the loss of "mineness" of mental experience. One example of this is "thought interference," like, for example, in a patient's own words *"when I'm thinking, you see, sometimes, it's a bit like*. . .*there are some words*. . .*some words or ideas which come into my mind*. . .*in a disconnected fashion*. . .*, which have nothing to do with what I'm thinking,*. . .*which can be banal*. . .*, which interfere with what I'm thinking*. . .*."*

*The Alteration of the Presence* corresponds to a broad range of phenomena that can be defined as a lack of immersion in the world. An example of the alteration of the presence is "a loss of natural evidence" (Blankenburg, 1971). It denotes a "lack of automatic, pre-reflective grasp of the meaning of everyday events, situations, people and objects" (Parnas et al., 2005) as, in the words of a patient, *"you see, for me, it's a bit like*. . .*as if nothing was obvious for me*. . .*The world is something*. . .*very complicated for me*. . .*It's tiring because I'm always thinking*. . . *I'm constantly wondering how people are going so easily through life, through things*. . . *Everything is a question for me*. . .*you see, I think about everything, and I can't help it*. . .*my mother always tells me I spend too much time wondering about too many things*. . .*for example*. . . *yesterday*. . . *I sat down in front of a wall, and, for one or two hours, asked myself how it had been built? Sometimes, I wonder why "paper" is called "paper"?".*

*'Corporeal experiences' or 'disembodiment'* denotes the feeling of being detached from oneself and one's actions, as if in a thirdperson perspective or without any perspective at all. One example of this is the "spatialization of bodily experiences" where the body is experienced as an object, with a weakened self experience, as in the following narrative: *"mmh*. . .*you see, it's like my body*. . .*it's like I can perceive inside my body, like things being a bit disconnected from my body*. . .*I frequently feel my heart beating*. . .*or the blow flowing in my veins*. . .*I can feel it, it's as if I can see it*. . .*or my muscles when I'm moving my arm*. . .*it's a bit like my body was constantly*. . .*was constantly present you see*. . .*as a thing in front of* *me*. . .*as if I wasn't really inside my body*. . .*."* A loss of self thereafter means a disconnectedness that can be described as detachment from one's physical body (de Haan and Fuchs, 2010).

The *altered sense of demarcation* can be understood as a difficulty discriminating self from not self, and, consequently, as a "loss or permeability of self-world boundary" (Parnas et al., 2005). One illustration of this can be seen in the following patient's explanation: "*you see, I'm like a house with a door that is constantly open*. . .*I'm living the fact that*. . .*as if people could come in*. . .*as if people could know what I'm thinking*. . . *As if*. . . *I've no barrier*. . .*as if there was no barrier between me and others*. . ."

All the above disturbances can be described as a loss of the center of gravity of experience (the "zero point").

The consideration of minimal SDs in psychiatry may have considerable applications, especially in differential diagnosis, inasmuch as SDs differ in schizophrenia and in bipolar disorders (Parnas et al., 2003). The evaluation of these minimal SDs may also be relevant for the *outcomes of schizophrenia*, such as suicidal behaviors, which appear to be correlated with the intensity of minimal SDs (Skodlar and Parnas, 2010). To date, however, detection of these disturbances relies on the verbal reports given by the patients. Such reports should be interpreted with caution, especially since the minimal self is related to non-verbal aspects of consciousness. The difficulty is to find a way of objectifying the minimal self with a non-verbal approach, while ensuring that this approach is relevant to the minimal self (Mishara, 2007; Mishara et al., 2014). Below, we discuss the possible role of time processing, which may be one of the key mechanisms underlying certain properties of the minimal self (Kiverstein, 2009). We sum up existing arguments in favor of elementary timing disorders in patients with schizophrenia and argue that such perceptual timing disorders should impact the sense of self, inasmuch as temporal aspects are involved in any states of feeling, whether regarding the self or the outer world.

## **COGNITIVE BASIS OF MINIMAL SELF DISTURBANCES IN SCHIZOPHRENIA: TOWARD THE QUESTION OF TIME?**

Little is known about the basic processes and cognitive correlates of minimal SDs. The question we ask here is whether known cognitive disturbances in patients with schizophrenia lead to minimal SDs. We present a synthetic review of works targeting this issue.

### **CLASSICAL NEUROCOGNITIVE DISTURBANCES IN SCHIZOPHRENIA AND SDs**

Neurocognitive impairments appear to be a core feature of schizophrenia, having a major impact on everyday functioning (Green, 1996; Green et al., 2000). The most common disturbances concern attention, memory, and executive processing (Heinrichs and Zakzanis, 1998).

These disturbances are frequently considered to be an important source of difficulties for patients in everyday life. For example, they are commonly considered to be the most important factor underlying the functional disabilities observed in schizophrenia, impinging upon patients' daily lives (Velligan et al., 2006) and employment. They constitute the target for cognitive remediation, a promising therapeutic tool (Demily and Franck, 2008).

Despite their frequency and importance in terms of disabilities, the link between these cognitive disturbances and minimal SDs in schizophrenia is unclear. Very few studies have attempted to explore these relationships. Haug (Haug et al., 2012) explored cognitive functioning (psychomotor speed, working memory, executive and memory functioning) and correlated cognitive impairments in schizophrenia with minimal SDs as assessed using EASE. However, they found few correlations between cognitive functioning and SDs.

We argue that cognitive functions as explored in the usual neuropsychological batteries may not be the most suitable approach. Below, we present the results of studies that are potentially closer to the question of the minimal self. There have been many studies that have explored the concept of agency. Agency is the feeling of being the agent of an action and has mostly been explored by explicitly asking subjects whether or not they are at the origin of a given action. As such, agency is reflexive rather than pre-reflexive. However, the mechanisms hypothetized as being involved in this emerging feeling are largely unconscious and pre-reflexive and could affect the minimal self. Moreover, some observations raise the question of timing. For these reasons, we start by presenting a summary of the studies exploring agency.

## **INTERNAL MODELS AND MINIMAL SDs**

The last few decades have seen the emergence of a set of cognitive hypotheses targeting some of the clinical features of schizophrenia such as delusions of alien control (first rank symptoms according to Schneider, 1995) and the delusional feeling of being controlled or influenced by other agents, both common manifestations in schizophrenia. Although these manifestations appear to reflect disorders of the reflexive self, they may still be connected with minimal SDs. Indeed, there are a number of arguments that have led us to develop the question of agency despite its being selfreflexive. In fact, impaired agency is observed mainly in acute phases of schizophrenia and only affects patients permanently in exceptional cases, although some disorders might persist in the interval between acute phases. It has been proposed that impaired agency emerges as a result of a combination of impairments, some of which would not be explicit and might be related to minimal SDs (Frith, 2005). Such impairments would lead to a loss of control, a frequent feeling which can arise, for example, when an action has not been performed in an optimal fashion (Pacherie, 2008). While not resulting in a loss of agency, it may weaken our implicit sense of being at the origin of our action. Insofar as it is not reflexive, such a loss of control might thus be connected with minimal SDs. It is additional impairments, such as abnormal interpretations of causal relationships, which would lead to explicit agency impairments and delusions of control. To make our arguments more concrete, we detail below the mechanisms that have been hypothesized as being involved in the loss of control. This in turn raises time issues.

Empirical work has been conducted in connection with "forward models," including mechanisms not related to conscious awareness. According to this approach, the sense of agency among healthy subjects is based on mechanisms that allow us to prepare an action and to translate intentions into actions (Frith, 2005). Based on Wolpert's (1997) model, the intention is translated into a motor program by means of an "inverse" model which allows the motor system to adjust the motor program to the intention even before the movement is initiated. Desmurget and Sirigu (2012) proposed that these first steps are associated with both a "wanting to move" and an "urge to move." Such feelings may be conscious, but they are not self-reflexive. Like the example given above of perceiving a tree, when we want to move, we focus on the action, and not on the "I" performing the action. It is worth emphasizing that we do not discuss the issue of will in this paper. "Wanting" to move suggests there is a conscious decision prior to moving. However, when we take a conscious decision, we usually focus our attention on the decision itself, not on the "I" taking the decision. Once again, the "I" will only be present implicitly. Yet, insofar as this "I" is defined in relation to the action, impairments in the ability to convert the first steps of action planning into real action might impair the implicit sense of "I," i.e., the minimal self. In fact, some studies have suggested that some aspects of planning are impaired in schizophrenia in the case of simple action sequences (Zalla et al., 2006; Delevoye-Turrell et al., 2007). For example, it has been shown that patients with schizophrenia have planning difficulties when tapping a surface with their finger. Tapping involves lowering the finger and then lifting it. It has been shown that healthy subjects can plan to lift their finger before the action onset and do not need to wait for sensory feedback. Patients, however, appear to wait for sensory information regarding surface contact before lifting their finger. Similarly, they have repeatedly been shown to be impaired every time separate parts of a movement should succeed each other smoothly (Delevoye-Turrell et al., 2003, 2007). Inasmuch as such impairments may weaken the ability to convert an intention into a controlled action efficiently, it might impact the implicit and pre-reflexive feeling of control associated with motor actions. It is noteworthy that all these impairments are observed when several components of an action must follow one another, and that they are reflected in abnormally long intervals between separate motor elements making up the action.

These findings emphasize the timing component of action in patients (Delevoye-Turrell et al., 2012; Turgeon et al., 2012). This is also the case with another component of motor action, the "efference copy." Once a motor program is adjusted, it generates an "efference copy," which is used to predict the sensory outcome of the action by means of the forward model (von Holst and Mittelstaedt, 1950). The action is then adjusted by comparing the expected outcome with the actual sensory feedback occurring as a result of the action. When the expected and actual outcome of the action match, the correspondence of these signals reinforces the sense of having initiated the action (Frith, 2005). It has been proposed that the efference copy is disturbed in patients with schizophrenia (Kelso, 1977; Franck et al., 2001; Jeannerod, 2009; Voss et al., 2010; Synofzik et al., 2013). As a result of this disturbance, patients would not benefit from the match between the predicted and actual outcome of the action. Again, the consequence would be a weakening of the sense of having initiated the action. As already emphasized, this does not lead directly to the delusional belief of being controlled by an external agent, which, in order to develop, would require additional impairments (Frith, 2005). Inasmuch as the weakening of the sense of initiating the action occurs implicitly, without the subject being able to report it explicitly, it might be related to minimal SDs. Interestingly, the mechanisms associated with producing an efference copy might also involve a timing component. The efference copy involves a temporal dimension, inasmuch as it is used to predict sensory feedback, and hence the timing of such feedback. The temporal dimension of the forward model had been underlined by Wolpert himself, and more recently by a series of other authors (e.g., Waters and Jablensky, 2009). Interestingly, the temporal distortion of the sensory feedback appears to impact all patients with schizophrenia and to affect their sense of agency independently of the association with delusions of control (Franck et al., 2001). Time processing disorders might be more stable markers than those ultimately resulting in delusions of control. In other words, timing disorders might be trait markers that persist in chronic phases of the pathology. This is consistent with the idea that such disorders weaken patients, possibly by inducing minimal SDs, but produce agency disturbances only secondarily. Patients may interpret their basic disorders in different ways, which explains why agency disturbances can take many different forms, with patients attributing their action to various agents (e.g., God, extra-terrestrial beings, television). However, these different interpretations would stem from a similar basic impairment. In that sense, the agency studies shed light on minimal SDs in schizophrenia (see Gallese and Ferri, 2013).

Yet, this approach may not provide the full explanation for minimal SDs in schizophrenia. First of all, forward models account for only part of minimal SDs because they target mainly motor action. Consequently, they give preference to exploring the sense of agency based on body perception and do not easily address other clinical manifestations, such as the alteration of presence.

Moreover, some models of the minimal self (Gallagher, 2000) distinguish between two components: the sense of "agency" involving forward models, and the sense of "ownership." The sense of ownership may be closer to the minimal self than the sense of agency. We feel our body as our own body even when we do not move, or when our action is involuntary rather than deliberate. This feeling corresponds to the sense of "ownership." As in previous examples, it is not necessarily self-reflexive. We can say "my body" without reflecting on the "my." The current assumption in the literature is that only one component of the minimal self, the sense of agency, is disturbed in schizophrenia. However, Parnas suggests both aspects of minimal self are concomitantly impaired in schizophrenia. Indeed, the EASE assessment suggests minimal-SDs include both, impairments of ownership and agency (Parnas et al., 2005). However, it is difficult to disentangle ownership and agency disorders, and to the best of our knowledge the implicit sense of ownership has not been extensively explored in patients (but see de Haan and Fuchs, 2010), such that the phenomenological frontiers between these two components of the minimal self need to be more accurately defined. Lastly, it is still difficult to distinguish between the reflexive and pre-reflexive parts of agency, and more work is required in order to understand which aspects of motor impairments relate to minimal SD in schizophrenia and to what extent.

## **TIME AND MINIMAL SELF IN SCHIZOPHRENIA: WHAT EVIDENCE IN THE LITERATURE?**

After this short review of the motor impairments possibly associated with minimal SDs in schizophrenia, we focus more specifically on exploring the potential implications of time deficits in schizophrenia in minimal SDs.

We shall present two series of arguments to justify time processing as a relevant issue to explore and with a view to understanding minimal SDs in this pathology. Mind structuring disorders, namely arguments deriving from psychopathology and experimental psychology studies in schizophrenia are presented below.

### **EVIDENCE STEMMING FROM CLINICAL, PSYCHOPATHOLOGICAL, AND PHENOMENOLOGICAL APPROACHES**

From a psychopathology perspective, the phenomenological approach proposes a "useful conceptual framework within which the explanation of pathological experiences could be ventured" (Wiggins et al., 2003), to include the role of time in the genesis of minimal SDs. In what follows we try to summarize the hypotheses put forward with the help of this particular conceptual framework. It should be emphasized that these hypotheses rely not on experimental work but on the understanding of time issues derived from phenomenology and on clinical observations.

A first step has to do with the nature of time processing itself. In phenomenology, time is not necessarily investigated and understood as a content of consciousness. Rather, it is a key component structuring the form of consciousness. Thus, time appears to be a very basic, "ontological" component of reality (Wiggins et al., 2003).

At the lowest layers of world-constituting processes, philosopher Husserl (1991) locates the question of time and, more precisely, what he calls "the intimate consciousness of time" (inneres Zeitbewusstsein). He describes a tripartite structure of time consciousness, that is seen as an integration of the past, the present, and the future. He gives the example of music. When we listen to a tune, we are conscious of the present note but still have the previous note in mind ("retention") and usually anticipate the note to come ("protention"; Gallagher and Zahavi, 2014). As Fuchs (2007) points out, "these synthetic functions, operating at the most basic layer of consciousness in an implicit, tacit or automatic way, are capable of integrating the sequence of single moments into an intentional arc," allowing the subject to connect tightly with the world and, thus, structuring consciousness. It should be emphasized that the term "intentional" does not mean the integration of past, present, and future moments is deliberate and reflexive. On the contrary, it is passive. According to phenomenologists, it is a basic mechanism whereby we can consciously experience the world as a whole and as continuous in time. Moreover, such mechanisms would shape all our experiences and affect our sense of self (Fuchs, 2007). The ability passively to integrate past, present, and future moments would allow us to think in meaningful units. In a similar way to what has been described in the domain of motor control, protention would allow us to anticipate the next thought and words when we speak.

This implicit or automatic temporal synthesis (or, in Husserlian terms, the "passive" temporal synthesis) contributes to the stability of the perception of the world. Husserl calls this "doxa," e.g., the certainty that the world will be invariant, allowing the subject to recognize him- or herself in the world. This is one of the most basic processes guaranteeing that the world can be "taken-for-granted" (Wiggins et al., 2003). "As a consequence, the actions of normal people can presuppose a pre-given world, a spatial, temporal, causal, and social order that their mental lives consistently constitute" in an automatic way (Wiggins et al., 2003).

A considerable number of psychopathological works describe a breakdown of this intentional arc in schizophrenia, which could lead to the first incidents of schizophrenic experiences or, put differently, to minimal SDs (Minkowski, 1933/2005; Binswanger, 1965; Tatossian, 2002) or a "weakened ego" (Wiggins et al., 2003). "If these syntheses should cease to occur in my mental life, the ongoing existence of the world and its objects would cease for me" (Wiggins et al., 2003), and the world can no longer be "taken for granted" (Wiggins et al., 2003). Consequently, because the lowest strata of mental life are impaired, the person feels threatened with a kind of selflessness and worldlessness and experiences "ontological anxiety." The "selflessness" experienced here is not reflexive. The presence of "selflessness" is hypothesized by phenomenologists who base their argument on how the experience of the world is supposed to lead to the minimal self. Phenomenological inquiry also relies on clinical observations. In particular clinical observations can be reinterpreted as conscious compensation for minimal SDs. Because patients' experience of the world is distorted, they have to compensate for it. The world has to be actively reconstructed (the person must engage in "rational reconstruction"). A wide range of apparent symptoms can be understood as being a consequence of this fundamental disturbance of minimal SDs, such as hyperreflexivity or the loss of natural evidence. Hence, various aspects of minimal self could be understood or may manifest themselves as an attempt by the patient "to actively busy him- or herself with re-laying the ontological foundations of reality" (Wiggins et al., 2003). This is understandable insofar as there is a balance between the prereflexive and reflexive self. If the pre-reflexive self is weakened, patients would offset this weakness by giving explicitly thought to questions we usually ignore (Nelson et al., 2009). Although an interesting idea, it has often been argued, and rightly so, that it is difficult to demonstrate (Mishara, 2007). However, there is also more direct evidence of the distorted world experience of patients, who appear to stray from a lived and dynamic time. A thorough examination of the literature suggests direct experiences of an explicit feeling of time disruption might be more frequent than suggested by clinical experience (Hartocollis, 1983). For example, patients describe feeling that time is standing still. One of Minkowski's patients gave the following explanation: "I'm looking for the immobility and I have a tendency to immobilize all the things around me" (Minkowski, 1927/2002). Another patient reported: "Things go too quick for my mind. [. . .] It's as if you were seeing one picture one minute and another picture the next" (Chapman, 1966). Similarly, in the words of another patient, "What is the future? One cannot reach it. [. . .] Time stands still [. . .]. This is boring, stretched time without an end" (Fischer, 1929). In the context of our discussion, the following statement by a schizophrenic patient of Bin Kimura quoted in Fuchs (2013) is the following: "Time is also running strangely. It falls apart and no longer progresses. There arise only innumerable separate now, now, now . . . quite crazy and without rules or order. It is the same with myself. From moment to moment, various "selves" arise and disappear entirely at random. There is no connection between my present ego and the one before." As we shall show below, schizophrenia patients' performance in a specific psychophysical task shows behavior interpretable as a disturbance in the continuity of time, as if they were "stuck" in time and could not progress temporally.

This short excursion into the phenomenological approach suggests time is a key component of experience and, more than a content of consciousness, a dynamic component of consciousness. However, as emphasized above, the evidence is based mainly on conscious introspection and verbal reports, and the conclusions are thus necessarily limited. As we have seen, phenomenological theorization interprets verbal complaints to explain non-verbal, pre-reflexive mechanisms. This could be considered a major flaw in the approach. In the following, we develop the contribution of experimental psychology as regards more hidden aspects of time processing which might usefully complement the phenomenological analyses and patients' reports.

### **EVIDENCE FROM COGNITIVE STUDIES TARGETING THE QUESTION OF TIME**

Research into time perception often relies on explicit judgments, meaning that subjects are explicitly asked to make a judgment about a temporal property of external stimuli, such as their duration, order or simultaneity. However, as we have seen above, timing may also intervene incidentally and may be involved in the structuring of consciousness. Indeed, a distinction currently made when exploring time in cognitive science is between "explicit" and "implicit" timing, resting on different neural networks (Coull and Nobre, 2008). As we have seen, explicit processing concerns any mental activity wherever subjects make a deliberate judgment, e.g., about temporal simultaneity or duration. Conversely, implicit processing occurs without a specific instruction and can be either conscious or not (Coull and Nobre, 2008). The implicit processing of time may be especially relevant for minimal self but cannot be explored without considering explicit aspects.

Paradigms based on explicit timing have been used to assess different aspects of temporal processes in schizophrenia. The most typical are "simultaneity judgment" (e.g., assessing the simultaneity vs asynchrony of two stimuli), "temporal order judgment" (e.g., the capacity to order events) and "duration estimation" (e.g., the ability to determine the duration of a time interval; Pöppel, 1997; Wittmann, 1999; van Wassenhove, 2009; Grondin, 2010).

Many studies have been conducted to investigate duration judgment in patients with schizophrenia (Wahl and Sieg, 1980; Tysk, 1983, 1984; Tracy et al., 1998; Volz et al., 2001; Elvevåg et al., 2003; Davalos et al., 2005; Penney et al., 2005; Carroll et al., 2008; Lee et al., 2009; Waters and Jablensky, 2009) or simultaneity judgment (Schwartz et al., 1984; Foucher et al., 2007; Giersch et al., 2009; Schmidt et al., 2011; Martin et al., 2013). The general conclusion of these studies is that patients with schizophrenia find it more difficult than controls to estimate duration (e.g., a greater variability of the performances) and require bigger differences before being able to distinguish the duration of two stimuli. They also require longer inter-stimulus intervals to be able to make temporal order judgments.

However, the significance of these results seems limited, for several reasons. First, there were only a few correlations with clinical dimensions. Moreover, confounding factors such as attention (Brown and Boltz, 2002), memory processing (Zakay and Block, 2004), and emotional characteristics of the stimuli (Droit-Volet et al., 2004; Grommet et al., 2011) may explain the greater variability in performance, especially for duration judgment. Last but not least, the fact that these evaluations rely on explicit judgments implies they explore the content of time and not the temporal structure imposed on consciousness by time processing.

Another way to explore time processing in schizophrenia is to assess "implicit timing." According to Coull and Nobre (2008) "implicit timing is engaged, even without a specific instruction to time, whenever sensorimotor information is temporally structured." It appears that this definition of time is similar to the phenomenological description of human time inasmuch as time is seen as an implicit aspect of processing which does not necessarily lead to a content of consciousness but, rather, shapes the conscious experience. Here, we focus on implicit aspects of timing because they might underlie the involvement of timing in minimal self. We first review the evidence that implicit aspects of timing are impaired in schizophrenia and then discuss how this might impact the minimal self.

Few studies have explored implicit timing in schizophrenia, particularly in perception. Giersch et al. (2009) used a potentially useful priming paradigm, and Lalanne et al. (2012a,b) used the Simon effect to track temporal processes (**Figure 1**); Martin et al. (2013) assessed the temporal constraints of multi-sensory integration in schizophrenia; Posada and Franck (2002) explored the automation of rules and Exner et al. (2006) motor sequence learning.

One example of these studies is the one by Lalanne et al. (2012a,b) using the "Simon effect." In it, subjects have to decide whether two squares are displayed on a screen simultaneously or asynchronously. They respond by hitting a left or right response key (**Figure 1**). Results revealed an enlarged time window with patients, irrespective of the squares' position (intra- vs interhemispheric presentation), and independently of a nonspecific difficulty with processing the information (Giersch et al., 2009). The implicit processing of asynchrony was explored by means of the Simon effect, which refers to the finding that manual responses are biased (lower reaction times and higher accuracy) to the side of the stimulus independently of the task at hand. Basically, the idea is that when stimuli are presented simultaneously on both sides of the screen, no Simon effect can occur, because information on both sides is perfectly symmetrical. However, when stimuli are asynchronous, there is an asymmetry again between the left and right side, due to the temporal delay. This allowed the authors to measure a Simon effect related to this temporal asynchrony. The results showed that healthy subjects were systematically biased to press the button on the side of the second stimulus (Lalanne et al., 2012a,b), and additional studies suggested they are able to follow stimuli in time at a non-conscious

side of the second stimulus, as if able to follow stimuli in time, whereas patients are biased to the side of the first stimulus, as if stuck with this stimulus and unable to follow stimuli with time. With larger SOAs biases are to the side of the second stimulus in all subjects.

level, i.e., even when they do not consciously detect an asynchrony (Giersch et al., in press). At this non-conscious level, patients also distinguished stimuli in time. However, for asynchronies eliciting "simultaneous" judgments, patients' responses were biased to the side of the first square instead of the second one.

The authors interpreted these results as evidence of the fact that at an implicit level patients process stimuli as if they are isolated rather than in succession. They propose that this impairment may be related to disturbed predictive coding. Indeed, stimuli which are less than 20 ms apart mean that subjects process a first stimulus while being prepared for another stimulus which may follow. When the second stimulus is presented only 20 ms after the first one, attention cannot be shifted immediately from the first to the second. If after a while the second stimulus is nonetheless processed and prioritized, it means the system is prepared to process a second stimulus and to shift attention toward it. This might be based on a processing loop which allows new events to be anticipated while the focus is still on current information. This loop could be supplied by the recurrent system of information processing described within the predictive coding framework (Friston, 2008), which is in accordance with an embodied approach of predictive interoceptive coding (Seth, 2013). This hypothesis relates the timing results with those observed with motor control, insofar as both outcomes suggest impairments in the predictive loops that allow us to anticipate events, either the consequence of the action or the next perceptual event. The impairment patients have regarding time may reflect a fragmentation of the processing of information that would impact on the sense of time continuity. In particular, it could be linked to the feeling of "frozen time" described above. Indeed, if patients have difficulty following events in time at an implicit level, it would disrupt their ability to process events in a continuous way, matching what some patients explicitly reported. Moreover, it has recently been shown that patients have difficulty discriminating temporal order at a subjective level (Capa et al., 2014), which reinforces the idea that they have difficulty processing the flow of events. Below, we discuss how such impairments could be related to minimal SDs.

## **TIME PROCESSING AND THE MINIMAL SELF**

If the studies described above doubtlessly suggest impairments at both explicit and implicit levels of time processing (**Table 1**), their impact on minimal SDs is far from straightforward. Despite the limitations of the phenomenological approach, it provides the concepts that allow us to propose a link between the subjective

### **Table 1 | Summary of the results on implicit and explicit timing in schizophrenia.**


experience and the minimal self. What the experimental approach provides on top of that is the possibility to objectify the mechanisms subtending the distortion of the world experience. As emphasized at the beginning of this manuscript, and as suggested in some phenomenological approaches, we consider that the basic experience of oneself is intimately associated with the embodied experience of the world, whether during action or perception. It is because we can establish a close link between our intentions and our actions or between what we expect to feel and what really happens that we can implicitly feel we are the one perceiving or acting. Thus, any disruption of our ability to predict or feel the continuity in the flow of our perceptions and actions should result in a disruption of the minimal self. This can apply even in the case of passive movements. There is usually continuity in our perception of our own body, which is confirmed from one moment to the next. If such continuity is disrupted, however, it should affect our implicit feeling of being one.

As emphasized above, the minimal self is a constituent part of the feeling of being present here and now. This implies that both (i) sensory information processing, and (ii) the way information is processed in time play a role in the formation of the minimal self. As already noted, this information does not have to be unconscious to play a role in the minimal self. It only has to exclude an explicit reflection on the self. Diverse information may thus be integrated in the minimal self, including peripheral as well as central sensory information, which conveys information about our body, but also about the environment and how the body behaves in the environment. This shapes our experience and thus yields a sense of self in the environment, prior to language. Besides, being myself "now" also implies having a sense of present time, and since oneself is felt to be continuous in time, it additionally requires a sense of temporal continuity. In other words the minimal self involves many different mechanisms, including timing mechanisms. This means timing impairments should lead to minimal SDs.

We will thus first detail which timing mechanisms are involved in minimal self, and then how deficits in patients with schizophrenia may induce minimal SDs. Let us first consider the sense of present and its possible relation with temporal windows of information processing. The fact that all sensory information is processed continuously from moment to moment necessitates a temporal integration process that binds multimodal information, into one single representation at one present moment<sup>1</sup> (Pöppel, 1997; van Wassenhove, 2009; Wittmann, 2011). This is the dual nature of time consciousness: the flow of events over time and the feeling of "nowness." These dual aspects are based on the processing of sequences of events over time and the temporal integration of these successive events through multiple and nested functional brain states of various temporal levels (Northoff, 2013; Fingelkurts and Fingelkurts, 2014). Regarding the relationship between temporal integration windows and the self, however, what is especially crucial is to stabilize this representation for a short duration. Otherwise, our perception would be one of an uninterrupted flow of inputs conveyed by different sensory channels. This would lead to a number of difficulties, and especially a difficulty to integrate these different inputs. When considering only one sensory modality, it has been repeatedly suggested and confirmed that there must be a minimal delay of 20–40 ms between two stimuli for this asynchrony to be detected (von Baer, 1864; Brecher, 1932; Elliott et al., 2006, 2007; Fink et al., 2006; Babkoff and Fostick, 2013). The integration of complex information from different channels within one single representation requires even more time (van Wassenhove et al., 2007). It is, however, not as straightforward as it seems. For example, visual and auditory information travels at different speed, and it must be recalibrated to allow for its integration within a single representation (Pöppel et al., 1990). It is only because the motion of the lips and the syllable utterance are integrated within the same time window and processed as co-temporal that we access the representation of another person speaking instead of disconnected visual and auditory information (van Wassenhove et al., 2007). In a similar way, we integrate sensory information from our body and of the environment in single representations which are stabilized from moment to moment. A disruption of these temporal windows may thus fragilize the minimal self. The lengthening of time windows (Giersch et al., 2009) may represent a mechanism of self-disruption in patients with schizophrenia.

Additionally, and as emphasized above, the representation of ourselves is not only stabilized within temporal windows. The self is generally experienced as being continuous in time. This requires that sensory information be processed and integrated in such a way that we experience continuity over time (Dainton, 2010; Wittmann, 2011; Northoff, 2013). Fragmentation of the mental life would mean a disruption of the continuous processing of the sensory information underlying the minimal self. The implicit processing of information over time might be particularly important for our feeling that we go along with the flow of events in a continuous fashion. Conversely, disruption of these implicit mechanisms in patients may disrupt their feeling of temporal continuity. The fact that this impairment occurs implicitly may also explain why they find it difficult to report this impairment clearly. In summary, a number of time processing impairments might contribute to minimal SDs. First, deficient integration of information within temporal windows may hamper information being stabilized. Second, an inability to process sensory information continuously would lead to fragmentation of the information subtending the minimal self.

Time disorders observed in schizophrenia might thus be closely related to minimal SDs, and might subtend a series of cognitive impairments. The existence of these relationships needs to be confirmed in future studies. Although it is difficult to test minimal SDs directly, it is possible to explore the impact of the impairments described on conscious experience. Such studies involving patients with schizophrenia should provide insight into how time disorders affect our ability to feel ourselves implicitly to be at the center of our perceptions and actions. Thus, the question of whether the alterations observed in the simultaneity/asynchrony judgments are related to the contents of consciousness warrants investigation.

<sup>1</sup>Although the concept of representation is not at the center of the phenomenological approach, the idea of information integration leads to the notion of representation. In this sense, minimal self would be similar to the concept of core consciousness as defined by Damasio (1998).

## **IMPLICATIONS FOR CURRENT RESEARCH ON TIME AND MINIMAL SELF IN SCHIZOPHRENIA: METHODOLOGICAL CONSIDERATIONS**

### **NECESSITY OF RELATING PSYCHOPATHOLOGY WITH EXPERIMENTAL APPROACHES**

Although there is a growing interest in minimal SDs in schizophrenia, there is an apparent neglect as regards experimental approaches regarding psychopathological hypotheses.

Whereas cognitive research into minimal SDs in schizophrenia rarely includes the issue of time, in the case of cognitive research into time processing in schizophrenia the question of minimal SDs is rarely included. This is all the more paradoxical given that these areas were originally united (Minkowski, 1933/2005), and that psychopathological analysis has gathered data and developed concepts about both for a long time. More recently, however, these two lines of research have developed separately. According to Nelson et al. (2013): "Although progress has been made in understanding phenomenological and neurocognitive disturbances in schizophrenia, these "levels" or domains of enquiry have tended to remain separate from each other". Combining psychopathological phenomena with experimental approaches seems a promising way to proceed, at least on a heuristic level.

### **ASSESSING THE MINIMAL SELF: INCLUDING SUBTLE AND DEMANDING CLINICAL EXPLORATIONS IN RESEARCH PROGRAMS**

Assessing minimal SDs requires indisputable clinical expertise, with the help, for example, of EASE, a phenomenologically oriented instrument (Parnas et al., 2005). Regarding EASE, "the interviewer must possess good prior interviewing skills, detailed knowledge of psychopathology in general and of the schizophrenia spectrum conditions in particular." So "a familiarity with phenomenological description of the structures of human consciousness is indispensable in using the EASE for pragmatic, psychometric purposes" (Parnas et al., 2005).

Consequently, whether using EASE or some other scale, assessing minimal SDs requires that investigators involved in experimental research incorporate subtle clinical analyses within their research programs.

### **"TIME OF PERCEPTION" IS NOT "PERCEPTION OF TIME": ASPECTS OF EMBODIMENT**

As claimed in phenomenology, human time is not necessarily a content of consciousness. Accordingly, as stated by Pöppel (2009), "it is important to realize that we cannot perceive time itself." We believe that paradigms based on implicit time processing as proposed by Giersch (with the Simon effect) or Coull (using automatic temporal accumulation) are promising tasks for exploring correlations between implicit time processing and minimal SDs. However, this does not preclude investigating explicit time, i.e., the judgment of duration, especially since it has been shown that the bodily self is related to time perception in the seconds' range (Wittmann, 2013). These empirical findings confirm embodied notions of subjective time (Craig, 2009) and link the experiences of emotion, time, and interoception for the creation of a self (Seth, 2013). Inasmuch as the minimal self is related to conscious and embodied experiences, such investigations are necessary for understanding how implicit (and explicit) time processing could affect impairments observed on a more subjective level.

## **CONCLUSION**

According to research in psychopathology, temporal processing and minimal SDs appear as core components of the schizophrenia condition. In general, the notions of time and the minimal self are two fundamental components of human consciousness. "Time and the self, time and consciousness, affects and time are pairs of subjective reality, phenomena that appear together in the course of man's ontogenetic development and define human nature [. . .]. Even though conceptualized independently, they cannot be experienced separately, they cannot exist without each other" (Hartocollis, 1983, p. 56). Combining the two dimensions of minimal self and time in experimental studies requires subtle clinical analysis as well as the exploration of implicit time processes, which, by definition, are not reflective. What we have shown, however, is that they can disrupt a patient's experience of the world at fundamental levels, thus contributing to the minimal SDs. In particular, we presented evidence of our own experimental studies relating to an implicit timing disturbance in patients with schizophrenia which strikingly mirrors patients' reports of subjective time. We have potentially found an experimental task which directly assesses the implicit timing disturbances of patients with schizophrenia who, on an explicit level, become aware of theses impairments. The underlying mechanisms might also come into play in our ability to feel in control. Experimental approaches can yield further information about the consequences of these basic impairments for how subjects experience their perceptions and actions. We are aware that this is not enough to fill the gap that still exists between the phenomenological approach and experimental investigations exploring elementary aspects of timing, but we argue there is enough ground to at least try to relate minimal self and timing disorders. Furthermore, we propose that exploring the relationships between implicit timing and the temporal structure of consciousness may help further close the gap between the experimental and phenomenological approaches.

## **AUTHOR CONTRIBUTIONS**

Brice Martin designed and wrote the first draft of the paper, Marc Wittmann and Anne Giersch made substantial contributions to the design of the manuscript, and important revisions. Nicolas Franck contributed to the elaboration of the concepts and the revision of the manuscript regarding clinical aspects. Michel Cermolacce contributed to the elaboration and the revision of the paper regarding phenomenological aspects. Fabrice Berna contributed to the elaboration and revision of the manuscript regarding the concept of self. All authors gave final approval of the submitted version and agreed to be accountable for all aspects of the work.

## **ACKNOWLEDGMENTS**

This work was supported by the French National Institute for Health and Medical Research (INSERM), and the French National Research Agency (ANR-10-BLAN-1903-01 to Anne Giersch). We would like to thank Virginie van Wassenhove and the Timely Cost network for useful discussions prior to the writing of this manuscript.

### **REFERENCES**


Minkowski, E. (2002). *La Schizophrénie*. Paris: Payot (1st publication 1927).


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 25 June 2014; accepted: 26 September 2014; published online: 21 October 2014.*

*Citation: Martin B, Wittmann M, Franck N, Cermolacce M, Berna F and Giersch A (2014) Temporal structure of consciousness and minimal self in schizophrenia. Front. Psychol. 5:1175. doi: 10.3389/fpsyg.2014.01175*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Martin, Wittmann, Franck, Cermolacce, Berna and Giersch. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## On the temporality of creative insight: a psychological and phenomenological perspective

## *Diego Cosmelli\* and David D. Preiss*

Escuela de Psicología, Facultad de Ciencias Sociales, Pontificia Universidad Católica de Chile, Santiago, Chile

### *Edited by:*

Claire Zedelius, University of California Santa Barbara, USA

### *Reviewed by:*

Zbigniew R. Struzik, The University of Tokyo, Japan Claire Zedelius, University of California Santa Barbara, USA Liane Gabora, University of British Columbia, Canada

### *\*Correspondence:*

Diego Cosmelli, Escuela de Psicología, Facultad de Ciencias Sociales, Pontificia Universidad Católica de Chile, Avenida Vicuña Mackenna 4860, Macul, Santiago, Chile e-mail: dcosmelli@uc.cl

Research into creative insight has had a strong emphasis on the psychological processes underlying problem-solving situations as a standard model for the empirical study of this phenomenon. Although this model has produced significant advances in our scientific understanding of the nature of insight, we believe that a full comprehension of insight requires complementing cognitive and neuroscientific studies with a descriptive, firstperson, phenomenological approach into how creative insight is experienced. Here we propose to take such first-person perspective while paying special attention to the temporal aspects of this experience. When this first-person perspective is taken into account, a dynamic past–future interplay can be identified at the core of the experience of creative insight, a structure that is compatible with both biological and biographical evidences. We believe this approach could complement and help bring together biological and psychological perspectives. Furthermore, we argue that because of its spontaneous but recurrent nature, creative insight could represent a relevant target for the phenomenological investigation of the flow of experience itself.

**Keywords: creativity, insight, flow of experience, temporality, socio-cultural approach**

## **INTRODUCTION**

Try to recall the last time you had the experience of a sudden breakthrough in thinking about some unresolved issue. It can be as academic, as artistic, or as domestic as you please. This experience is usually accompanied by a more perceptible, sudden experience of resolution, which is called the insight into the problem's nature (the Aha! experience). Try to recall the feeling of insight itself, which is sometimes related to a positive feeling that goes beyond the cognitive restructuring of the problem. Would you agree that, in some sense, the moment of creative insight involves an embodied expression of surprise and familiarity? Of something that is at the same time both new (future oriented) and already known (past oriented)?

The field of insight and creativity research is extremely active (see Runco, 2004; Hennessey and Amabile, 2010; Sawyer, 2012; Kounios and Beeman,2014; for comprehensive reviews). Although we place ourselves within the general socio-cultural approach, here we will focus on one particular aspect of creativity, namely the experience of creative insight (Sternberg and Davidson, 1995). Despite the extensive development of research on the psychological, cognitive, neuroscientific, and social aspects of creativity, it is worth noting that a more descriptive, phenomenological approach to how creative insight is experienced is still quite limited (Bindeman, 1998; Nelson, 2005). Here, we will argue that, when such phenomenological approach is taken into account, the temporal nuances of the experience of insight are brought to the fore in a way that has consequences that transcend the study of creativity. These temporal nuances highlight the pastoriented and future-looking dimensions of insight. Interestingly, this temporal structure resembles biological dynamics that span from brain activity to evolution. Moreover, attention to the

phenomenological structure of insight could help understand the connections between neurobiological accounts stemming from cognitive neuroscience and more biographical and psychological accounts collected by sociocultural and systems approaches to creativity (Sawyer, 2012).

### **PSYCHOLOGICAL STUDIES OF CREATIVITY AND INSIGHT**

It was not until Guilford's (1950) APA address that the notion of creativity drew the attention of psychologists as a matter of scientific interest (Sternberg and Lubart,1999; Sternberg and Grigorenko, 2001). Guilford also inspired the study of creativity in everyday situations by using psychometric methods. Since, divergent thinking tests, such as the Torrance Tests of Creative Thinking (Torrance, 1972), which were based on work by Guilford (1967), have been extensively used. These tests show participants questions that can be answered in an open way: for example, "List things that would happen if we lived in a world without gravity." Many of these tests are based on verbal stimuli, although they can also be figural and involve, for example, the completion of a drawing. The answers are scored in dimensions such as fluency (how many responses are produced) and originality (the responses' uniqueness), among others. Despite their influence, divergent thinking tests have been accused of trivializing the concept of "creativity," for reasons we agree with (Sternberg and Lubart, 1999; Sternberg and Grigorenko, 2001). Specifically, the divergent thinking construct reduces the concept of creativity to a problem solving process that is restricted to a specific task and moment in time. Yet, as illustrated by both descriptive evidence about the creative process (Bindeman, 1998) and biographical analysis through case studies (Gruber and Wallace, 1999) or in-depth interviews (Csikszentmihalyi, 1996), the generation of a creative product in

a relevant cultural domain is neither an instantaneous event nor simply a matter of purely reflective problem solving. It is well known that the acquisition and mastery of specialized knowledge of the domain, involving a sophisticated interplay between convergent and divergent psychological processes (Goel, 2014) are necessary to account for real-world creative activity. A case in point is that of Darwin, who, as documented by Gruber and Wallace (1999), developed four or five stages of his theory of evolution from 1831 to 1838, as demonstrated by the record of his observations and his thinking, which includes both verbal and visual graphics. The analyses of cases such as that of Darwin illustrate that "it would be difficult if not impossible to construct the narrative of a case study using only one timescale. Short-term activities and experiences are embedded within longer episodes, and so on" (Gruber and Wallace, 1999, p. 104).

Contrasting with the more encompassing problem of creativity as such, research on the more temporally restricted phenomenon of insight as an event has a longer history, extending back to the contributions of the Gestalt movement. It is interesting to note that whereas in its origins the concept of insight was related to the observation of problem solving behavior in open-ended situations—such as those observed by Kohler (1925) with chimpanzees—by the end of the 20th century it was mostly assessed in experimental contexts through closed-ended problems. Thus, creative insight became mainly conceptualized as a specific type of problem-solving process, specifically, one that is not lived incrementally but is rather characterized by an impasse and a sudden, abrupt, and unpredictable reconfiguration of the problem (Metcalfe and Wiebe, 1987; Sternberg and Davidson, 1995). Because it involves a non-analytical strategy that mobilizes both explicit and implicit processes to produce usually unexpected solutions, insight has been considered a core element of creative problem solving. In many occasions, this has led to culturally equating insight with creativity itself (see Sawyer, 2012 for a critical view).

From Wallas's four-stage model of preparation, incubation, illumination, and verification (Wallas, 1926, in Hélie and Sun, 2010), to recent work in psychology of insight and intuition, there is a wide agreement that finding creative solutions, despite being usually characterized by a sudden and holistic "Aha!" is not exhausted by this local, more overwhelming aspect of the experience. As mentioned above, the importance of systematic involvement and expertise in the domain of knowledge as conditions that make successful insight-based solutions possible are well known (Runco, 2004; Hennessey and Amabile, 2010; Sawyer, 2012). Likewise, the facilitating effect of incubation periods, where the question is put to rest, has been consistently reported (Sio and Ormerod, 2009; Hélie and Sun, 2010; Baird et al., 2012). Furthermore, it is noteworthy that in many insight-type problem solving tasks, subjects use back-and-forth iterative formulation of possible solutions that are associated with smaller, partial insights (Schooler et al., 1993; Bowden et al., 2005; Hélie and Sun, 2010). These studies show that the time leading up to the moment of insight can be as important as the insight experience itself, by providing the relevant conditions and the (mostly implicit) interpretative context where the Aha!-experience makes sense (see Elements for A Phenomenology of Creative Insight). This

is consistent with work in the cognitive neuroscience of insight showing that resting-state brain activity prior to solving a problem can be used to predict whether it will be solved through insight or non-insight strategies (Kounios et al., 2006, 2008). It has also been shown that the likelihood of producing insightful solutions can be modulated by internal states such as mood and attentional distribution (Subramaniam et al., 2009).

As much as the abovementioned approaches have contributed to a more comprehensive understanding of creative insight from process-based psychological and sociological approaches, a descriptive, phenomenological investigation into the way insight is experienced *subjectively* remains surprisingly underdeveloped (Bindeman, 1998; Nelson, 2005). One can only speculate about the reasons for this neglect, some of which are probably related to the underprivileged status first-person data has historically had in the cognitive sciences (Varela and Shear, 1999). However, the tide is slowly turning with mainstream journals publishing studies that take advantage of systematic, rigorous first-person descriptions to guide empirical questions and analysis (Lutz et al., 2002; David et al., 2003; Cosmelli and Thompson, 2007; Christoff et al., 2009). Even if a phenomenological description does not manage, eventually, to bridge the biological, behavioral, and psychological perspectives (Petitot et al., 2000; Schwartz and Metcalfe, 2011), it could provide relevant analogies or heuristics to expand our understanding of this deeply significant experience (Sass, 2001).

## **ELEMENTS FOR A PHENOMENOLOGY OF CREATIVE INSIGHT**

Consider again how the sudden "Aha!" is experienced during creative insight. Among other aspects, this moment is usually accompanied by a positive affective feeling of something "coming together,""making sense," or somehow "falling into place" (see also Schooler et al., 1995, pp. 578–579). From a cognitive psychology approach, this feeling has been proposed to be dependent on a sudden gain in processing fluency (Topolinski and Reber, 2010). From a phenomenological perspective, this moment reveals two complementary aspects of how the experience unfolds in time.

On the one hand, the felt relevance of an answer obtained through insight (independent of its eventual correctness) is always related in a co-generative manner to a prior "wanting" or "lacking" context to which such answer responds. Accordingly, insight solutions are commonly experienced as "gap-filling" (Gruber, 1995; see also Pelaprat and Cole, 2011 for a convergent view regarding imagination), something that can only make experiential sense if both ends of the gap are available at some point of the process (Hélie and Sun, 2010). By virtue of this gap filling, the moment of insight brings with it a very strong and sharp reference to what was going on the moment before. In Runco's words, "A creative insight is not a quick "aha!" but instead is protracted" (Runco, 2004, p. 662). In this sense, the moment of insight bears a notable resemblance to the resolution of the tip-of-the-tongue experience. It has been proposed that in such cases, this could be associated with succeeding in bringing forth phonological information to match a context of previously activated but incomplete semantic information (Gollan and Brown, 2006). During linear, incremental reasoning the prior context is transparent and fully available in a manner

that does not depend on subsequent steps in the thought process. In contrast, when the solution emerges during insight, it retrospectively illuminates the previously opaque context, and makes sense by referring to something that was, until a moment ago, unavailable. In doing so it highlights how the present and the immediate-past are deeply intertwined in the formation of novel meaning.

On the other hand, when the experience of creative insight happens it does not only solve or close a previously posed problem. Both in controlled problem solving settings and during spontaneous insights (those unrelated to a specific contextual problem), it is important to differentiate another aspect of its temporality, one that has received much less attention, especially from the cognitive neurosciences. Insight solutions are creative not only because they solve a given problem in an unexpected and novel way. They are also creative because they can (and usually do) involve a change in the perception or representation of the problem itself (also known as restructuring, see Weisberg, 1995; Chi, 1997). As such, creative insight can open up a potential set of new problems by changing the way the current problem is interpreted vis-à-vis its future consequences.

In contrast to the gap-filling, past-oriented side, this futureoriented aspect is most clearly illustrated through biographical accounts of spontaneous insight, and interviews with individuals—in many cases famous scientists—that discover a new way of looking at an old problem or produce a theoretical synthesis of previously unrelated phenomena, (Csikszentmihalyi and Sawyer, 1995; Gruber, 1995). Poincaré's description of the consequences of intuiting the order revealed by a mathematical demonstration is illustrative: "A mathematical demonstration is not a simple juxtaposition of syllogisms, it is syllogisms *placed in a certain order*, and the order in which these elements are placed is much more important than the elements themselves. If I have the feeling, the intuition, so to speak, of this order, so as to perceive at a glance the reasoning as a whole, I need no longer fear lest I forget one of the elements, for each of them will take its allotted place in the array, and that without any effort of memory on my part" (Poincaré, 1910, p. 324). And as he goes on to say when speaking about individual differences in mathematical ability: "Others, finally, will possess in a less or greater degree the special intuition referred to, and then not only can they understand mathematics even if their memory is nothing extraordinary, but they may become creators and try to invent with more or less success according as this intuition is more or less developed in them." Phenomenologically speaking, more than the knowledge just gained, this generative, forward-looking aspect of the experience emphasizes the direction (or rather potential directions) toward which one is left facing, so to speak, as a consequence of one's insight into the problem's nature. During spontaneous insight this aspect of creative insight can be very powerful as one's flow of experience becomes unexpectedly diverted toward the consequences of the realization.

It is worth considering that the two-sided past-closing/futureopening structure is analogous across other levels of organization. For instance, it is consistent with the view of the brain and body as a system driven mainly by endogenous, historically dependent dynamics, which support action perception cycles by continuously minimizing prediction errors (Friston, 2005; Clark, 2013). As pointed out by neurodynamicists reaching back to Karl Lashley, it makes no sense to analyze neural activity purely as happening instantaneously or just as a reaction to impinging stimuli, while ignoring its ongoing, predictive nature (Freeman, 2000; Thompson and Varela, 2001; Raichle and Gusnard, 2005; Cosmelli et al., 2007). In other words, ongoing brain activity creatively prefigures virtual, motor and perceptual possibilities by continuously bringing forth our history of interactions into the present "now" (Varela, 2000). Or consider the dynamics of biological evolution, whose similarities with creativity and insight have been pointed out previously (Simonton, 1999, 2013). For example, in an analogous way to what happens during restructuring, it can be argued that, evolutionarily, flying is not just a prior problem waiting to be solved with wings or membranous forelimbs (as in bats, see Sears et al., 2006). Flying is also possible *as a set of new problems* because wings or membranous forelimbs are available. Evolution is not exhausted by pure random variation; it is the emergence of novelty within boundary conditions, conditions that are (self) affected precisely by that which emerges. It is tempting to think that such analogies might point to underlying common mechanisms in biological systems (see also Perkins, 1995).

### **CONSEQUENCES AND FUTURE DIRECTIONS**

We believe that the previous analysis has a number of consequences that are relevant for studies on creative insight, but also for a phenomenology of the flow of experience. Take for instance the past-oriented, gap-filling aspect. As we discussed above, it brings to the fore in a very palpable manner, the intimate relation that exists between the present and the immediate past in experience (James, 2007; first edition-1890). Such past-looking, *retentive* aspect that is always available in the experience of the present "now" has been consistently described in the phenomenological philosophy tradition (Merleau-Ponty, 1976; Husserl, 1991; Sherover, 2001) and has been proposed to play a role in providing unity to the flow of experience (Varela, 2000). The fact that it is made so evident during the "Aha!" moment is what we wish to highlight here: it suggests that spontaneous occurrences of insight situations could be a natural target for the phenomenological inquiry into the ongoing flow of experience. One of the difficulties faced by phenomenological investigation is pinpointing the object of description, especially when its appearance is unpredictable. By providing a recognizable, easily relatable, "anchor point," creative insight can facilitate taking a phenomenological attitude toward a well-defined target that preserves the spontaneity of the flow of experience. In this sense, it could play a role similar to that proposed by Schwartz and Metcalfe (2011) for tip-of-the-tongue experiences, becoming a natural candidate for phenomenological study that can be contrasted with psychological, cognitive or neuroscientific accounts.

It remains an open question whether a more in-depth exploration of the moments leading up to the experience of sudden insight can be the subject of phenomenological investigation, for the sake of understanding creative insight itself. As studies since Metcalfe and Metcalfe and Wiebe's (1987) seminal work have shown much of what is going on during the incubation period prior to the moment of the Aha! experience is not accessible to the subject. This would in principle challenge the utility of phenomenological descriptions of this moment of the experience. However, one need not stay exclusively with the phenomenological approach. That resting state activity prior to confronting a problem can, in principle, predict whether a problem will be solved through insight or not suggests a possible strategy. Consider the proliferation of signal analysis algorithms that are currently available for ongoing EEG decomposition in the context of brain–computer interface development (Krusienski et al.,2011). Subjects could be prompted for descriptions of their ongoing experience when characteristic features of brain activity associated with insight solutions (i.e., changes in peak frequency in the low alpha band or drop in alpha band power over mid frontal and left anterior temporal regions, see Kounios et al., 2006) are detected. In particular, the locus of attention could be a relevant target as it has been proposed that a diffuse distribution of visual attention might be a characteristic feature of insight-based solutions (Kounios et al., 2008, p. 283, see also Subramaniam et al., 2009). This would be a brain activity-based sophistication of the strategy used originally by Metcalfe and Wiebe (1987), whereby subjects were asked at different moments prior to the Aha! experience to evaluate their feeling of approaching a solution.

The more future-oriented, *protentive* side of insight, on the other hand, underscores the self-affecting nature of experience in a very clear way. Restructuring implies that the problem takes its final form (meaning, consequences, etc.) only when the solution is discovered. As such, it suggests that the less studied restructuring aspect could be a relevant focus, for example, when seen in a more learning oriented setting. Most psychological and cognitive neuroscience studies have dealt with those mental or brain processes leading up to or facilitating the moment of restructuring (Metcalfe and Wiebe, 1987; Bowden et al., 2005; Kounios et al., 2008; Kounios and Beeman, 2009; Subramaniam et al., 2009; Eubanks et al., 2010). However, much less is known regarding the consequences of restructuring for future, possibly recurrent encounters with similar problems. For instance, it would be interesting to study if and under which conditions repetitive exposure to certain type of insight problems can eventually lead to generalizations or the development of strategies to deal with them (see also Weisberg, 1995 and Eubanks et al., 2010). If this restructuring aspect, which is more clearly available in understanding-type insights (Gruber, 1995), is characteristic of creative insight in general, one would predict that every instance of closed-problem insightful resolution would change to some degree the way the problem is judged. Such change might be small and difficult to detect for each individual event, but if it exists, it would be reasonable to expect that at some point of recurrent encounter with a given type of such problems, some kind of meta-insight regarding the underlying logic should become available to the person. This expertise-related prediction should, in principle, be amenable to experimental testing with current problem-solving based approaches both in terms of changes in behavior and in brain activity.

An obvious limitation of any phenomenological enterprise is that it deals, by definition, with experience as described by the same subject of that experience. Here we have taken this as a starting point but we have strived to triangulate our observations with psychological studies and cognitive neuroscience results in the spirit of cross-validation. We believe that the results of this triangulation are encouraging and point to potentially relevant lines for further inquiry. In the limited space of this essay we cannot tackle a full-fledged phenomenological investigation into creative insight. We have, however, focused on a two outstanding aspects that warrant further attention not only from a phenomenological perspective but also in terms of psychological and biological approaches. These descriptions are open to contrast with other researcher's experience on the one hand, and with future results from experimental approaches on the other. By its very nature, phenomenological investigation has to start from the individual and seek intersubjective contrast, refinement, and eventually, validation. The perspective here adopted aims to invite others to adopt a much-needed phenomenological stance in order to contribute to the understanding of the experience of creative insight.

We have argued here that creativity in general—and insight in particular—offers an extremely rich case, phenomenologically speaking, for the study of the temporal structure of human experience, one that represents a challenging venue of research. Indeed, creative insight brings forth and coherently embodies the futurepast polarity of experience in a very explicit way. As such, it is at the same time the experience of something surprisingly novel but profoundly familiar.

### **ACKNOWLEDGMENTS**

The authors thank FONDECYT Grant N◦ 1130836 "Individual Differences in Attention and Metacognition: evaluating their impact on argumentative and narrative reading and writing in high-school and undergraduate university students." DC thanks CONICTY: project"Anillo en Complejidad Social"SOC-1101. The authors wish to thank Luz M. Alliende for technical help in the preparation of the manuscript.

### **REFERENCES**


eds P. D. Zelazo, M. Moscovitch, and E. Thompson (Cambridge: Cambridge University Press), 731–774.


Wallas, G. (1926). *The Art of Thought*. New York, NY: Harcourt, Brace.

Weisberg, R. W. (1995). "Prolegomena to theories of insight in problem solving: a taxonomy of problems," *The Nature of Insight*, eds R. J. Sternberg and J. E. Davidson (Cambridge, CA: The MIT Press), 157–196.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 June 2014; accepted: 29 September 2014; published online: 17 October 2014.*

*Citation: Cosmelli D and Preiss DD (2014) On the temporality of creative insight: a psychological and phenomenological perspective. Front. Psychol. 5:1184. doi: 10.3389/fpsyg.2014.01184*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Cosmelli and Preiss. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## The long is not just a sum of the shorts: on time experienced and other times

## *Jirí Wackermann ˇ \**

Empirical and Analytical Psychophysics, Institute for Frontier Areas of Psychology and Mental Health, Freiburg im Breisgau, Germany \*Correspondence: mail@jiri-wackermann.eu

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

### *Reviewed by:*

Alexander Fingelkurts, BM-SCIENCE Brain and Mind Technologies Research Centre, Finland Andrew A. Fingelkurts, BM-SCIENCE Brain and Mind Technologies Research Centre, Finland

**Keywords: time experience, time perception, biographic time, phenomenology, psychophysics**

Psychology should be a positive, empirically informed science of human existence; that is, a science of human experience of and acting in the world, in their unity. Therefore, psychology has to take human experience seriously. What does it mean for "psychology of time"?

The term was coined more than a century ago (Nichols, 1891); but what "psychology of time" is or should be has never been defined with full clarity. Its core topic always has been "time perception"—originating in early psychophysical (Mach, 1865) and physiological (Vierordt, 1868) studies of "time sense"—with miscellaneous time-related topics (psychological, developmental, cultural) attached (Fraisse, 1963; Doob, 1971; Grondin, 2008). Recently, "recognition of the centrality of psychology of time" has been advocated (Hancock and Block, 2012), but what is that specific for psychology? Time is a universal concept used across scientific disciplines, from physics down to paleontology or archaeology. In physics, dynamical descriptions of the phenomena under study are essential, and thus dimension of time plays a fundamental rôle; yet there is no special chapter of physics named "physics of time" that could claim such a "centrality." Psychology studies mental phenomena in their temporal course or relations—e. g., reaction times, choice times, dynamics of perception and cognition—but this still does not constitute a special "psychology of time." As social and cultural beings, we act "in" time and with regard to temporal schedules: wide fields for psychological research, but why have a particular sub-discipline for that? Why not leave study of habitual latecomers to psychology of personality, and decision times to psychology of consumer behavior?

What makes an essential difference between events just occurring "in" time motions of stars, geyser eruptions, heart contractions—and human acting "in" time is that we are *aware* of time. Or, more precisely: temporal characters, change, flux of world-states and bodily states, are integral part of our experience of the world and of our-selves in it.

The term "experience of time" is mostly used synonymously with that of "time perception," though it may also cover experience of duration *plus* awareness of other, qualitative temporal characters (sensible present, past–future, etc.). I will use it in the more restrictive sense. Of course, "time perception" is a misnomer; "time is not an apple" (Woodrow, 1951). We do not perceive time; what we *do* perceive are events, occurring "out there" in the world, their temporal qualities and relations. Here we are interested not so much in performance of human subjects in "time perception" tasks but rather in the subjective experience as the primary basis upon which the notion and knowledge of time are constituted.

Consider a simple psychophysical experiment: a luminous stimulus of duration varied across several orders of magnitude (o. o. m.) is presented to the observer. For very short durations, from 1/1000 s up to about 1/100 s, only a flash of light of indefinitely short duration is seen. Variations of physical duration do not translate directly into experienced duration; it is rather the integral magnitude of the luminous sensation—its "volume," so to speak—that changes. Conversely, changes in the exposure time can be counter-balanced by changing the luminous flux to obtain the same sensation magnitude (Talbot's law). From some critical duration on (∼1/30 s), the percept gains an elementary temporal quality, so that the observer is able to distinguish between shorter and longer exposures, but its singular "flash-like" character is preserved. Finally, at exposure times of about 1/2 to 1 s, the onset and outset of the luminous appearance can be differentiated as two distinct events. From here on, the temporal extension of the stimulus is really *perceived* as duration.

The experiment illustrates existence of *regions* on the physical continuum of the control parameter (duration) delimited by more-or-less well defined *boundaries* (Wackermann, 2007). Stimuli from within the same region elicit percepts of the same quality; transgressing a boundary causes a transition to a different kind of experience. Descriptions applicable within one region cannot be transfered naïvely to another region, as exemplified by the brightness–duration interaction. This says that subjective experience is not deliberately "scalable" or decomposable into smaller elements. In our experiment, an extremely fast flash of light—say, a xenontube electric discharge in the o. o. m. of microseconds—will result in an irreducible sensory datum where the notions of "shorter" or "longer" cannot be applied meaningfully. It is not just "shorter" than in the o. o. m. of milliseconds; it is *qualitatively different* and thus *incommensurable*. In physics, 1 ms = 1000 × 1µs; but one perceived "milliflash" does not consist of a thousand "microflashes."

Perception of duration does not hold for deliberately long times, either. For example, in *duration reproduction* experiments with stimuli prolonged up to a few tens of seconds, the response curve i. e., reproduced duration expressed as a function of the presented duration progressively flattens, discrimination acuity decreases, and the observers become uncertain or unable of beholding the duration as a singular, indivisible experiential datum. This phenomenon finds a natural interpretation within the framework of the "lossy integration" model of neural representation of time (Wackermann and Ehm, 2006; Sysoeva et al., 2011): as the durations become comparable with the relaxation times of the lossy integrators, duration discrimination necessarily deteriorates. This is why we call this upper bound of "live" time experience the "horizon of reproducibility" (Wackermann, 2007). There is no sharp boundary, <sup>1</sup> but the ability of duration representation as a unitary experiential datum definitely ceases somewhere in the o. o. m. of a few minutes. Alternative mechanisms may be involved and additional temporal cues invoked for a cognitive re-construction of longer durations.

This is *a fortiori* true for still longer times of hours, days, years. The multitude of conventional units alone indicates how time becomes fractioned and structured in a system of parallel overlapping time-scales, maintained by clocks and calendars. Clearly, it is the inability of human mind to keep a "measure of time" across longer intervals that enforced the invention of *external* time-keeping, chronometry, time reckoning (Whitrow, 1988; Birth, 2012). On these time-scales there is no "time experience"; these times are not "perceived," only *known*. Also, external time-keeping allows us to refer even to events beyond the biological limits of our lives, and creates a kind of impersonal objectivity we associate with the order of world-time.

Touching the problem of human life-time: we should distinguish between biological time, seen from outside, and biographic time, seen from inside. Humans not only live but "conduct" their lives (Plessner, 1975), and are aware of times of their lives. This existential fact finds its psychological counter-part in the concept of "mental time travel," that is, the ability "to mentally project themselves backwards in time to re-live, or forwards to pre-live, events" (Suddendorf and Corballis, 2007). It is a highly problematic metaphor, as the notion of "time travel" itself. Nonetheless, the catchy metaphor has made a career in the literature, and motivated research in comparative psychology, personality psychology, cognitive neuroscience, etc. (Suddendorf et al., 2009; Nyberg et al., 2010).

Now, representation of past events, or anticipation of future events refers explicitly to *biographic* time, consisting of (only partially ordered) biographic moments, episodes of personal relevance and existential importance, separated by indeterminate "time-spaces." Some of past moments may be identified in terms of calendar time, but their "time indices" are more often given by a network of logical and material causalities, social circumstances, etc. The content of those episodes may be "experienced" (memory recall, anticipating imagination), but not their times; these are accessible only by cognitive (re)construction.

By contrast, experience of temporal characters of perceivable events (for short: "time perception") happens on a "local timescale": in the range (approximately) from 10<sup>0</sup> s to 102 s (Wackermann, 2007), i. e., of about two decadic o. o. m. A unique temporal quality specific for this domain is *duration*, as a measure of "temporal distance" between distinct events and sub-events. Only in this relatively narrow domain we can speak properly of "experience of time," or "time experienced."

These are two different orders of different kinds of events, or plainly: *two different times*. Biographic time is *not* a sum of chunks of experiential time; and conversely, experienced durations are *not* measures of fragments of biographic time. It is not only a matter of different timescales: the long does not result from summation or multiplication of the short. The two orders of times are mutually irreducible and thus *incommensurable*. Of course, we can map both orders of time, those "perceived" and those "known," onto the continuum of physical worldtime. The difference of time scales does not play any rôle, since physical time is arbitrarily scalable. In physics, we can measure times of planetary motions by times of light-wave oscillations, or *vice versa*—not so in the realm of human experience<sup>2</sup> . The mapping onto a common background of world-time obscures or eliminates experiential content of the originals and the essential difference between them. Projections of two things onto a photographic plate give one fused image, but this does not make the two originals to be one thing!

Psychologists aiming at a unified "psychology of time" may be mislead by the concept of universal time of physics, applicable across all time-scales, and search for its analogy in the mental domain. Although they recognize differences between time of subjective experience and the objective clock-time evergreen of popular "psychologies of time" since ever—they adhere to the idea of a unitary (though not inter-individually identical) order of mental or biographic events, arranged along a continuum of subjective time indices: "psychological time." I will not reiterate my earlier critique of this concept (Wackermann, 2008); may it suffice to say that there is no necessity for the construct of "psychological time" and no compelling evidence for it. In fact, phenomenology of our experience of time speaks rather *against* it.

Once again: experience matters. We should observe the very structure and texture of experience precisely and adjust our conceptual schemes and theories to what is observed—not the other way round. Findings from experimental or clinical neuroscience may provide novel insights (Fingelkurts et al., 2010; Nyberg et al., 2010; Østby et al., 2012) but we should be aware that these are only *supplementary*

<sup>1</sup> If our interpretation in terms of "lossy integration" is correct, the fuzziness of the upper boundary may be explained by inter-individual differences (Sysoeva et al., 2010) and intra-individual state-dependence (Späti, 2005; Wackermann et al., 2008) of the relaxation times of the integrators.

<sup>2</sup> In the model of time as arithmetic continuum, time indices of instants and time intervals between instants are naturally connected by arithmetic operations, and so we easily forget the generic difference between the indices ("when-times") and durations ("betweentimes"). In subjective experience, however, these are two different things!

data, informing but not determining our concepts. Regarding arguments from neural correlates and, particularly, conceptual unifications based on "shared networks": there is only one brain for all kinds of mental functioning; resources are limited, so functional re-use is likely and probably necessary. The same applies, *mutatis mutandis*, to studies with neurological patients (El Haj et al., 2013).

The qualitative difference between "time experienced" and "time known" is evident and, in my view, unsurmountable. Instead of a forceful unification, a theory adequate to the structure of human experience should acknowledge and further elaborate this duality of times. How this can be done—this would be a subject for another essay. Here a few scarce remarks must suffice:

One-dimensional time is not an untouchable dogma; it is only a special feature of the arithmetic *model of time* employed in physics and public chronometry. We may think of the two times, biographic time and experienced time, not as two scales imposed on the same dimension but as two orthogonal dimensions (**Figure 1**). The *co-presence* of events within the horizon of actual presence is represented by the vertical, "depth" dimension, while the order of successive but essentially discontinuous episodes is represented by the horizontal dimension. It is only *post hoc*, in chronometric reconstructions, that both dimensions are collapsed and aligned with the continuum of the physical world time. Unlike popular metaphors of "passage of time," "time flow" or point-like "now" sliding along the time axis, our picture suggests a different kinetic metaphor: a transversal wave in a stationary medium, where the "local motions" of the medium are not identical with the "global motion" (i. e., wave propagation) but orthogonal to it.

## **REFERENCES**


Woodrow, H. (1951). "Time perception," in *Handbook of Experimental Psychology*, ed S. S. Stevens (New York, NY: Wiley), 1224–1236.

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 April 2014; accepted: 11 May 2014; published online: 28 May 2014.*

*Citation: Wackermann J (2014) The long is not just a sum of the shorts: on time experienced and other times. Front. Psychol. 5:516. doi: 10.3389/fpsyg.2014.00516 This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Wackermann. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Heterogeneous timescales are spatially represented

#### *Mario Bonato1 \* and Carlo Umiltà2*

<sup>1</sup> Department of Experimental Psychology, Ghent University, Ghent, Belgium

<sup>2</sup> Department of General Psychology, University of Padova, Padua, Italy

\*Correspondence: mario.bonato@ugent.be

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

### *Reviewed by:*

Jiri Wackermann, Institute for Frontier Areas of Psychology and Mental Health, Germany

**Keywords: time, spatial cognition, SNARC effect, mental representations, mental time travel, time perception, mental time line**

There is little doubt that cognitive processes involved in perceiving and comparing the duration of the sounds played by a musical instrument are very different from those involved, for instance, in recalling when our latest holidays took place. However, despite their clear difference, we ascribe both processes to the time domain, either in the form of a duration or of a mental time travel. Here we maintain that, despite the several meanings of "time" and its intrinsically heterogeneous nature, a common and somehow surprising characteristic of its representation is its spatial nature. Evidence will be provided suggesting that sensory time (the sounds of the instrument), time travel (last holidays) as well as conceptual time are all spatially represented. Whether this spatial format is the same for all "categories" of time is still unclear, as well as it is unclear whether category-specific spatial layouts exist. The possibility that several heterogeneous aspects of time might be all processed in spatial terms allows to better understand a number of apparently disconnected findings within the vast time domain. We will describe the prominent role played by writing habits in giving a direction to this representation. The possibility will be discussed that a more general tendency exists to represent in a spatial format all ordered sequences, as well as abstract concepts.

The interest on the way humans process and perceive time can be readily explained by the ubiquity of time in human life and by its multifaceted and sometimes elusive characteristics. Moreover, time processing is relevant for several domains, including not only philosophy, linguistics, cognitive science, and neuroscience, but also anthropology, history, and biology. Humans are equipped with a device (or several devices, this does not seem to be currently known) allowing them to deal effectively with all aspects of time, including duration estimates, time travel, and time conceptualization. A key empirical finding supporting our purposes here is that, across a number of formats and paradigms, time processing interacts with spatial processing and with the mechanisms subtending the deployment of spatial attention. This allows to consider the way humans perceive different time durations, and represent heterogeneous time concepts, as spatial in nature. A thorough review on these aspects can be found in Bonato et al. (2012a), where a number of analogies with the spatial representation within the numerical domain -which is also characterized by ordinality- are discussed. The more consistent line of evidence supporting the possibility that time is spatially represented comes from those studies showing an interaction between a time-related characteristic and the response side. Regardless of the aspect of time that is under investigation (perceptual or representational), the typical experimental manipulation requires the use of two stimulus-response mappings, whereby the same time-related stimulus receives a left response in one mapping and a right response in the other. By using two temporal durations Vallesi et al. (2008, exp 1) have shown consistent associations between left side and shorter durations (1 s) and between right side and long durations (3 s). The interaction does not depend on the responding hand but on the side of space where response is performed (exp 3 and 4) and was still present when more durations (six, from 0.5 to 3.5 s) had to be compared (exp 5; see Conson et al., 2008 for similar findings with auditory stimuli). An interaction with lateralized response keys is also found when the to-be-judged temporal references are related to mental time travel/time concepts, (e.g., beforeafter). These studies typically adopt much longer temporal durations. It is the case, for instance, of the months of the year, whereby faster responses are found when one of the first months (e.g., February) has to be responded to with a left-sided response and one of the last months (e.g., January) with a right-sided response (Gevers et al., 2003). Evidence for the automaticity of this association stems from the fact that it was found also when the task did not involve explicit access to the time aspect (Gevers et al., 2003). Leftright associations can be elicited also when the timeframe in a temporal comparison task extends to several years, as shown by Weger and Pratt (2008) who asked for a comparison of actors who were famous several decades ago (e.g., Charlie Chaplin) vs. those who were famous at the time of testing (e.g., Brad Pitt). Ishihara et al. (2008) provided evidence that left-before vs. right-after associations are present also for brief, sensory stimuli (sounds with a 20 ms duration). In their study, the time of appearance of a target sound after a sequence of seven equally spaced (500 ms interval) sounds had to be compared with the temporal lag of half a second dividing the preceding sounds. Participants showed a preference for right-sided responses when the target sound was presented later in time with respect to the expected interval (delay of 215 ms) and a preference for left-sided responses for a target sound presented 215 ms before the expected timing. The effect disappeared when vertically arranged response keys were adopted. Crucially, a clear left/before right/after association is also found when arbitrary, non-temporal sequences are used (Previtali et al., 2010). In this latter case the temporal aspect is somehow implicit and related to the order of stimuli presentation (time range: several seconds). The association of left responses with past and of right responses with future still persists when past- related vs. future-related tenses and words are presented (Santiago et al., 2007).

An interaction with response side can be also found when merging together (very) different temporal lengths, ranging from a few seconds to several decades (Fuhrman and Boroditsky, 2010). In this study participants had to determine, by means of two lateralized response keys, whether the second of two sequentially presented pictures referred to an event occurring earlier/later in time than the first picture presented. Images represented either brief sequences (secondsminutes: a banana turning into an empty peel) or long sequences (decades: a boy turning into an adult). English speakers showed a clear left/before right/after preference in 36 out of 38 sequences. By directly comparing the performance of English-speaking people to Hebrew speakers, Fuhrman and Boroditsky (2010) showed that the effects of preferential spatial arrangements with lateralized keys is related to writing directions (see also the seminal study by Tversky et al., 1991). A similar effect of writing habits can be found when temporal durations (1 vs. 3 s) have to be compared (Vallesi et al., in press), once again confirming the commonalities in representing durations and time order. In other words, time representation(s) can be dramatically influenced by cultural factors (Núñez and Cooperrider, 2013). Populations that are not very familiar with reading and writing show a number of different spatial layouts for time (Boroditsky and Gaby, 2010). However, also in Western cultures changes in the postural sway according to a past/backwards and future/forward association has been described (Miles et al., 2010).

In summary, a spatial component (lateralized responses) has been described to be robustly associated with strikingly different time intervals, either with more perceptual or with more conceptual tasks. This suggests that very different time aspects can be all spatially represented. One can make the analogy even stronger by pointing out that this supra-modality of time processing resembles the similarities found for perceived vs. represented spatial locations.

It can be claimed that the ubiquity of these effects can be task-induced rather than reflect a truly spontaneous spatial representation. A line of evidence allowing one to confirm the genuinely spatial origin of time representation can be found in those cases where spatial attention is distorted as a consequence of brain damage. Damage affecting right-hemisphere areas related to spatial processing has been described to induce a spatial bias in a detection task where the cue was a timerelated, centrally presented, word (Pun et al., 2010). Moreover, two recent studies with right hemisphere damaged patients have described temporal distortions in the presence of neglect, a neuropsychological syndrome hampering the processing and the internal representation of contralesional space. These findings provide further support for the spatial nature of time representation for both mental time travel (Saj et al., 2014) and temporal durations (Oliveri et al., 2013).

The idea behind Saj et al. (2014) is similar to the rationale of the study by Zorzi et al. (2002), who suggested that the overestimation found in patients with left neglect in a numerical bisection task resembled the length effect neglect patients show in line bisection tasks. In the Saj et al. (2014) study right brain damaged patients with and without spatial neglect encoded behavioral habits an imaginary person used to have in the past (10 years before) or will have in the future (in 10 years). Patients with left hemispatial neglect showed a specific deficit when remembering and attributing items to the past, whereas this "distortion" was absent in right brain damaged patients without neglect and in healthy controls. By using prismatic adaptation Oliveri et al. (2013) crucially demonstrated that the disturbances in the time domain shown by neglect patients result from their impaired spatial processing rather than by other, more general, cognitive impairments (Bonato et al., 2012b). In right brain damaged patients with neglect a leftward attentional deviation reduced the -overall more severe than in patients without neglect- time underestimation deficit. An experimental manipulation (e.g., prismatic adaptation) affecting spatial processing thus influenced also a temporal bisection task. Effects of spatial attention distortions upon time aspects can be found in healthy participants as well (see for instance Vicario et al., 2007; Di Bono et al., 2012).

Which are the cognitive mechanisms subtending time-space interactions and their distortions ? Within the numerical domain Priftis et al. (2006) maintained that the distortions due to neglect in numerical processing reflect impaired access rather than a distorted representation itself. Supporting evidence can be found in Zorzi et al. (2012), where spatio-numerical deficits shown by neglect patients varied according to the task at hand (present in magnitude comparison but absent in parity judgement). Leaving terminological issues aside, the possibility of a deficit in accessing an intact representation does not seem at odds with those studies that have highlighted that working memory for sequences is characterized by a spatial layout. It might then be reasonable to assume that most time-space associations are based on flexible, short-term associations made on the spot according to the task at hand. In turn, this view would be compatible with the findings of van Dijck and Fias (2011), who have described a left to right spatial mapping for ordered items in working memory. This mapping influences not only lateralized responses but also target detection tasks (van Dijck et al., 2013).

Finally, if we reason that perceptual time is in a way represented and conceptualized before response, it seems possible that an even broader tendency to spatially represent all abstract concepts exists. For instance, Chasteen et al. (2010) have shown, through a target detection task, that the concepts of God and Devil produce shifts of attention and lead to faster responses when visual targets are presented at compatible locations with the concepts of God (up/right locations) or Devil (down/left locations). They framed their findings as supporting the link between internal spatial representations and locations where attention is allocated in the external environment. The tendency to represent rather abstract concepts in a spatial format might further extend to a number of representations of the self, in addition to spatial and temporal frames of reference (Parkinson et al., 2014).

### **ACKNOWLEDGMENT**

Mario Bonato is supported by a FWO Pegasus Marie Curie fellowship.

### **REFERENCES**


spatially organized. *Cognition* 87, B87–B95. doi: 10.1016/S0010-0277(02)00234-2


codes: understanding the cognitive representations of time. *Cognition* 107, 501–527. doi: 10.1016/j.cognition.2007.10.011


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 05 May 2014; accepted: 15 May 2014; published online: 03 June 2014.*

*Citation: Bonato M and Umiltà C (2014) Heterogeneous timescales are spatially represented. Front. Psychol. 5:542. doi: 10.3389/fpsyg.2014.00542*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Bonato and Umiltà. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

#### *Richard A. Block1 \* and Simon Grondin2*

<sup>1</sup> Department of Psychology, Montana State University, Bozeman, MT, USA

<sup>2</sup> École de psychologie, Université Laval, Québec City, QC, Canada

\*Correspondence: block@montana.edu

### *Edited by:*

Claire Zedelius, University of California, Santa Barbara, USA

*Reviewed by:* Hedderik Van Rijn, University of Groningen, Netherlands

**Keywords: time perception, attention, animals, humans, commentary**

### **A commentary on**

### **Perception and estimation of time**

*by Fraisse, P. (1984). Annu. Rev. Psychol. 35, 1–36. doi: 10.1146/annurev.ps.35.020184. 000245*

**Properties of the internal clock: first- and second-order principles of subjective time** *by Allman, M. J., Teki, S., Griffiths, T. D., and Meck, W. H. (2014). Annu. Rev. Psychol. 65, 743–771. doi: 10.1146/ annurev-psych-010213-115117*

A clear example of the progress in the field of timing and time perception could be obtained by contrasting two articles published 30 years apart in the influential *Annual Review of Psychology* (ARP): one by Fraisse (1984), and one by Allman et al. (2014). The fact that there was one author 30 years ago, and a group of authors now, is a tangible sign of the contemporary way of approaching scientific research. In his review, Fraisse emphasized the distinction between time perception and time estimation; in their review, Allman et al. focused on the internal clock and the cerebral bases of timing and time perception.

Fraisse's review was published when a very important event happened in the field of timing and time perception: a conference was held in New York, in 1983, where researchers from both human and animal time perception met to communicate with one another. The conference led to the publication of the classical book edited by the late John Gibbon and the late Lorraine Allan (Gibbon and Allan, 1984). This meeting probably catalyzed the research on timing and time perception, especially the one emphasizing the scalar expectancy theory and, more generally speaking, the internal clock perspective, a clock described as a pacemaker-counter device.

It is somewhat surprising that there was no mention in Fraisse (1984) of this promising (to say the least) pacemakercounter perspective, which was already available in the human timing literature (Creelman, 1962; Treisman, 1963). Moreover, the modest portions of information in Fraisse dedicated to the cerebral bases of timing exemplify the gap between the contemporary research in the field and the state of the literature 30 years ago.

With its emphasis on neuroscience literature (e.g., brain areas, cortical circuits, pharmacological effects, and pathologies), Allman et al. wrote an important, wellstructured, and interesting state-of-the-art review on the cerebral bases of the time perception mechanisms. It is a bit surprising though that the scalar property is taken for granted, given actually Fraisse's fundamental distinction between time perception and time estimation, a distinction that could find some echoes in the limitation of the stability of the Weber fraction for time (see Figure 3 in Gibbon et al., 1997; or, for instance, Grondin, 2001, 2010b, 2012, 2015). Moreover, assuming the linearity between psychological and physical time (psychophysical law) remains disputable (Eisler, 1976).

By emphasizing the internal clock perspective, it was not possible for Allman et al. (2014) to refer to other recent developments in the field. Amongst the portions of the literature the reader might want to consider, there is one on retrospective timing (Block and Zakay, 1997; Tobin et al., 2010). There is also some interesting research (e.g., Boltz, 1998; Brown, 2008) offering a purely cognitive explanation of psychological time and timing without reference to an internal clock (see reviews by Block et al., 1999, 2010; Block, 2003). Even within the perspective of an internal clock, the attentional-gate model (see for example, Zakay and Block, 1995 and later articles), which in an extension of the scalar expectancy theory, is worth mentioning.

Indeed, with the large increase of research in the field of timing and time perception in the Twenty-first century, it is not surprising to see so many recent special issues of journals on this topic, or close variants of them. The explosion is such that researchers have written a large number of recent review articles (see **Table 1**). This was partly described in an annotated bibliography on "Time Perception" (Block and Hancock, 2013). Another tangible sign of the vitality of this research field is exemplified by a large COST grant funded by the E.U. (title: "Time In MEntaL activitY," or "TIMELY") and the resulting founding of the Brill's new scientific journal dedicated to the psychology of time, Timing and Time Perception, co-edited by Meck et al.

In conclusion, being a researcher in the field of timing and time perception has never been as exciting as it is at present, given the growth of its popularity, which has been enhanced by the arrival of contributions from neuroscientists. This

### **Table 1 | Selected list (in reverse chronological order) of reviews since 2010 on the psychology of time.**


Book is an edited book. Rev is a review article. SI is a special issue. Bib is a bibliography.

excitement could be extended if one considers psychological time in an even larger perspective, or larger scale from the memory for the past events (Friedman, 1993) to the capacity to predict the duration of future events (Roy et al., 2005).

### **ACKNOWLEDGMENT**

We thank Keith Hutchison and an anonymous reviewer for their very helpful suggestions on drafts of this commentary.

### **REFERENCES**


hints on future research. *J. Physiol.* 105, 153–163. doi: 10.1016/j.jphysparis.2011.09.008


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 02 May 2014; accepted: 06 June 2014; published online: 29 July 2014.*

*Citation: Block RA and Grondin S (2014) Timing and time perception: A selective review and commentary on recent reviews. Front. Psychol. 5:648. doi: 10.3389/fpsyg. 2014.00648*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Block and Grondin. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Attention and working memory: two basic mechanisms for constructing temporal experiences

## *Giorgio Marchetti\**

Mind, Consciousness, and Language Research Net, Milan, Italy

### *Edited by:*

James M. Broadway, University of California at Santa Barbara, USA

### *Reviewed by:*

Giulio Benedetti, Formerly affiliated with Società di Cultura Metodologico-Operativa, Italy James M. Broadway, University of California at Santa Barbara, USA

### *\*Correspondence:*

Giorgio Marchetti, Mind, Consciousness, and Language Research Net, via Guido Rossa 8, Pozzo d'Adda, Milan, Italy e-mail: info@mind-consciousnesslanguage.com

## **INTRODUCTION**

Tulving (1972) noted that one of the most fascinating achievements of the human mind is its ability to mentally travel through time. We can relive experiences by thinking back to situations that occurred in the past; likewise, we can mentally project ourselves into the anticipated future through our imagination, daydreams and fantasies.

Time travel is just one of the main *forms* that consciousness can assume. As phenomenologists have shown (Gallagher and Zahavi, 2008), there are various modes of givenness of objects and events, that is, ways in which objects and events appear to us. It is possible for one and the same object or event to appear in a variety of other different ways: from this or that perspective, linguistically represented, dreamt, imagined, perceived, wished for, or feared.

Each of these forms of consciousness possesses its own phenomenal quality: perceiving an object feels different from imagining the same object, which in turn feels different from remembering it. Despite the phenomenal differences, it can be shown that most of these various forms of consciousness depend on *constructive* processes based on some common mechanisms. This is suggested for example by the observation that they can be "added" to every conscious content, in the sense that the same conscious content can be experienced in one or the other form of consciousness by means of just mentally processing it through one or the other form of consciousness (for example, we can experience the same event as occurring either in the past or future by means of just mentally adding to it the temporal dimension of past or future, respectively, Nyberg et al., 2010).

In this article I will provide psychological and neurophysiological data to substantiate the idea that some of the most important forms of consciousness – episodic memory, episodic future thought, perception, language, and conscious thinking – are based on constructive processes based on two common mechanisms:

Various kinds of observations show that the ability of human beings to both consciously relive past events – episodic memory – and conceive future events, entails an active process of construction. This construction process also underpins many other important aspects of conscious human life, such as perceptions, language, and conscious thinking. This article provides an explanation of what makes the constructive process possible and how it works. The process mainly relies on attentional activity, which has a discrete and periodic nature, and working memory, which allows for the combination of discrete attentional operations. An explanation is also provided of how past and future events are constructed.

**Keywords: mental time travel, language, thought, attention, working memory, duration, past, future**

attention and working memory. The sections *Time travel* and *Perception, language and conscious thinking* will provide evidence supporting the general idea that these forms of consciousness are the result of an active process of construction performed by the subject. In the section *The underlying mechanisms of constructive processes: attention and working memory* I will try to show that the constructive processes underpinning the various forms of consciousness, rely on two common and fundamental mechanisms: attention and working memory. Finally, the section*Variously using attention and working memory yields various construction processes* will show how constructive processes can yield various forms of consciousness in general, and specifically how the conscious experience of temporality is constructed.

## **TIME TRAVEL EPISODIC MEMORY**

Episodic memory – the ability to consciously relive personal past events (Tulving, 1985) – is an active process of construction, rather than a faithful re-enactment of the past (Rosenfield, 1988; Suddendorf et al., 2009). Various kinds of observations show this.

As Schacter (1999) has shown, memory generally suffers from different types of "sins," which can be classified into three main categories. The first category involves forgetting: memory of facts and events typically becomes less accessible over time, even when we deliberately search our memory in an attempt to recall a specific fact or event; memory of facts and events is affected by attention: when insufficient attention is paid to a stimulus at the time of encoding or retrieval, forgetting is likely to occur, as the phenomenon of change blindness shows (Simons and Levin, 1997); even when a fact or event has been deeply encoded, and has not been lost over time, it may occasionally be temporarily inaccessible (one of the most common and frequent instances is

the tip-of-the-tongue state). The second category involves distortion or inaccuracy: even if people may remember correctly a fact from a past experience, they can misattribute it to a wrong source; suggestions or misleading questions that are made when one is attempting to recall an experience, may alter the recollection of the original event; memories can be biased and distorted by expectations, beliefs, current knowledge, etc. The third category refers to pathological remembrances: facts and events, such as traumatic events, that we cannot forget even if we wish we could. While, on the one hand, these "sins" demonstrate the occurrence of memory distortions, on the other hand, they clearly support the idea that memory is not so much a literal replay of the past as a complex process of assembling and integrating several bits of information.

Moreover, as Tulving (1972) already noted, different memory systems differ in their susceptibility to transform and lose stored information, which further complicates the picture drawn by Schachter. For example, forgetting appears to be more readily produced in episodic memory than in semantic memory. Since information in the former is always temporally dated, and can only be retrieved if the retrieval cue accurately specifies its temporal date, interference with temporal coding may render access to it difficult or impossible. On the contrary, information in semantic memory, being usually encoded as part of a rich multidimensional structure of concepts and their relations, is better protected by such embeddedness from interference by other inputs.

The phenomenon of déjà-vu, in which we have the experience of reliving a past event in the absence of an actual memory, shows that the sense of "pastness" of the event is not inherent in the memory itself and that it depends on a process of active construction.

Conversely, merely using a memory representation of a prior event is not sufficient to ensure the subjective experience of remembering it: as Suddendorf et al. (2009) notice, semantic memory allows us to know where and when we were born but this does not suffice to consciously relive that moment.

### **EPISODIC FUTURE THOUGHT**

Episodic future thought is the ability to simulate specific personal episodes that may potentially occur in the future (Szpunar, 2010). In many cases, it is patent that episodic future thought involves a process of active construction of events that have not yet occurred, such as when future thought depends on a (novel) recombination of episodic details (whether of perceptual or imaginal source) into a hypothetical event. This is what experiments performed using the word-cueing paradigm (Galton, 1880; Crovitz and Schiffman, 1974) generally reveal: when participants are given a word cue and are asked to use it to mentally generate personal future details, they tend to imagine themselves in the context of familiar settings and people (D'Argembeau and Van der Linden, 2004).

It should be noted, however, that not always doesfuture thought depend on a recombination of episodic details. Sometimes, future thoughts and episodes may be constructed without the need to necessarily rely on the contents of specific episodic memory *per se*: a general, abstract knowledge of the context, environment or situation is sufficient to imagine how things can evolve in future (Szpunar, 2010).

The constructive process involved in episodic future thought is highlighted by experiments such as Nyberg et al.'s (2010). Nyberg et al. (2010) scanned well-trained subjects using fMRI during experimental tasks that require the capacity to be aware of subjective time (chronesthesia), such as remembering a recent short walk along a familiar route or imagining a future short walk along the same route. Brain activity during these tasks was compared with activity during a matched task that does not require chronesthesia: the subjects were instructed to take a mental walk through the same route in the present moment, without any thoughts about specific personal past or future happenings. By contrasting the remembering and imagining tasks with the mental walk task, Nyberg et al. (2010) could measure brain activity correlated with "pure" conscious states of different moments of subjective time. They reported that the left lateral parietal cortex, as well as the left frontal cortex, cerebellum, and thalamus were preferentially engaged as the subjects thought about taking walks in the past or future as compared to taking the same walk in the present moment.

This study, while revealing which brain regions are specifically related to the capacity to be aware of subjective time, also clearly shows that an event (such as taking a short walk from a certain point to another) can alternatively be experienced as occurring in the past or future by means of just mentally adding to it the temporal dimension of past or future, respectively. That is, an event that is otherwise lived as present may become a past or future event if we actively construct it as such, by mentally adding to it a specific past or future temporal quality. (Incidentally, it should be noted that I use the term "construction" here in a wider sense than; Nyberg et al., 2010. While they use it to refer to the sole process of constructing the scene of walking, my usage also includes the process of adding a temporal dimension, and hence of constructing the subjective experience of time).

That subjective time requires to be mentally constructed in order to be consciously experienced, is further evidenced by all those processes that, despite dealing with future events, can take place independently from being experienced as occurring in time. For example, Kwan et al.'s (2012) study shows a clear dissociation between imagining and knowing about the future. K. C., a person with episodic amnesia and unable to imagine future experiences, can still value future reward and make a decision about the future in a way comparable to healthy subjects, despite being unable to construct the details of either past or future events. Therefore, the ability to make decisions about the future is dissociable from the ability to imagine one's possible future. It should be noted that the decision-making strategy of healthy individuals – like those of the controls in Kwan et al.'s (2012) study – usually involves future-oriented imagery (see also Peters and Büchel, 2010, who show how the ability to imagine one's possible future modulates future decision-making). This is additional evidence that humans can add a temporal dimension to a process that otherwise occurs outside of any temporal experience.

Another case in which the capacity to deal with future events does not necessarily depend on the capacity to simulate or imagine future events is anticipatory behavior. As observed by Suddendorf and Corballis (1997), instincts, such as hibernation, provide a mechanism for dealing with environmental changes without the need for the organism to actually imagine the future (hibernators prepare for winter even if they have not experienced that season before). This parallels the distinction between episodic and other memory systems, which explains how the past can influence behavior without necessarily involving mental time travel into the past.

To summarize, both episodic memory and episodic future thought are best conceived as constructive processes, rather than literal records of the past or a projection in the future of carefully represented episodic memories, respectively. This common characteristic is further highlighted by research findings from neuroimaging, neuropsychology, and clinical psychology that have shown a close relationship between episodic memory and episodic future thought: (a) neural regions believed to underlie the retrieval of personal memories are similarly engaged by episodic future thoughts; (b) damage to these regions is associated with impairments of both remembering and episodic future thought; (c) patients characterized by poor episodic memory exhibit a concurrent inability to imagine their future in a vivid way (for a review, see Szpunar, 2010; however, it should be noted that differences between the two processes exist as well: see Schacter et al., 2012).

### **PERCEPTION, LANGUAGE, AND CONSCIOUS THINKING**

Episodic memory and episodic future thought are not the only forms of human consciousness based on an active construction process. Construction processes also underlie many other important forms of human conscious life, such as perceptions, language, and conscious thinking.

### **PERCEPTION**

Just as memories are not a faithful re-enactment of the past, so too perceptions are not a faithful representation of a world specified prior to, and independently of, the activity of the subject. Daily experiences and perceptual illusions show how situational and contextual factors, as well as expectations, education and emotions, shape our perceptions. If we consider the perception of space alone, we see that different sense-organs generate different spatial perceptions of the same object: a piece of rosemary stuck in a tooth feels enormous until it is felt with your fingers. Likewise, emotions change our perception of space: in despair, for example, there is nowhere to go, the sky closes in, actions seem pointless, and the ordinary depths of the world are transformed (Morris, 2004). The perception of the dimension of the same object changes in relation to the different context in which it is placed: a piece of furniture may seem smaller when seen in a shop than when seen inside our house. Finally, the perception of space also changes with age: what seemed to be big, large and high in size when we were children (for example, the house in which we were born) may appear to be small, narrow and low as adults.

These examples clearly show that perceptions are not a passive duplication of a ready-made world, but the result of an active process of construction performed by the subject. Evidence of this process of construction is provided by various kinds of findings. Neural data show that the complex representations generated by the visual system are built out of distinct streams of processing: an occipitotemporal ventral stream in which cells are sensitive to information pertaining to the identity of objects and an occipitoparietal dorsal stream in which cells are sensitive to spatial information (Ungerleider and Mishkin, 1982). Within each of these streams, information diverges further: there are distinct streams for processing color, shape and motion, and multiple representations of space within the posterior parietal cortex. Moreover, activity in the visual system can be modulated by attention (Treue, 2001; Boynton, 2005; Carrasco, 2011), which allows the organism to adequately cope with contextually and behaviorally relevant information.

Psychological phenomena such as the continuous wagon-wheel illusion (VanRullen and Koch, 2003; Simpson et al., 2005; Van-Rullen et al., 2005, 2006), perceived causality (Shallice, 1964) and apparent simultaneity (Hirsh and Sherrick, 1961) and neurophysiological observations (Fingelkurts et al., 2010) reveal that our experience of the surrounding world as a continuous, seamless flow of information, is actually the result of the combination and assembly of distinct processing epochs. Studies on the phenomenology of temporal perception of events (Pöppel, 1997, 2004; Wittmann, 2011) show that one can identify at least three different levels at which successive events are fused to form distinct subjective experiences, each possessing its own specific qualitative characteristics. According to Wittmann (2011), on a first, basic level there is the "functional moment," an elementary temporal building block of perception in the range of milliseconds, which has no perceivable duration because individual events are processed as co-temporal and the temporal order of events is not detected. On a second level, successive functional moments are grouped on a time scale of up to around 3 s, yielding the "experienced moment," where events are perceived as occurring in an extended now. Within the experienced moment, successive events are strongly and orderly bound together: when listening to a metronome at moderate speed, we do not hear so much a train of individual beats, as perceptual gestalts having an accent on every *n*th beat, such as "1–2, 1–2" or "1–2–3, 1–2–3." If, on the contrary, the metronome is too fast, we experience a fast train of beats that does not contain any temporally ordered structure of distinct events. Likewise, if the metronome is too slow, we perceive only individual beats which are not related to each other. A third level of integration exceeding about 3 s leads to "mental presence," a sequence of experienced moments enclosed within the temporal window of a unified experience of presence, which enables the continuous awareness of oneself as presently perceiving and acting within an environment (Fingelkurts and Fingelkurts, 2014). In sum, whereas the duration of the functional moment is not perceived at all, an experienced moment is perceived as happening now, for a short but extended moment. On the contrary, mental presence involves the experience of a perceiving and feeling agent within a window of extended present.

Despite the fact that we experience the world surrounding us as a continuous, seamless flow of information, as we do when watching a movie, we actually extract and process information in distinct moments, similar to the snapshots of a camera. This observation is supported by empirical findings. For example, Latour (1967) found that the visual threshold for detecting a flash of light varied periodically in the few milliseconds preceding the onset of an eye saccade, and that the visual threshold for detecting two flashes displayed successively varied periodically as a function of the time interval between them. This data can be interpreted as evidence of periodical processing on the grounds that the probability of detecting a brief stimulus varies as a function of the state of the process: during a "no-processing" state, the detection rate of the stimulus decreases, whereas it increases during the "processing" state. Periodicities were also observed in reaction time distributions (Venables, 1960; Dehaene, 1993), which can be explained in the following way: a stimulus with an onset occurring during a "no processing" state must wait until the next processing state in order to be processed, which leads to a longer reaction time.

In a face identification task, Blais et al. (2013) modulated the signal-to-noise ratio of faces through time, so that at some moments visual information was available, whereas at other moments no information was available, and analyzed how different temporal profiles of signal-to-noise ratio impacted face identification performance. The aim was to test the hypothesis that visual information would be sampled periodically. The underlying assumption was that if the information useful for the task is available at the right moment, the participant is more likely to respond correctly. In contrast, if the information useful for the task is not available at the right moment during the processing, then the participant is less likely to respond correctly. The results of Blais et al.'s (2013) experiments show the existence of a discrete sampling of visual processing, operating at a rate of about 10–15 Hz. Moreover, their findings support the view that this periodical mechanism synchronizes with the visual stimulation.

Data from electrophysiological recordings show that electrical neural oscillations could provide the physiological basis of periodic perceptual phenomena. Varela et al. (1981) found a close correlation between the perception of apparent simultaneity and the alpha phase at which stimuli are presented: two flashes of light that always have the same stimulus onset asynchrony are judged to be simultaneous when presented at one particular phase, but sequential when presented at the opposite phase. Busch et al. (2009) and Mathewson et al. (2009) show that the phase of ongoing oscillations influence whether a stimulus is perceived at all, which indicates that the visual detection threshold is not constant over time but fluctuates along with the phase of spontaneous electroencephalogram (EEG) oscillations. Drewes and VanRullen (2011) show that the prestimulus oscillatory phase modulates human saccadic reaction time. Investigating the role of prestimulus phase coupling on visual perception in an attentional blink paradigm, Kranczioch et al. (2007) found that low levels of prestimulus alpha phase coupling predict correct perception of the second target stimulus, whereas high levels of prestimulus alpha phase coupling predict a miss of the second target stimulus. Doesburg et al. (2009) showed that perceptual switching during binocular rivalry is time-locked to gamma-band synchronizations which recur at a theta rate, indicating that the onset of new conscious percepts coincides with the emergence of a new

gamma-synchronous assembly that is locked to an ongoing theta rhythm. Doesburg et al. (2009, ibid., p. 2) infer that "only one truly discrete perceptual experience may exist within a single theta cycle, and that the emergence of new perceptual experiences may be time locked to a particular phase of ongoing cortical theta rhythms."

There is evidence that the amplitude of prestimulus oscillations in the alpha range also significantly affects the perceptual outcome. Van Dijk et al. (2008) found that contrast-discrimination ability is modulated by prestimulus alpha power: an increase in posterior alpha power correlates with a decrease in discrimination ability. By directly stimulating visual area via short transcranial magnetic stimulation (TMS), Romei et al. (2010) tested whether oscillation in the alpha band causally shapes perception, relative to control stimulations in the theta and beta bands: they found that occipital and parietal TMS at alpha frequency impairs target visibility in the visual field contralateral to the stimulated hemisphere and enhanced it ipsilaterally.

It still remains to be defined which parameter of neural oscillations – amplitude, phase consistency, or phase coupling – predicts periodicity in perception better than others (Hanslmayr et al., 2011). Likewise, it still remains unclear why different frequencies correlate with different periodic perceptual phenomena. Various hypotheses could be supported. For example, one can think that the sampling frequency could vary as a function of the kind of stimulation, synchronizing with stimulation, or that it could evolve without synchronizing with the external world, as a passive ongoing, random oscillation (Blais et al., 2013). Despite all these open questions, however, the bulk of current studies clearly points to the idea of a discrete sampling of perceptual processing.

### **LANGUAGE AND CONSCIOUS THINKING**

There is no doubt that the human capacities that most apparently involve an active construction process are language and conscious thinking. Language allows us to combine the single words and thus convey vastly, literally infinite new meanings and conscious experiences. The power of language as a unique and specialized tool in connecting objects and events is exemplified by the strong connection between language and the ability to encode and represent the order of discrete elements occurring in a sequence (sequential learning; Conway and Christiansen, 2001). Likewise, conscious thinking develops and evolves thanks to the combination of ideas, concepts, images, memories etc. As Baumeister and Masicampo (2010, p. 956) observe: "conscious thinking and speech involve a process of actively combining concepts to make something that may have additional, unforeseen, newly emergent properties."

As extensively showed by the works of linguists, logicians, and philosophers, the combinatorial property of language and conscious thinking relies on some specific relational units, whose function is to tie together two or more semantic elements, be they simple words, other relational units, complex thoughts, or else. Scholars have variously identified and termed these relational units. Sapir (1921) named them "relational concepts" and classified them as concrete and pure. Ceccato (1972; see also Ceccato and Zonta, 1980) termed them "correlators." Linguistically, correlators are designated by prepositions, conjunctions, cases (genitive, dative, etc.) and the implicit correlator (Benedetti, 2011). Other examples of correlators are the mathematical and logical operators. As Benedetti's(2009, 2011) in-depth analysis shows, correlators are themselves constituted by sequences of elemental mental operations (among which those of attention play a key role) that are assembled and chunked together by means of working memory and procedural memory. The assembling and chunking processes are supposed to be supported by analogous processes at the neurophysiological level within the nested hierarchy of brain operational architecture (Benedetti et al., 2010; Fingelkurts et al., 2010, 2013).

The constructive character of language is further evidenced by its intrinsic periodicities. Speech is not produced in a continuous, uninterrupted flow but in spurts. Spurts reflect not only a biological necessity (that is, the speaker's need to replace the air that he expels from his lungs when he produces speech sounds) but also the basic functional segmentations of discourse. Chafe (1994) refers to these segments of language as "intonation units." Intonation units, whose size averages one to four words, can be identified on the basis of a variety of criteria, among which are pauses or breaks in timing, acceleration and deceleration, changes in overall pitch level, etc. Each intonation unit "verbalizes a small amount of information which, it is plausible to suppose, is part of the speaker's model of reality on which his or her consciousness is focused at that moment. In a socially interactive situation it is the portion on which the speaker intends that the listener's consciousness be focused as a result of hearing the intonation unit. This limited activation allows a person to interact with the surrounding world in a maximally productive way, for it would hardly be useful to activate everything a person knew at once" (Chafe, 1994, ibid., p. 29).

Similarly, Duncan (2013) explains the combinatorial nature of human thought and, more in general, of cognition as the most appropriate and adaptive answer to the complex problems posed by the environment. It allows us to flexibly address very complex problems through the solution of simpler sub-problems. As made clear by Artificial Intelligence studies, the most effective way of solving complex problems is by decomposing them into simpler components (see also Fingelkurts et al., 2012). If all the aspects of a problem were considered at once, the search space of possible alternative solutions would simply be too large and unconstrained, yielding too many concurrent, suboptimal choices. "Effective cognition requires a series of selections from this space, each defining a subproblem of relevant inputs, actions, and potential achievements. Often these will be organized hierarchically, so that each subgoal or task is divided further into subgoals of its own" (Duncan, 2013, ibid., p. 36).

## **THE UNDERLYING MECHANISMS OF CONSTRUCTIVE PROCESSES: ATTENTION AND WORKING MEMORY**

As we have seen, disparate forms of consciousness such as time travel, perception, language, and conscious thinking are all based on an active construction process. As revealed by research reviewed in the previous sections, this construction process is implemented at various levels and underpinned by various mechanisms. Given the commonality and close relations between these forms of consciousness (language influences perception and *vice versa*, conscious thinking and language show extensive commonalities, mental time travel is heavily based on perceived objects and events), it is legitimate to ask whether the various forms of consciousness are underpinned by a common construction process or, at least, whether the constructive processes underpinning them are based on some common mechanisms.

My analysis (Marchetti, 2010) reveals that (1) different forms of consciousness are produced by different construction processes, and (2) two mechanisms implement the different construction processes that underpin most of the different forms of consciousness: these two mechanisms are attention and working memory. Attention ensures the selection of basic elements (or pieces of information); working memory ensures that the selected elements are maintained active during processing and assembled. Various forms of consciousness result from the different ways that attention and working memory operate, that is, from the different ways of selecting, maintaining and assembling basic elements.

Some form of attention is always necessary to produce consciousness (Bor and Seth, 2012; Marchetti, 2012), even though the former does not always produce conscious outcomes. Working memory, while being necessary for most of our conscious experiences, does not seem to be always necessary: very simple and basic perceptions, but also more complex forms of perception involving top-down attention control, do not require working memory. Kane et al. (2006) showed that even in contexts (such as command search task) in which subjects have to endogenously control visual attention by moving it strategically though search arrays, the high- and low- working memory subjects performed equivalently: that is, even visual search tasks that present minimal demands to actively maintain or update goal-relevant information, but which are still difficult and involve top-down attention control, are independent of working memory.

The combined working of attention and working memory is necessary to produce most of our conscious experiences. However, attention and working memory are not always sufficient: most forms of consciousness also need other components, such as long term memory, semantic memory, sense-organs, and somatosensory organs, and what I call the schema of self (Marchetti, 2010).

Before examining how attention and working memory implement construction processes, I will firstly present evidence that attention operates in a periodic, pulse-like manner, thus providing a plausible explanation for the periodicities observed in perceptions and language, as well as for the selections performed in conscious thinking and time travel. Secondly, I will briefly consider the neurophysiological bases of working memory, and exemplify the role it plays in a form of consciousness, episodic future thought.

### **ATTENTION**

Psychologists have long studied attention as a mechanism capable of coping with the limits of our sensory, perceptual and memory systems in managing the flow of information with which we are constantly confronted. By allowing for the selection of the information, attention reduces the input to a manageable amount: it isolates and amplifies pieces of information, which can be variously combined to yield theoretically infinite chains of constructs. As observed by VanRullen et al. (2007), overt periodic sampling of the environment is a ubiquitous property of sensory systems (saccades in vision, sniffs in olfaction, whisker movements in rat somatosensation, and even electrolocation in the electric fish) and attention might have evolved from these periodic processes as a more economical means of covertly sampling endogenous representations.

Moreover, as argued by Duncan (2013, p. 36), attention proves to be an effective tool in dealing with the complex problems posed by the environment: in fact, it allows for the segmentation of the flow of information into "attentional episodes," each episode admitting into consideration only the contents of momentary, focused subproblems. More specifically, attentional discrete processing has various advantages from a purely computational point of view. According to Buschman and Miller (2010), restricting computations to discrete windows of time would: (a) ensure that informative spikes occur with the temporal precision that is both necessary for integration by downstream neurons and for spike-timing dependent plasticity; (b) act to stabilize and organize the neural network and its computations: periods of inhibition may act to "reset" the network to a base state, effectively limiting the number of states that neurons could obtain; (c) allow for easier coordination of processing within and between brain regions, by providing a specific moment at which information must be available for computation in a specific region, and at which the outcome of the computation is available.

Finally, the attentional selection process has the side effect of creating new experiential dimensions on top of the ones from which they originate. By selecting and combining otherwise unrelated elements, we can imagine and simulate new events, scenarios and conditions that we would have never consciously experienced if we had not been endowed with selective and constructive capacities. As observed by Baumeister and Masicampo (2010, p. 958), the full power of human consciousness consists in using the mental capacity for constructing sequential thoughts to conduct simulations during wakefulness, without relying on sensory input.

Although the working of attention can be theoretically conceived as an uninterrupted, continuous process, which rapidly switches between different targets, evidence seems to favor the hypothesis that attention operates in a periodic, pulse-like manner. VanRullen et al. (2007) found that attention, even when focused on a single target location, samples information periodically like a blinking spotlight. Moreover, by analyzing the correlation between detection performance for attended and unattended stimuli and the phase of ongoing EEG oscillations, Busch and VanRullen (2010) showed that detection performance for attended stimuli actually fluctuated over time along with the phase of spontaneous oscillations in the θ (≈7 Hz) frequency band just before stimulus onset. This fluctuation was absent for unattended stimuli. This pattern of results suggests that attention in fact exerts its facilitative effect on perception in a periodic fashion. The alpha phase plays a crucial role in the attentional blink phenomenon (Hanslmayr et al., 2011). Doesburg et al.'s (2008) findings support the view that gamma-band synchronization is the mechanism that implements the selective properties of attention, its integrative properties, and the special relationship between attention and consciousness. Landau and Fries' (2012) study shows that selective attention samples stimuli in a rhythmic way. Further evidence of the link between attention and periodic brain processes is provided by studies in which oscillatory brain responses are entrained by periodic stimulation (Jones et al., 2002).

An interesting theoretical framework has been proposed by Fingelkurts and Fingelkurts (in press) as to the neural mechanisms of attention. Fingelkurts et al. (2009,2010,2013) conceptualize attention within their theory of the operational architectonics (OA) of brain and mind functioning. According to this theory, involuntary (bottom-up) attention arises as a result of self-organized formation of neuronal assemblies whose operations are divided by rapid transients (so called RTPs in the brain oscillations) that signify the breakpoints of attention leading to an attentional disengagement, shift, and/or allocation to a new operation. Fingelkurts and Fingelkurts (in press) further suggest that the duration of these operations is determined by external stimuli and modulated by arousal as well as affective reinforcement. Voluntary (top-down) attention emerges as a result of binding of multiple operations responsible for sensory percepts or motor programs in a contextdependent way as a function of a saliency, prior knowledge and expectancies. During this process, the ever-changing and multiform stream of cognition and conscious experiences is somehow "frozen" and "classified," thus leading to the phenomenological experience of acts or moments of focused attention in which our consciousness is kept focused as a mental magnifying lens at the attended object or scene. According to Fingelkurts and Fingelkurts (in press), the skill to voluntarily focus attention on a specific image, object or thought is guided by a specific fronto-parietal operational module (OM) that serves as an order parameter and determines which particular OM of cortical dynamics (synchronized spatial-temporal pattern of brain activity) should be reinforced at any given moment of time in order to present a particular image, object or thought in the focus of attention.

Finally, it should be noted that the periodic nature of attentional processing is also visible at wider temporal scales: spontaneous eyeblinks, which occur 15–20 times per minute on average, are closely correlated to attentional processing in that they tend to occur at breakpoints of attention, such as the end of a sentence while reading, a pause by the speaker while listening to a speech, and implicit breakpoints while viewing videos. This close correlation has led Nakano et al. (2013, p. 702) to hypothesize that "eyeblinks are actively involved in the process of attentional disengagement during cognitive behavior by momentarily activating the default-mode network while deactivating the dorsal attention network."

All this evidence points, on the one hand, to the periodic nature of attention and, on the other hand, to the close correlation between the periodicity of attention and brain oscillations. Above all Busch and VanRullen's (2010) shows two important facts about attention. Firstly, attention has a natural or default periodicity (around 7 Hz): it samples information even when only a single location has to be monitored. Secondly, attention cannot be allocated at any given time but only at specific phases of an oscillatory cycle. It is therefore highly plausible to theorize that attention is the product of, or is underpinned by (one or some of) such brain oscillations.

### **WORKING MEMORY**

In order to maintain and combine the elements selected and isolated by attention, a specific mechanism is required. This mechanism is working memory.

Working memory is generally considered as a system that helps to simultaneously manipulate information over a short period and update it in memory. More specifically, as highlighted by Unsworth and Engle (2007), working memory is needed to maintain new and novel information in a heightened state of activity, and to correctly discriminate between relevant and irrelevant information with regard to the task to be performed, by preventing the interference of automatic tendencies and routines. In this sense, working memory is "not directly about remembering *per se*, but instead reflects a more general ability to control attention and exert top-down control over cognition" (Broadway and Engle, 2011b, p. 1).

The role of working memory in flexibly and freely combining content elements into new structures is explicitly theorized by Oberauer (2009). According to Oberauer (2009), working memory is a system that is able (among other functions) to build and maintain new structural representations by establishing and holding temporary *bindings* between contents (objects, events, words) and contexts (such as positions in a generic cognitive spatial or coordinate system, or argument variables in structure templates). Neurophysiological studies have started to elucidate this system. Experimental findings using the OA methodology in EEG analysis clearly point to the fact that the binding of sensory feature representations into phenomenal (subjective) "objects," active encoding, maintenance and retrieval of these mental "objects" during working memory are critically dependent on dynamic millisecond-range synchronization of multiple operations performed by local neuronal assemblies that operate on different temporal (oscillations) scales nested within the same operational hierarchy (Fingelkurts et al., 2010; Monto, 2012). In particular, medium life-span OMs of brain activity (that "cover" certain cortical areas) seem necessary to achieve successful memorization (Fingelkurts et al., 1998, 2003). Indeed, although memory encoding, retention and retrieval often share common regions of the cortex, the operational synchrony of these areas is always unique and presented as a mosaic of nested OMs for each stage of the short-term memory task (Fingelkurts et al., 1998, 2003). When there are too few or too many OMs and their life-span is either too short or too long, then such conditions lead to cessation of efficient memorization.

By supporting arbitrary bindings between virtually any content with any context, working memory enables the compositionality of thought, and the creation of a theoretically unlimited number of different ideas. The binding of contents and contexts allows for the arrangement and representation of objects and events in a spatial and temporal coordinate system, as well as in some other quantitative dimensions such as size, brightness, intelligence, etc.

Recent findings showing that working memory plays a role in the construction of novel future events are provided by Hill and Emery (2013). They started from the observation that, unlike past episodic recall which requires reconstructing the elements of a previously experienced event, future thought depends on a novel recombination of episodic details into a hypothetical event (Addis and Schacter, 2011). This requirement for a novel recombination suggests that future event construction involves additional cognitive and neural processes that are not as involved in autobiographical memory reconstruction. Furthermore, the observed involvement of both prefrontal and hippocampal regions in the construction of future events (Addis et al., 2007), as well as the involvement of the episodic buffer of working memory in prospective mind wandering (Baird et al., 2011) and in recombining semantic personal information (D'Argembeau and Mathy, 2011) and information from multiple modalities (Szpunar et al., 2009), suggests that a combination of executive and memory binding functions may contribute to this novel constructive process. This points to a potential cognitive role for working memory in imagining future episodes, above and beyond the contributions provided by access to the autobiographical database. Using a composite score of working memory capacity (WMC), Hill and Emery (2013) examined the extent to which residual working memory variance contributes to future thought while controlling for autobiographical memory. Subjects had to complete simple and complex measures of working memory and were cued to recall autobiographical memories and imagine future autobiographical events consisting of various levels of specificity (i.e., ranging from generic to increasingly specific and detailed events). They found (1) that the ability to imagine personally relevant events in the future is strongly related to autobiographical memory and (2) that after controlling for autobiographical memory, residual working memory variance *independently* predicts future episodic specificity. That is, when imagining future events, working memory contributes to the construction of a single, coherent, future events depiction.

It is interesting to note that what I propose to be the two main underlying mechanisms of constructive processes – attention and working memory – interact so closely, and seem not to be able to operate without each other, that some scholars have put forward models explicitly including either the former in the latter (Engle, 2002; Oberauer, 2009), or the latter in the former (Knudsen, 2007).

## **VARIOUSLY USING ATTENTION AND WORKING MEMORY YIELDS VARIOUS CONSTRUCTION PROCESSES**

In order to occur, any construction process requires the availability of some basic elements to be assembled, and a mechanism that allows for the assembly of these basic elements (Benedetti et al., 2010; Fingelkurts et al., 2010, 2013). Regarding the construction processes underpinning the various forms of consciousness, we have seen that the basic elements are provided by attention, and that their assembly is ensured by working memory.

Attention can be variously applied and used: it can be focused internally or externally (Chun et al., 2011); it can be focused at variable levels of size, being set either widely across a display of objects or narrowly to the size of a single object (Jonides, 1983); it can be focused at variable levels of intensity (La Berge, 1983), etc. Likewise, working memory can be variously used to perform arithmetical operations, compare pieces of information, combine items, etc. In some situations it is optimal to maintain as many distinctive items as possible active in working memory, while in some others it is optimal to maintain only one item (Unsworth and Engle, 2007). Some tasks may require a more intensive involvement of the procedural part of working memory as compared to the declarative part, while some others may require the opposite (Oberauer, 2009), etc.

It is precisely the fact that both the selection of basic elements and their assembly can be performed in various ways, that allows various construction processes to be performed, thereby obtaining various forms of consciousness. This parallels any other construction process. Just as a house can be built using bricks rather than stones or wood, so too conscious experiences can be built using current information from the outer world rather than information retrieved from memory. Likewise, just as bricks can be assembled in order to build a corner rather than a wall, so too information retrieved from memory can be combined in order to imagine future events rather than to relive past events.

This paper describes the constructive process that yields the conscious experience of temporality. Therefore, I will specifically focus on which operations attention and working memory perform in order to produce such a conscious experience. However, I will also briefly describe the operations of attention and working memory involved in the construction of some other form of consciousness, so as to provide a set of comparable analyses for future empirical verification.

### **THE CONSTRUCTION OF TEMPORAL EVENTS**

If one wants to investigate how the construction of temporal events occurs, one must necessarily start from the analysis of its most basic manifestation: duration. The subjective experience of temporality is fundamentally durational in nature, in the sense that duration represents a prerequisite for the development of the other important experiences that are usually conceived as being strictly linked to time, such as the awareness of change, the experience of succession, and the possibility of distinguishing past from present and from future. Some researchers (Gibson, 1975, 1979; Lakoff and Johnson, 1999) maintain the primacy of event detection over duration: according to them, temporal experience would primarily be, and derive from, the awareness of change exhibited by events in the world. However, some facts clearly show that the experience of time is not based on event comparison (for a detailed review, see Evans, 2004). Firstly, we can experience the passage of time whether there has actually been a change in the world-state or not, as evidenced by situations of relative sensory-deprivation (such as windowless, sound-proofed cells) in which subjects are still aware of the passage of time. Secondly, the experience of duration is independent of the nature of external events: the experience of protracted duration can result from both states in which the stimulus array is impoverished and events that, on the contrary, are extremely rich in sense-perceptory terms.

How is then the experience of duration constructed? According to my analysis (Marchetti, 2009a), an event or object assumes a durational dimension when we devote, in an incremental manner, part of our attention to the conscious experience of the event or object. More specifically, the duration of a given event is determined by the cumulative quantity of labor performed by the portion of our attention (*A*t) that is kept focused on the conscious experience of the event. Since the event itself is constructed by means of another portion of one's attention (*A*e), duration judgments can be considered equivalent to dividedattention tasks, in which attention must be divided between temporal and non-temporal information processing (Block and Zakay, 2001; Zakay and Block, 2004). The labor performed by *A*<sup>t</sup> is cumulated thanks to working memory. The cumulative amount of labor performed by *A*<sup>t</sup> constitutes the basis on which the conscious experience of duration and more in general time-sensation are anchored.

Some alternative explanations of the psychological phenomenon of duration have been put forward, but have various drawbacks. A very well-known alternative is the internal-clock model (Treisman, 1963; Wearden et al., 1998; Wearden, 2001). As an example of internal-clock models, let us consider the "scalar expectancy theory" (SET) proposed by Wearden (2001). The SET model is composed of three parts: a pacemaker-accumulator, a memory system, and a comparison or decision process. To understand how such a model operates, consider the problem of timing the duration of a stimulus *t*<sup>1</sup> through comparison with the duration of another stimulus, *t*<sup>2</sup> (whether, for example, they are equal or different in length). Onset of stimulus *t*<sup>1</sup> causes the pulses, that is, the "ticks" of the inner clock, to flow from the pacemaker to the accumulator. Offset of stimulus causes the interruption of the flow of pulses: the accumulation of pulses by the accumulator is then stopped. The memory system allows duration representations to be stored either in a long-term memory or in a short-term memory. Thanks to the memory system, the duration of the first stimulus *t*<sup>1</sup> can be stored until after the second one, *t*2, has been presented: a comparison between the two stimuli is then possible. Finally, *t*<sup>1</sup> and *t*<sup>2</sup> are compared and a response can be delivered.

Internal-clock models can certainly account for some phenomena, such as the differences in judging the duration of auditory stimuli versus visual ones. However, they face various kinds of problems. Generally speaking, internal-clock models seem inadequate to explain the inherent inaccuracy of human duration judgments, that is, the fact that organisms provided with such a precise mechanism as an internal clock very often exhibit inaccurate timing behaviors (Block, 1990). Methodologically speaking, internal-clock models present many drawbacks (Block, 2003), among which the facts that most of the evidence comes from a few relatively simple paradigms (such as the peak procedure and the bisection task), from studies in which animals estimate the duration of a single stimulus or an interval between two stimuli, and from experiments during which no external stimuli are presented internal-clock models cannot readily explain the effects of attention on psychological time; many of the findings that internal-clock models explain are generic, that is, they are not unique to the time dimension. The same findings of internal-clock models could be explained by models composed of very basic modules, such as a perceptual system, without resorting to an additional component such as the pacemaker.

Additionally, internal-clock models face the problem of the individuation of the internal clock (Ornstein, 1969). Apparently, human beings are provided with a number of different mechanisms that could all equally and finely act as internal clocks: heart rate, breathing rate, cellular metabolism, toe-nail growth, alpha rhythm, etc. What are the criteria for judging a given physiological process to be an internal "chronometer"? Why could hair growth or toe-nail growth rather than alpha rhythm not be designated as the internal time keeper? Richelle et al. (1985, p. 90) go so far as to pose the provocative question: "Why not admit that there are as many clocks as there are behaviors exhibiting timing properties?" This admission definitely confirms the uselessness and lack of parsimony of the notion of the internal clock for a general analysis and explanation of time experience.

Finally, the explanation put forward by internal-clock models implies a fallacious circularity (Marchetti, 2010). Merely naming a given process as a "time keeper" or "internal clock" cannot automatically suffice to appoint it as the mechanism responsible for time experience. A counter or a timer, like any clock, can only provide the raw material necessary for counting. But there must be someone who performs the counting. As Vicario argues: "The clock says the hour only when we look at it" (Vicario, 2005, p. 165). It is we who assign the physical mechanism – whether it is a pendulum, the sun, a clock, or something else – the capacity to trace the flowing of our conscious experiences and to estimate their duration. To realize this, just consider the fact that a clock which has stopped or is not working, despite not measuring any actual time, can still be interpreted by an observer as telling the time!

Another possible explanation accounts for time experience in terms of the passage from one conscious state to another: that is, the fact that an event that is being experienced now, becomes no longer present, and passes into the domain of memory. The experience of the passage of time would be produced by the experience of change of conscious state and the phenomenal differences that exist between the various conscious states (the experience of remembering an object is phenomenally different from the experience of actually perceiving it). By taking into account the experiences implied by the change of conscious state, this explanation certainly captures part of the origin of the experience of time. However, it still remains at a very phenomenological and surface level, without investigating the possible neurophysiological mechanisms underpinning the phenomenon. In fact, the experience of the change of conscious state, and of the differences between the various conscious states, can be explained by and reduced to a more basic level: the working of attention and working memory. Actually, in order for one to realize that "something-that-ispresent-now" has become "something-that-is-no-longer-present," one must first isolate the two experiences (the "something-thatis-present-now" and the "something-that-is-no-longer-present") and then compare them. That is, one must first attentionally focus on them separately, and then keep one of them active in working memory so as to allow for the comparison with the other.

Let us now see what evidence could support my hypothesis about the involvement of attention and working memory in the construction of the experience of duration.

### **EVIDENCE OF THE INVOLVEMENT OF ATTENTIONAL AND WORKING MEMORY IN THE EXPERIENCE OF DURATION**

Evidence of the attentional basis of the experience of duration is confirmed by a number of empirical findings. Experiments in which subjects are asked to prospectively<sup>1</sup> judge the duration of the time period in which they had to perform a certain task, reveal that the judged time decreases linearly with the increased processing demands of the non-temporal information, and that experienced duration increases to the extent that subjects can allocate more attentional resources to the flow of time itself (Hicks et al., 1976, 1977; Brown, 1985; Coull et al., 2004). In prospective time judgments, negative high-arousal stimuli, inducing a stronger attentional response, are overestimated compared with positive high-arousal stimuli, inducing a weaker attentional response (Angrilli et al., 1997). Likewise, Tse et al. (2004) found that the engagement of attention by an unexpected event increases the rate of information processing brought to bear on a stimulus, thus inducing an overestimation of the duration of the stimulus.

A confirmation of the role played by attention in constructing temporal experience also comes indirectly from experiments in which subjects are asked to retrospectively<sup>2</sup> judge the duration of events. Such experiments show that subjects remember a time period as being longer in duration to the extent that there are greater context changes (such as a variation in background stimuli or interoceptive stimuli, the psychological context, the processing context, etc.; Block and Zakay, 2001; Zakay and Block, 2004). Since, as Glicksohn (2001), observes retrospective time estimation entails reperceiving (imaginally) the event, it is conceivable that the retrospective judgment of time is determined (at least in part) by what would have been a prospective judgment of time. Therefore, the phenomena observed in retrospective time estimation can also be ascribed to attention (for a more detailed discussion, see Marchetti, 2010).

Finally, it should be noted that attention has been found to also determine other important aspects of temporal experience not principally related to the experience of duration. For example, the phenomenon known as prior-entry shows that when a person attends to a stimulus, he/she perceives it as having occurred earlier in time than it would if he or she was not attending to it (Shore et al., 2001; Shore and Spence, 2004).

As concerns the role of working memory in the experience of duration, to my knowledge there is no direct evidence as yet of its involvement in the terms that are explained in this work. Broadway and Engle (2011a,b) showed, in a series of temporal reproduction tasks, the close relationship between WMC and duration judgment. Using naturally occurring individual differences in WMC to mimic load manipulations, Broadway and Engle (2011b) found that low-WMC individuals are less sensitive than

<sup>1</sup>That is subjects are alerted in advance that time judgments will be required.

<sup>2</sup>That is subjects do not know in advance that they will later be asked to judge the duration of a time period.

high-WMC to identifying the longer of two comparison intervals across a range of absolute durations and duration differences. That is, individual differences in WMC predict differences in temporal discrimination. As Broadway and Engle explain, WMC is necessary for performing temporal reproduction tasks because a person would need to encode and maintain access to two distinct representations of elapsed time in an ongoing dynamic manner for comparison and temporal judgment. Therefore, further experiments are needed to verify what my analysis shows, that is, that working memory is necessary to consciously experience duration *per se*, independently of any possible duration judgment.

Several models and empirical findings point specifically to a cumulative build-up mechanism as a possible basis for the experience of duration (for a review, see Wittmann, 2013). It should be noted however that none of them explicitly refer to working memory as the main mechanism responsible for integrating information. According to the dual klepsydra model by Wackermann and Ehm (2006), time duration is represented by the states of inflow-outflow units, which function as leaky integrators. The state of the integrator is a non-linear climbing function of physical time. Craig (2009) theorizes that the anterior insula integrates representations of body states with cognitive and motivational states, creating a series of emotional moments, each of which is a coherent representation of all feelings experienced at that time. The experience of duration would develop from the integration of a series of states over time. Craig's model suggests that the subjective dilation of time during periods of high emotional salience results from the high rate of salience accumulation, which would rapidly fill up global emotional moments.

From an empirical point of view, neurophysiological findings in primates and humans show that climbing neural activity in several brain regions is related to the experience of duration (Niki and Watanabe, 1979; Lebedev et al., 2008; Mita et al., 2009; Wittmann et al., 2010, 2011; Casini and Vidal, 2011; Merchant et al., 2011). For example, functional fMRI experiments in which subjects have to temporally reproduce acoustic stimuli of various lengths (Wittmann et al., 2010, 2011), show an accumulating pattern of activity within left and right dorsal posterior insula and superior temporal cortex during the encoding phase of the task (with the activity peaking at the end of the interval), and an accumulating activation in the anterior insula, medial frontal and inferior frontal cortex in the reproduction phase of the task (with the activity peaking shortly before the button press indicating the reproduced length by the subject). As Merchant et al. (2013) observe, the ubiquitous increases or decreases in cell discharge rate as a function of time across different timing tasks and brain areas, suggest that ramping activity is a fundamental element of the timing mechanism.

### **THE EXPERIENCE OF THE PAST, PRESENT AND FUTURE**

As we have seen, we have the capacity to experience the same event as occurring either in the present, past, or future (Nyberg et al., 2010). What construction process makes this possible?

According to my analysis, a temporal event can acquire a past or future dimension if it is placed in a temporal coordinate system having the "present" as its reference point.

As many scholars have highlighted (Revonsuo, 2006; Droege, 2009; Dresp-Langley and Durup, 2012; Fingelkurts and Fingelkurts,2014), the temporal dimension of "present"is constitutive of conscious experiences3. Without such a dimension, there would be no conscious experiences as we currently live them: it is the reference point that allows for the construction of past and future. As Fingelkurts and Fingelkurts (2014) state: "even remembering the past images and planning the future events cannot be performed other than in the present moment and in relation to current state of affairs." There can be cases of conscious experiences lacking the characteristic of time (as well as of for-me-ness) such as those achieved by trained subjects who practice meditation. However, these cases are very uncommon, and can be attained either in exceptional cases or via extended practice.

Once a temporal event is placed in a temporal coordinate system where the "present" acts a reference point, it can be related to this reference point, and consequently assume either a past or a future property.

How can a temporal coordinate system be constructed? It can be obtained from the most elementary and primitive experience of time, that is, duration. As we saw, the experience of duration is based on the *cumulative* quantity of labor performed by the portion of attention (*A*t) that is kept focused on the conscious experience of the event. Being cumulative, the quantity of labor performed by *A*<sup>t</sup> can only increase. This makes it possible to arrange events in a univocal and irreversible way, which is precisely the condition necessary to construct a temporal coordinate system made of "past,""present," and "future." Generally speaking, if we consider for example that a given event X can be associated with a certain amount of labor performed by *A*t, an event Y that is associated with a higher amount of labor performed by *A*<sup>t</sup> appears to us to happen "after" X, whereas an event Z that is associated with a lower amount of labor performed by *A*<sup>t</sup> appears to happen "before" X. That is, once X, Y, and Z are assigned a specific location in a cognitive coordinate system characterized by onedimensionality and irreversibility, they are ordered according to the temporal dimension. More specifically, if event X occurs in the "present," Z will be experienced as occurring in the "past" and Y as occurring in the "future."

The existence of past and future makes it possible to construct the conscious experiences of remembering past events and imagining future events. As suggested by Ceccato and Zonta's (1980) work, and more specifically by phenomenological analysis (see Thompson, 2008), the subjective experience of remembering an event derives from adding the temporal dimension of past to the event. In remembering, one lives experiences as having occurred in the "past" and not as occurring now. In a similar way, the subjective experience of imagining a future event derives from adding the temporal dimension of future to the event. The operation of adding a (past or future) temporal dimension to an event is performed thanks to working memory, which binds the event to a position in the temporal coordinate system (Oberauer, 2009).

<sup>3</sup>The dimension of "present" is not the only constitutive feature of consciousness. Some other dimensions have also been identified, such as the dimensions of "here" and "for-me-ness," that is, the fact that of all of our experiences are characterized implicitly by a quality of mineness: see Revonsuo (2006) and Gallagher and Zahavi (2008).

Empirical evidence supporting this analysis is still partial and indirect, and a specific investigation must be performed in order to validate the analysis. As we have seen, Hill and Emery's (2013) work confirms the role played by working memory in mental time travel, specifically when imagining future events. The close link between attention and the conscious experience of reliving past events was reviewed by De Brigard (2012). Behavioral studies using divided attention paradigms show that when internal attention (Chun et al., 2011) to material-congruent deeply encoded information is disrupted during retrieval, recollection is significantly impaired (Fernandes and Moscovitch, 2000; Hicks and Marsh, 2000). Likewise, neuropsychological studies show that under free-recall conditions, patients with parietal cortex damage, which usually impairs attention to external stimuli, tend to retrieve less episodic perceptual details and lower levels of vividness in their recollections from their autobiographical memories relative to both cued-recall and healthy controls (Berryhill et al., 2007). Davidson et al. (2008) also show that patients with parietal lesions produce a reduced number of "remember" responses, which are associated with increased subjective experience of recollection, relative to both "know" responses and controls.

To summarize the analysis I have put forward: the conscious experience of duration is produced by two (non-conscious) mechanisms: attention and working memory. The conscious experiences of past, present and future are in turn built on the conscious experience of duration. By adding the temporal dimensions of past and future to an event, it is possible to subjectively experience that event as remembered or occurring in the future, respectively.

This kind of explanation of temporal experience does not rely on mechanisms purposefully designed to process time (such as an "internal-clock"), but rather on mechanisms (attention and working memory) that have other, more basic and

general-purpose functions, not necessarily related to the encoding of duration and time. As such, the circularity implied in many other explanations of temporal experience (Marchetti, 2009a) is avoided.

### **HOW THE CONSTRUCTION PROCESS WORKS: A COMPARISON WITH SOME OTHER FORMS OF CONSCIOUSNESS**

As we have seen, the attentional selection of basic elements and their assembly by means of working memory can be performed in various ways. This allows various construction processes to be performed, thereby obtaining various forms of consciousness. Here I will briefly show how variously using attention and working memory yields different forms of consciousness (see **Table 1** for an overview). In this way, I intend to provide an initial set of comparable analyses that can be used to empirically verify my analyses.

### *The conscious experience of space*

The conscious experience of space is primarily based on bodily movements (Berthoz, 2000; Morris, 2004). However, bodily movements, albeit necessary, are not sufficient (Marchetti, 2009b). The conscious experience of space also requires that: (a) attention is internally applied to prorioceptors and the vestibular system in order to isolate and bring to consciousness the single perceptions entailed by movement. External attention is also required to build some conscious experiences of space (Berthoz, 2000); (b) working memory assemblies these single perceptions, by keeping them present in an incremental way. It is this latter operation that allows for the construction of a "sequence" or "succession" of perceptions, which is the basis for the formation of two-dimensional constructs, such as "path," "line," and "distance."

The need for working memory (in addition to bodily movement) in the construction of the conscious experience of space is evident when comparing the different conscious experiences of


**Table 1 |The different involvement of attention and working memory in some forms of consciousness.**

"movement" and "line" (or "path") we have when performing the same act. For example, move your index finger slowly. Now, look at the tip of the finger while the finger moves, and consider it as a moving object. Next, repeat the movement and consider the path or line drawn by the tip of the finger. You will notice that in the former case you will simply follow the tip of the finger, maybe anticipating its direction, but without keeping track of the positions previously occupied by it; on the contrary, in the latter case you will follow the tip of the finger by constantly keeping track of the positions it occupied, moment after moment, since it started moving.

### *Language and conscious thinking*

Language allows us to variously and, theoretically, endlessly combine meanings. This combinatorial power is made possible by relational units (such as conjunctions and prepositions: Sapir, 1921; Ceccato, 1972; Ceccato and Zonta, 1980; Benedetti, 2006, 2009, 2011), which, tying together two or more semantic elements (simple words, other relational units, complex thoughts, etc.), allow for the construction of correlational networks (or minimal unit of linguistic thought).

As shown by Benedetti's (2006), analysis the production of correlational networks is made possible by working memory and procedural memory. These two forms of memory, albeit necessary, are not sufficient. Theoretically, correlational networks can be infinite. However, the well-known limits of working memory constrain this possibility. In fact, sentences have a limited length and are separated by semicolons, full stops, pauses, etc. At full stops, working memory stops being loaded, and what has been present in it up to that moment, must be in some way stored in a summarized form in a short-term memory. Pronouns have the function of reloading into working memory what has been stored in short-term memory.

It should be noted that, compared and contrary to the experiences of time and space, language does not specifically require that working memory cumulates attentional states. In language, working memory has the primary and most general function of supporting relational units in (variously) combining semantic elements. Moreover, language specifically needs the involvement of semantic memory, which is only rarely, or indirectly, involved when constructing temporal and spatial experiences. Additionally, language deeply requires other forms of attention, such as shared attention (Tomasello, 1999, 2004; Oakley, 2009).

Language and the kind of thought it entails (linguistic or correlational thought: Ceccato and Zonta, 1980; Benedetti, 2011) can be considered a specific subset of conscious thinking (Marchetti, 2010). In many cases, we have dynamic and evolving visuo-spatial thought (such as when we think, for example, about a flower that opens), or forms of thoughts involving other senses. Moreover, most of the times, thoughts do not just combine elements, but produce results, such as when a solution suddenly pops into one's mind after having searched for it for a while. Generally speaking, conscious thinking requires operators other than just the relational units involved by language: these operators should allow, for example, for the transformation of the object of thought, or the production of new conscious experiences from earlier ones, or the comparison of elements.

Therefore, conscious thinking, compared to language, sometimes requires a deeper involvement of working memory (such as when evolving representations are produced), and the presence of dedicated (and usually unconscious) frames, representational and operational systems that are not strictly required by language (such as those that allow one to draw inferences, or make decisions).

A final consideration should be made about the empirical verification of the analyses I have put forward here and elsewhere (Marchetti, 1997, 2010). Generally speaking, it should be noted that these analyses are particularly suited to be verified by an empirical approach centered on the notion of operation and its combinatorial power. In fact, my analyses describe in a sufficiently detailed way what operations (that is, how basic elements are assembled and combined) must be performed in order to obtain certain forms of consciousness. As already shown in another paper (Benedetti et al., 2010), the Fingelkurts brothers' OA (Fingelkurts and Fingelkurts, 2001, 2005; Fingelkurts et al., 2009, 2010, 2012, 2013) offers such an empirical approach. According to OA, simple cognitive operations that present some partial aspect of an object/scene/concept or thought are presented in the brain by local 3D-fields produced by discrete and transient neuronal assemblies, which can be recorded by an EEG. More complex operations that constitute the whole object/scene or thought are brought into existence by joint (synchronized) simple operations in the form of coupled 3D-fields – so called OMs of varied complexity. OA does not put forward specific analyses in operational terms of phenomenological contents and forms. However, because of the hierarchical organization implied by its theoretical framework, OA is very suited to verify precisely this kind of analysis.

### **CONCLUSION**

This article is an attempt to demonstrate that some of the most important forms of consciousness – episodic memory, episodic future thought, perception, language and conscious thinking – are based on an active constructive process. Despite the fact that we experience the world surrounding us as a continuous, seamless flow of information, many psychological and neurophysiological observations reveal that information is actually extracted and processed in distinct moments, similar to the snapshots of a camera.

For most of the forms of consciousness to occur, a construction combining the various moments or snapshots is required. The main plausible mechanisms implied in this construction process are attention and working memory. Attention allows for the selection of the basic elements to be assembled. Empirical evidence shows that attention works on a period basis (around 7 Hz): it samples information even when only a single location has to be monitored. Working memory represents the mechanism that allows for the assembly of the basic elements selected by attention, by establishing and holding temporary *bindings* between contents and contexts.

Both the selection of basic elements and their assembly can be performed in various ways, thus allowing various construction processes to be performed, thereby obtaining various forms of consciousness.

Temporal experience is based on a specific kind of working of attention and combination of attentional moments. Namely, the duration of a given event is determined by the cumulative quantity of labor performed by the portion of our attention (*A*t) that is kept focused on the conscious experience of the event. The labor performed by *A*<sup>t</sup> is cumulated thanks to working memory. The experience of duration provides the basis for the construction of the conscious experiences of past, present and future. By adding the temporal dimensions of past and future to an event, it is possible to subjectively experience that event as remembered or occurring in the future.

### **REFERENCES**


Ceccato, S. (1972). *La Mente Vista da un Cibernetico*. Torino: Eri.


Vicario, G. B. (2005). *Il Tempo. Saggio di Psicologia Sperimentale*. Bologna: Il Mulino.


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 17 May 2014; accepted: 24 July 2014; published online: 14 August 2014.*

*Citation: Marchetti G (2014) Attention and working memory: two basic mechanisms for constructing temporal experiences. Front. Psychol. 5:880. doi: 10.3389/fpsyg.2014.00880*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Marchetti. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Parallel effects of memory set activation and search on timing and working memory capacity

#### *Richard Schweickert <sup>1</sup> \*, Claudette Fortin2, Zhuangzhuang Xi 1† and Charles Viau-Quesnel 2†*

<sup>1</sup> Department of Psychological Sciences, Purdue University, West Lafayette, IN, USA

<sup>2</sup> École de Psychologie, Université Laval, Québec, QC, Canada

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

### *Reviewed by:*

Marian Berryhill, University of Nevada, Reno, USA Trevor B. Penney, National University of Singapore, Singapore

### *\*Correspondence:*

Richard Schweickert, Department of Psychological Sciences, Purdue University, 703 Third St., West Lafayette, IN 47907, USA e-mail: swike@psych.purdue.edu

### *†Present address:*

Zhuangzhuang Xi, AIR Worldwide, Boston, MA, USA; Charles Viau-Quesnel, Psychoeducation Department, Université du Québec á Trois-Riviéres, Trois-Riviéres, QC, Canada

Accurately estimating a time interval is required in everyday activities such as driving or cooking. Estimating time is relatively easy, provided a person attends to it. But a brief shift of attention to another task usually interferes with timing. Most processes carried out concurrently with timing interfere with it. Curiously, some do not. Literature on a few processes suggests a general proposition, the Timing and Complex-Span Hypothesis: A process interferes with concurrent timing if and only if process performance is related to complex span. Complex-span is the number of items correctly recalled in order, when each item presented for study is followed by a brief activity. Literature on task switching, visual search, memory search, word generation and mental time travel supports the hypothesis. Previous work found that another process, activation of a memory set in long term memory, is not related to complex-span. If the Timing and Complex-Span Hypothesis is true, activation should not interfere with concurrent timing in dual-task conditions. We tested such activation in single-task memory search task conditions and in dual-task conditions where memory search was executed with concurrent timing. In Experiment 1, activating a memory set increased reaction time, with no significant effect on time production. In Experiment 2, set size and memory set activation were manipulated. Activation and set size had a puzzling interaction for time productions, perhaps due to difficult conditions, leading us to use a related but easier task in Experiment 3. In Experiment 3 increasing set size lengthened time production, but memory activation had no significant effect. Results here and in previous literature on the whole support the Timing and Complex-Span Hypotheses. Results also support a sequential organization of activation and search of memory. This organization predicts activation and set size have additive effects on reaction time and multiplicative effects on percent correct, which was found.

**Keywords: time production, working memory capacity, complex memory span, activation, retrieval, memory search, additive factor method, selective influence**

## **INTRODUCTION**

Accurately estimating a brief time interval is important in numerous everyday activities including talking, playing music and performing in sports. In studying timing performance, people are often asked to reproduce a short time interval by tapping a finger twice. This is relatively easy, provided a person attends to it. A brief shift of attention to another task usually interferes with timing, however. According to a prevalent accumulation model, timing demands are limited and well defined: Pulses are generated by an internal pacemaker, a gate allows pulses to be sent to an accumulator, and when the pulse count reaches a criterion, a movement ending the temporal reproduction is prompted. Attention controls the gate, the criterion and accumulated pulses require memory storage, and comparing accumulated pulses with the criterion requires attention (Gibbon et al., 1984; Zakay and Block, 1996; Brown, 2006; Buhusi and Meck, 2009). For recent reviews see Buhusi and Meck (2005, 2009). Timing is sensitive to the relentless attention and memory requirements throughout the reproduced interval, making timing a sensitive indicator of demands in secondary tasks.

Timing is also likely to be sensitive to demands of ongoing internal processing, thinking, mind-wandering, and so on. Indeed, there is evidence that mental time travel interferes with timing (El Haj et al., 2013). As words or images arise internally during production of a time interval, they compete for resources allocated to timing. Further, they occasionally produce a cue or prime that by association activates information in secondary memory. Does mere activation of information interfere with timing, or does activated information interfere only if it is used? Internal cues are difficult to control experimentally, of course. Here we address the corresponding questions with regard to a secondary task.

Brown (1997) reported that many secondary tasks interfere with concurrent timing. But not all do. High on the list of candidates likely to influence concurrent timing, in Brown's view, are executive processes such as coordination and scheduling, because they demand attention and working memory. As an example, Brown (2006) showed that a particular executive process, random number generation, interfered with concurrent timing. But even among executive processes, some interfere with timing and some do not. For example, Fortin et al. (2010) showed that task switching, an executive process, did not.

A difficulty in ascertaining which processes interfere with timing is that the term "working memory" is broad. A way forward is provided by a well-specified measure of working memory capacity, complex span (Daneman and Carpenter, 1980). People with high complex span are more accurate at timing than those with low complex span (Broadway and Engle, 2011).

In a complex memory span task, a person performs an activity (such as subtraction), stores an item; performs another activity and stores another item, continuing until the sequence of activities and items is finished. Finally, items are recalled in order. The score is typically the number of items correctly recalled in correct serial positions (e.g., Unsworth and Engle, 2007, p. 110). Individuals with high complex-span perform better on various tasks than individuals with low complex-span. The executive attention view of working memory capacity (Engle and Kane, 2004; Kane et al., 2007) explains this by saying high-working memory capacity individuals have better ability to maintain goals. In a complex-span task it is important to maintain the goal of remembering items while carrying out an unrelated activity such as subtraction. For recent discussion of this view and of tasks related to working memory capacity, see Unsworth et al. (2012). This view is remarkably similar to the explanation that timing requires continual maintaining of the goal to keep time, and is interfered with by tasks that distract from the goal. With this view, when timing and a secondary task are done together, if interference occurs, it is the result of some particular secondary task process distracting from the goal of timing.

When we consider the few tasks whose effect on timing and whose relation to complex-span are both known, there are striking parallels. (a) Task switching does not interfere with timing (Fortin et al., 2010) and task switching performance is not related to complex-span (Kane et al., 2003). (b) The same is true for attention-demanding visual search (for timing, Fortin et al., 1993; Schweickert et al., 2007; for complex span, Kane et al., 2006). (c) Sternberg memory search interferes with concurrent timing (e.g., Fortin and Rousseau, 1987) and performance is related to complex-span (Conway and Engle, 1994). (d) Generating words starting with a given letter increases variance in time production (Ogden et al., 2011) and performance is related to complexspan (Unsworth et al., 2011; see also Rosen and Engle, 1997). We tentatively add a fifth, internal process. (e) Mental time travel is related to timing (El Haj et al., 2013) and is related to complex-span (Mrazek et al., 2012). The last statement is tentative because the timing experiment by El Haj et al. (2013) differs considerably from the others mentioned. Participants verbally estimated durations longer than 30 s in prospective and retrospective timing paradigms rather than producing intervals shorter than 5 s in a prospective paradigm. Further, evidence of mental time travel is indirect. It was inferred by El Haj et al. (2013) from Remember/Know judgments in a recognition task. Evidence was indirect also in Mrazek et al. (2012). They presented thought sampling probes while participants performed complexspan tasks; performance was negatively correlated with amount of attention to task unrelated concerns, much of which is likely to be mental time travel (Corballis, 2012). Reviews of timing tasks are in Brown (1997, 2006) and Fortin (1999). For tasks whose performance correlates well with Working Memory Capacity, see the review by Kane et al. (2007, p. 35).

In these examples complex-span is unrelated to processes that do not interfere with timing (task switching and visual search) but related to processes that do so (memory search, word generation and possibly, mental time travel). A generalization from these examples is that timing and a process executed concurrently with it interfere if and only if performance of the process varies with complex memory span. We call this the Timing and Complex-Span Hypothesis. If true the hypothesis tightens the previous characterization of processes interfering with timing as those that are executive. Of course, interference and variation are matters of degree. A more precise statement of the hypothesis is that a process interferes with concurrent timing to the degree that process performance varies with complex memory span. Because the literature typically classifies processes as interfering with timing or not, or as related to complex span or not, we discuss the hypothesis here in a dichotomous form.

A test immediately arises from the paper by Conway and Engle (1994). They examined two processes, short term memory search and activation (retrieval) of items to be searched. Memory search satisfies the proposition. Performance on activation was the same for low and high complex-span individuals. If the Timing and Complex-Span Hypothesis is true, activating a set of items to be searched will not interfere with concurrent timing. Activating a memory set is particularly interesting because it is an important component of complex memory span tasks. In the steps of a complex memory span trial listed above, is the penultimate step, activating items in long-term memory, a source of interference with timing?

Experiments here address whether activating a memory set interferes with timing; work of Conway and Engle (1994) already establishes activation is not related to complex span. In the Sternberg (1966) memory search task, a participant memorizes a short list, the memory set. A probe is then presented and the participant indicates whether or not the probe was present in the memory set. When a memory set has been learned so well that it is in long-term memory, but because of decay or interference is no longer in short-term memory, it must be activated before it can be searched (Wickens et al., 1981, 1985; Conway and Engle, 1994). The need for activation increases reaction time. Whether activation will affect concurrent timing or not is difficult to predict a priori because activation borders a process that interferes with timing (memory search) and a processes which does not (task switching). On the one hand, activating a memory set would seem to use some of the same resources as searching a memory set, which interferes consistently with time production (Fortin and Rousseau, 1987; Fortin et al., 2007; Rattat, 2010). On the other hand, activating a memory set when multiple sets have been learned is a switch from one memory set to another. Although there is a cost to switching memory sets (Humphreys et al., 2009), switching between digit classification and memory search did not interfere with time production (Fortin et al., 2010). Furthermore, activation is an automatic process in some theories (e.g., Anderson, 1983) and as such, should not interfere with other concurrent processes. Depending on what resources activation shares with memory search or with switching, activating a memory set would interfere or not with concurrent timing.

### **PROCESS ORGANIZATION**

We are also interested in memory set activation for a reason not directly related to what we have said so far, to consider predictions of a model of process organization originally proposed by Wickens et al. (1981), later used by Wickens et al. (1985) and Conway and Engle (1994). According to the model activating and searching a memory set are carried out successively (see the lower part of **Figure 1**). Wickens et al. (1981) gave participants a list to memorize, of size 2 or 4. They then presented a probe to search for in the list, either immediately or after an interval of counting backwards by threes. If the probe was presented after the interval, reaction time increased, attributed to the time needed to activate the memory set. When set size increased, reaction time also increased, explained by a longer time needed to search a larger memory set. The combined effect on reaction time of presenting the probe after an interval and of increasing set size was the sum of their individual effects. Such additivity was also found by Wickens et al. (1985). Additivity is explained if the participant first activates the memory set and then searches it. The time to complete the task is the sum of the durations of each process, so the combined effect of prolonging both processes is the sum of the individual effects (Sternberg, 1969).

In related experiments, Conway and Engle (1994) asked participants to memorize sets of different sizes. Then, on each trial, a cue was presented to indicate which memory set was relevant on the trial, followed by a probe to search for in the memory set. On some trials there was a delay between the cue and the probe, allowing time to activate the cued memory set. Conway and Engle (1994) found additive effects on reaction time of absence of the delay and of memory set size (set size 2 was sometimes an exception). Additivity is explained as by Wickens et al. (1981): the participant first activates the memory set and then searches it.

Such sequential organization, if it occurs, separates an effect of activation difficulty from an effect of search difficulty, facilitating an answer to the question of whether activation interferes with timing. If memory set activation and search are in series, and both are concurrent with time production, they are organized as in **Figure 1**. The organization is similar to that proposed for time production concurrent with a visual search task by Schweickert et al. (2007).

The primary issue here is whether activating information from long-term memory interferes with concurrent timing. Testing activation and search in memory allowed us to examine the secondary issue, whether activating and searching a memory set are executed successively or not. To study these issues, activating and searching a memory set were performed in two main conditions. In the Single-Task Condition the participant performed the search task alone, and reaction time was the main dependent measure. In the Dual-Task Condition, the participant performed the search task while concurrently producing a time interval, and time production was the main dependent measure. Errors in memory search were also analyzed. In both conditions, the participant sometimes had to activate the memory set. The Single-Task Condition allowed us to determine whether the need for activation increased the time to perform the task. The Dual-Task Condition allowed us to determine whether the need for activation interfered with timing. To explore the generality of results of Conway and Engle (1994), the first two experiments used a paradigm somewhat different from theirs. Participants memorized two short lists to a high criterion, so the lists were in secondary memory. To ensure that one particular list was always in the activated state, it was presented again at the start of each trial. Experiment 3 used the paradigm of Conway and Engle (1994).

## **EXPERIMENT 1**

### **MATERIALS AND METHODS**

Trials and blocks of trials had the same basic structure in Experiments 1 and 2. In both experiments, the participant memorized two sets, one of words and one of letters, at the beginning of each block of trials. Words were from the pool {BIB, CAR, CUB, DAM, DOG, HAT, HIP, KIT, KEG, MAN, MUD, PEN, PIT, RUG, SOD, TAB, TIN, WAX, WIG, ZIT}. The letter pool was the 20 consonants (excluding Y). One pool and the memory set formed from it had already been selected by the experimenter to be called active, the other pool and set formed from it to be called inactive.

At the beginning of each trial, the active set was presented again (see **Figure 2**, where the letter set is active). The inactive set was never presented after it was memorized. Then a probe was presented: a word or a letter. The task was to indicate whether the probe was present in either memory set or absent from both. (Logically, the probe could be present in at most one memory set). If a probe was from the active pool, the participant could search for the probe in immediate memory. But if the probe was from the inactive pool, the inactive set presented at the beginning of the block had to be activated before being searched. The pool the probe was from determined whether or not activation from long term memory was needed on a trial.

In Experiment 1, two factors were varied on trials: presence or absence of the probe in the memory set and whether the probe was from the active or inactive memory pool. Memory search was performed in a Single-Task Condition and in a Dual-Task Condition. A different group of participants was tested in each condition to avoid carry-over from one mode of responding to another.

### *Single-task condition*

*Participants.* Fourteen Purdue University undergraduates participated to partially fulfill an introductory psychology course requirement. Each was run individually in four sessions. The cutoff of 0.25 was the maximum error proportion allowed in each of the four factor combinations (probe present/absent and memory set active/inactive), averaged over all four sessions. Four participants exceeded this cutoff so their data were eliminated. (The memory task was more difficult than expected for the participants, as will be discussed later). Ages of the ten participants whose data were used (two men, eight women) ranged from 18 to 24 years (*M* = 19.9; *SD* = 1.85). Approval for human participants was obtained from the Purdue University Institutional Review Board. The experiment conforms to relevant regulatory standards. Informed consent was obtained from all participants.

*Stimuli and apparatus.* The experiment was controlled by E-Prime (1.1). Participants removed watches. They sat 60 cm in front of a computer screen. Reaction times and time productions were timed to the nearest millisecond. Responses were made with a button box (Psychology Software Tools), with the three leftmost buttons denoted left, middle and right respectively. Index, middle and ring fingers of the right hand rested on those buttons.

Letter sets consisted of five items selected randomly without replacement from the letter pool. Word sets were formed likewise, of size five. All stimuli were white on black background, Courier New, font size 18 pts.

*Design and procedure.* In each session six test blocks followed a practice block of trials. Each block had 25 trials. Participants were not told about the practice.

At the start of the learning phase of a block, a letter memory set and a word memory set appeared on the screen, one set at a time. Different sets were used in each block. On the first (practice) block, the set presented first was chosen at random. On half of the six test blocks, at random, the letter set appeared first; on the other half the word set appeared first. The five items of a memory set were presented in a vertical column, the first item (at the top) centered on the screen. The participant memorized the first memory set. Then the participant turned his or her head away from the screen and recited the items in order (top to bottom). The experimenter determined whether the set was recalled correctly. If recall was incorrect, the participant studied the set again. Recall was correct when all items were recited in order with no intrusions. When recall was correct, the participant pressed the middle button to display the next memory set. The participant memorized the second set and was tested as for the first. When finished, the participant pressed the middle button to start the search phase of the block.

In the search phase of a block, one of the two pools was chosen randomly to be active throughout the block. In three of the six test blocks, the letter pool was active. Each trial began with presentation of five asterisks in a row centered on the screen, indicating the participant could start the trial by pressing the middle button. When the middle button was pressed, "Remember" appeared, followed by the five items of the active memory set, one at a time, centered on screen. "Remember" and each item were presented for one second, with no delay between them. The last item was followed immediately by "Ready."

When ready, the participant pressed the middle button, starting a tone presentation. The probe ("C" in **Figure 2**) appeared when the tone began. The participant was instructed to ignore the tone (it was relevant in the Dual-Task Condition only). The participant then pressed the left button if the probe was present in either memory set and the right button if it was absent, under instructions to respond as quickly as possible. When the response was made, the tone ended and a row of five asterisks appeared on the screen, indicating a new trial starting.

For a given participant in a given session, probes were selected as follows. On the first trial of the practice block, the probe was randomly selected from one of the pools and was randomly present or absent. On the first trial of a test block, the probe was present on half the trials, at random. The first trial was not analyzed. On half of the 24 trials following the first, the probe was from the active memory pool and on half from the inactive memory pool, at random. On half of the 12 trials with the probe from the active pool, the probe was present in the active memory set and on half it was absent, at random. The same applied for the other 12 trials with the probe from the inactive pool.

Blocks of trials were separated by a 30 s pause. Each session lasted approximately 45 min. Two sessions were never on the same day or more than a week apart.

### *Dual-task condition*

*Participants.* Fifteen Purdue University undergraduates participated to partially fulfill an introductory psychology course requirement. Participants were run individually in four sessions. Data from four participants were dropped because of error proportions higher than the cutoff. Ages of the eleven remaining participants (two women, nine men) ranged from 18 to 22 years (*M* = 19.36; *SD* = 1.12). Approval for human participants was obtained from the Purdue University Institutional Review Board. The experiment conforms to relevant regulatory standards. Informed consent was obtained from all participants.

*Stimuli and apparatus.* These were identical to the Single-Task Condition.

*Design and procedure.* There were two parts in each session: practice at reproducing a time interval, then the list learning and search part.

Time production practice was in the first part of each session. A tone presented the interval, 2400 ms, five times and participants were asked to produce this duration. After these demonstration trials, the target interval was never presented again. The participant was to produce the same interval throughout the session. Each trial began with presentation of asterisks on the screen, indicating the participant could press the middle button to start time production when ready. When the middle button was pressed, a continuous tone was emitted indicating the start of interval production. The participant pressed the left or right button, as the participant wished, to end the tone when it was judged that the target time interval had elapsed. Feedback was given: if the produced interval was within a temporal window of 10% around the target duration (between 2280 and 2520 ms) the feedback was "correct." Otherwise, the feedback was "too long" or "too short," as appropriate.

Each of the three blocks of time production practice had 50 time-production trials. The third practice block was identical to the first two, but without feedback.

For the list learning and search part, design was as in the Single-Task Condition, except that in the search phase of a block the participant performed the memory search task concurrently with the time production task. On each trial of the search phase, the tone indicated the beginning of the 2400 ms interval to produce. The participant pressed the left or right button to end the tone when the target interval was judged to have elapsed. Instructions were to press the left button if the probe was from either of the two memory sets and to press the right button if it was absent from both. Each session lasted approximately 60 min.

In the Dual-Task Condition, one participant prematurely pressed the middle button by mistake in the list learning phase. Two blocks were thus invalidated and dropped.

### **RESULTS**

Reaction times and time productions averaged over trials in which responses were correct are in **Table 1**; percent errors are in **Table 2**.

### *Single-task condition*

For each session and participant, mean reaction times (RTs) for memory search correct trials and proportion of memory search errors were calculated in each combination of memory set active or inactive, probe present or absent. These means were averaged over sessions and the resulting mean RTs and mean error proportions were input to separate repeated measures ANOVAs with active/inactive and probe presence/absence as factors.

Mean RT was longer and error proportion was larger in the inactive than in the active condition, *F*(1, 9) = 42.36, *p* < 0.001,

#### **Table 1 | Experiment 1 reaction times and time productions.**


Standard deviations in parentheses, time in ms.

#### **Table 2 | Experiment 1 percent errors.**


Standard deviations in parentheses.

*MSE* <sup>=</sup> 688, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.83 and *<sup>F</sup>*(1, 9) <sup>=</sup> <sup>51</sup>.65, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, *MSE* <sup>=</sup> <sup>0</sup>.001, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.85, respectively. No other effects or interactions were significant.

There was not a significant effect of whether the probe was present or absent. Some experiments find such an effect on reaction time and some do not (e.g., Sternberg, 1975, **Figure 2**). Circumstances leading to a significant effect are not well understood.

### *Dual-task condition*

ANOVAs of the same form were conducted on time productions (TPs) and proportion of memory search errors. Activating a memory set had increased RT by 54 ms in the Single-Task Condition, but increased TP by only 14 ms, a non-significant effect, *<sup>F</sup>*(1, 10) <sup>=</sup> <sup>1</sup>.61, n.s., *MSE* <sup>=</sup> 1225, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.14. Power is high for rejecting at the 0.05 level the null hypothesis of no effect of activation on Time Production in the Dual-Task Condition. It was calculated with G\*Power 3 (Faul et al., 2007). For the alternative hypothesis of a small effect (Cohen's *f* = 0.10), power is 0.99996. For the power calculations, the non-sphericity correction ε is 1 and average correlation between repeated measurements in different conditions is 0.995. The power is sensitive to this correlation, which is notably high here, likely because the participant is trying to produce the same time interval every time.

Because the effect of memory set activation is important for the Timing and Complex-Span Hypothesis, we conducted an analysis from a different point of view. We compared two models accounting for the time productions with the Akaike Information Criterion (Akaike, 1974). Briefly, the Akaike Information Criterion (AIC) for a model is *AIC* = −2 ln(*L*) + 2*k*, where *L* is the likelihood and *k* is the number of parameters. The first term is smaller the better the goodness of fit of the model but the second term is larger the more parameters in the model. The AIC integrates a tradeoff between goodness of fit and number of parameters. The numerical value of the AIC is not informative on its own. But a set of models can be compared by selecting the one with smallest AIC. This is not an alternative way of doing a hypothesis test; rather it is a way of selecting the model in the set that is most parsimonious in representing the data.

The full model we considered has all main effects and interactions of the ANOVA that was conducted on time productions. The reduced model we considered omits the main effect of activation and all interactions involving activation. Analysis was done in R with the function lmer in the package lme4 (For discussion of model selection with AIC in R, see Venables and Ripley, 1994). Subjects was a random factor; other factors were fixed. Parameters were estimated with maximum likelihood. The reduced model had smaller AIC, *AIC* = 17.63. We conclude that the more parsimonious model does not include activation or interactions involving it.

Error proportion was higher in the inactive than in the active condition, *F*(1, 10) = 28.56, *p* < 0.001, *MSE* = 0.001, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.74. Other effects were non-significant for TP and errors.

We tested the difference in error proportion in the Singleand Dual-Task Conditions. An ANOVA in the same form as above but with the additional factor condition (Single- and Dual-Task) showed no effect of condition, *F*(1, 19) = 0.063, n.s., *MSE* <sup>=</sup> <sup>0</sup>.003, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.00 (percent errors were 7.38 and 7.05 in the Single- and Dual-Task conditions, respectively), and no interaction between condition and the active/inactive factor, *<sup>F</sup>*(1, 19) <sup>=</sup> <sup>0</sup>.20, n.s., *MSE* <sup>=</sup> <sup>0</sup>.001, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.01). There was no probe present/absent by condition interaction, *F*(1, 19) = 1.20, n.s., *MSE* <sup>=</sup> <sup>0</sup>.002, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.06.

### **DISCUSSION**

The objective of Experiment 1 was to test whether activating a memory set from long term memory would interfere with timing. Activation did not have a significant effect on concurrent timing in the Dual-Task Condition. In contrast, activating a memory set from long term memory increased RTs in the Single-Task Condition. Errors did not differ in the Dual- and Single-Task Conditions, showing that the dissociation cannot be explained by a speed-accuracy trade-off. With this paradigm using set size 5, we conclude that timing proceeds in the same way whether the memory set is active or inactive because we see no evidence of activating the memory set on time productions.

The percentage of trials in errors in the search task was relatively high. Averaged over conditions, there were errors in over 7% of the trials (**Table 2**). Furthermore, data from four participants of fourteen had to be eliminated because their errors exceeded the cutoff of 0.25 proportion of errors in each condition (see Participants section). Despite the difficulty, results are orderly. We see a clear effect of activating a memory set on reaction times, but not on time production.

Note that time productions were generally longer than the target interval to produce. This is a typical finding when time intervals are produced concurrently with other tasks. Our interpretation is the commonly accepted one that time productions are lengthened by general attention demands of a concurrent non-temporal task (Brown, 1997, 2006; Coull et al., 2004). The question of interest is not whether there is a general attention demand from the non-temporal task, but rather whether there is an additional specific demand due to activating a memory set. A lengthening specifically due to activating a memory set was not observed.

In Experiment 1 the memory set always contained five items. In Experiment 2, we tested the effect of varying set size along with the active-inactive manipulation. There were two reasons for investigating these two factors in Experiment 2. The first was to see whether the two factors would have different effects on produced intervals. One factor, increasing set size in memory search, consistently lengthens time intervals produced concurrently (Fortin and Rousseau, 1987; Fortin et al., 2007, 2010; Rattat, 2010). The other factor, activating a memory set, had no effect on concurrent timing in Experiment 1 here. Therefore, we predicted that time productions would lengthen with increasing memory set size, but that memory activation would have no effect.

The second reason for testing jointly set size and activation in Experiment 2 was to see whether the two factors would combine in our Single-Task reaction time condition in the pattern found by Conway and Engle (1994). They found additive effects of the two factors on RT, and additive effects on errors.

## **EXPERIMENT 2**

Conditions were as in Experiment 1 with a few exceptions. Experiment 2 used memory set sizes of 3 and 6. In each block of trials, one memory set was active and one inactive as in Experiment 1. If both sets were of size 3, the total number of items would be 6, and the participant could easily keep both sets active; the task would be equivalent to searching an active set of six items. To discourage this, we used two sizes, three and six, for which the sum exceeds short term memory capacity. Maintaining nine items active in memory being too difficult, we assumed participants would keep the inactive set in its inactive state during a block, activating it only when needed. Set size combinations used were size 3 active with size 6 inactive and size 6 active with size 3 inactive.

### **MATERIALS AND METHODS**

Method was as in Experiment 1, except memory set size varied, and slight modifications were made in the number of blocks and trials.

## *Single-task condition*

*Participants.* Fourteen Purdue University undergraduates completed the experiment in this condition, to partially fulfill an introductory psychology course requirement. Each was run individually in four sessions. Data from two participants were eliminated because their error proportions were higher than the cutoff. Error proportion was required to be less than 0.25 in each of eight factor combinations (probe present/absent, memory set active/inactive, size of searched memory set 3 or 6), averaged over all four sessions. Ages of the twelve remaining participants (five women, seven men) ranged from 18 to 21 years (*M* = 19.58; *SD* = 1.08). Approval for human participants was obtained from the Purdue University Institutional Review Board. The experiment conforms to relevant regulatory standards. Informed consent was obtained from all participants.

*Design and procedure.* Each session included nine 21-trial blocks. The first block and the first trial in each block were not analyzed. There were two memory sets for each block: one of three items and another of six items. On half of the eight test blocks, chosen randomly, the word set appeared first for learning. On half of the four blocks in which a word set appeared first, it was selected randomly to be the active memory set. In the two blocks in which the word set appeared first and was also selected as the active memory set, the word set for one block consisted of six items and that for the letters consisted of three. The same applied for the four blocks on which the letter set appeared first. On the first practice block, the memory set presented first, the active memory set, and the set size of the memory sets were all chosen randomly.

On the first trial of each block, the probe was selected as described in Experiment 1. On half the remaining 20 trials, the probe was from the active pool and on half from the inactive pool, randomly. On half of the 10 trials with a probe from the active pool, the probe was present in the memory set and on half it was absent, randomly. The same applied for the 10 trials with the probe from the inactive pool. Each session lasted approximately 40 min.

### *Dual-task condition*

*Participants.* Twenty Purdue undergraduates completed this condition to partially fulfill an introductory psychology course requirement. Data from five were dropped because error proportions were higher than the cutoff. Ages of the fifteen remaining participants (four women, eleven men) ranged from 18 to 23 (*M* = 19.33; *SD* = 1.45). Approval for human participants was obtained from the Purdue University Institutional Review Board. The experiment conforms to relevant regulatory standards. Informed consent was obtained from all participants.

*Design and procedure.* Design was as for the Dual-Task Condition of Experiment 1, but with set sizes and number of blocks and trials as in the Single-Task Condition above. Each session lasted approximately 60 min.

## **RESULTS**

Occasionally a participant pressed the middle button prematurely in the list learning phase, invalidating a block of trials. In the Single-Task Condition, two such blocks were dropped for one participant and one for another. In the Dual-Task Condition, two such blocks were dropped for one participant.

**Table 3** shows RTs and TPs averaged over trials in which responses in the search task were correct, with percent errors in memory search in **Table 4**. ANOVAs were performed as in Experiment 1, with Set Size an additional factor crossed with the

### **Table 3 | Experiment 2 reaction times and time productions.**


Standard deviations in parentheses, time in ms.

### **Table 4 | Experiment 2 percent errors.**


Standard deviations in parentheses.

other factors. Four separate ANOVAs were carried out, on RTs and error proportion in the Single-Task Condition, and on TPs and error proportion in the Dual-Task Condition.

### *Single-task condition*

RTs were longer in the inactive than in the active condition, *<sup>F</sup>*(1, 11) <sup>=</sup> <sup>11</sup>.08, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.01, *MSE* <sup>=</sup> 8885, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.50, and longer at set size 6 than 3, *F*(1, 11) = 13.44, *p* < 0.01, *MSE* <sup>=</sup> <sup>14</sup>, 700, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.55. The interaction between Set Size and Active/Inactive was not significant, *F*(1, 11) = 0.21, n.s., *MSE* <sup>=</sup> 3738, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.02.

The combined effect of Set Size and Active/Inactive is important because the model in which memory set activation precedes memory set search predicts additive effects of these factors on reaction time. We used the AIC to compare the full model for reaction time that has all main effects and interactions of the ANOVA previously conducted with a reduced model that omits the interaction of Set Size and Active/Inactive and all higher order interactions involving both factors. Analysis was done in R with the function lmer in the package lme4. Subjects was a random factor; other factors were fixed. Parameters were estimated with maximum likelihood. The reduced model had smaller AIC, *AIC* = 29.85. We conclude that the more parsimonious model does not include interactions involving set size and activation.

Error proportion was higher in the inactive than in active condition, *F*(1, 11) = 30.30, *p* < 0.001, *MSE* = 0.002, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.73, and higher at size 6 than 3, *<sup>F</sup>*(1, 11) <sup>=</sup> <sup>7</sup>.64, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.05, *MSE* <sup>=</sup> <sup>0</sup>.002, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.41. For errors, the interaction between active/inactive and set size was not significant, *<sup>F</sup>*(1, 11) <sup>=</sup> <sup>0</sup>.17, n.s., *MSE* <sup>&</sup>lt; <sup>0</sup>.001, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.02. Error proportion was higher when the probe was present than when it was absent, *F*(1, 11) = 19.69, *p* < 0.01, *MSE* = 0.003, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.64. Other effects were non-significant.

### *Dual-task condition*

For TPs, the main effect of active/inactive was not significant, *<sup>F</sup>*(1, 14) <sup>=</sup> <sup>1</sup>.99, n.s., *MSE* <sup>=</sup> 5459, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.12, but TPs were longer at set size 6 than at size 3, *F*(1, 14) = 11.96, *p* < 0.01, *MSE* <sup>=</sup> 2015, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.46.

The interaction between active/inactive and set size was significant, *<sup>F</sup>*(1, 14) <sup>=</sup> <sup>7</sup>.93, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.05, *MSE* <sup>=</sup> <sup>32</sup>, 089, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.36. The interaction is hard to interpret. To obtain details, simple main effects were tested. For set size 3, TPs were significantly longer for the inactive memory set, *F*(1, 14) = 8.04, *p* = 0.013, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.37. Mean TPs were 3419 ms and 3530 ms for active and inactive memory sets, respectively. For set size 6, TPs were significantly longer for the active memory set, *F*(1, 14) = 5.52, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.034, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.28. Mean TPs were 3539 ms and 3466 ms for active and inactive memory sets, respectively. Simple main effects of set size were also tested. When an active memory set was searched, TPs were significantly longer for set size 6, *<sup>F</sup>*(1, 14) <sup>=</sup> <sup>10</sup>.90, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.005, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.44. When an inactive memory set was searched, set size was not significant, *<sup>F</sup>*(1, 14) <sup>=</sup> <sup>4</sup>.31, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.057, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.24. (In case a correction for the number of *post hoc* tests is desired, *p* values are reported). The simple main effects of memory set activation are contrary to the Timing and Complex-Span Hypothesis, but it is hard to understand why the effect would go in opposite directions for different set sizes.

Error proportion was higher in the inactive than in the active condition, *F*(1, 14) = 73.60, *p* < 0.001, *MSE* = 0.002, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.84; at set size 6 than at set size 3, *<sup>F</sup>*(1, 14) <sup>=</sup> <sup>12</sup>.28, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.01, *MSE* <sup>=</sup> <sup>0</sup>.002, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.47; and when the probe was present rather than absent, *F*(1, 14) = 21.45, *p* < 0.001, *MSE* <sup>=</sup> <sup>0</sup>.002, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.61. No two-way interactions were significant. The three-way interaction was significant [*F*(1, 14) <sup>=</sup> <sup>7</sup>.65, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.05, *MSE* <sup>=</sup> <sup>0</sup>.001, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.35]. The three-way interaction has the following form. In **Table 4**, Dual-Task Condition, errors are always higher for probe present than absent, always higher for inactive memory set than active, and higher for set size 6 than 3 except for the single case of probe present, inactive memory set.

If processing in the memory task were done the same way in the Single- and Dual-Task conditions, error proportions would be comparable. To test this, an ANOVA was performed on the error proportions combining the two conditions. The ANOVA had form as those above, but with the additional factor of Condition (Single- vs. Dual-Task) crossed with the other factors. Condition was not significant, *F*(1, 25) = 3.21, n.s., *MSE* = 0.006, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.11. There was no significant interaction of Condition with any other factor or combination of factors. As far as one can determine from errors, processing the memory set was performed the same way in both conditions.

The main results expected in Experiment 2 were that (1) memory activation and increased set size would increase RTs in the Single-Task Condition, and that (2) time productions would lengthen with set size, but not with memory activation in the Dual-Task Condition. As expected, RTs increased with set size and were longer with memory activation. Even though on average time productions lengthened with set size and did not differ in the active and inactive conditions, an interaction was observed, showing opposite effects of activation for different set sizes. Before discussing this puzzling result, we consider the second objective of Experiment 2. Specifically, effects of activation and set size will be examined to test process organization.

### *Process organization*

In the model of Wickens et al. (1981), Wickens et al. (1985), and Conway and Engle (1994), the memory set is activated and then it is searched (see lower part of **Figure 1**). If two factors selectively influence two processes in series, the factors are predicted to have additive effects on reaction time (Sternberg, 1969). It is sometimes thought that for the Additive Factor Method to apply, errors must be the same in all conditions, or responses must be speeded, but such stringent conditions are not needed (Schweickert, 1985; Schweickert et al., 2012). As the model predicts, the two factors, active/inactive memory set and set size, have significant and additive effects on reaction time in the Single-Task Condition. (Probe presence/absence had no effect on reaction time). An analogous non-significant interaction was also found by Wickens et al. (1981), Wickens et al. (1985) and Conway and Engle (1994), set size two sometimes an exception in the last study. These results support the serial organization of activation and search.

Conway and Engle (1994) also reported that factors influencing activation and search had additive effects on error probability. This can be explained with the same serial process organization (Schweickert, 1985; Schweickert et al., 2012). Suppose the probability of a correct response equals.

P[Correctly Activate Memory Set]

× P[Correctly Search|Correctly Activate Memory Set].

Now suppose one factor changes the probability of correctly activating the memory set, and another factor changes the probability of correctly searching, given correct activation of the memory set, each factor changing only one probability. Then the combined effect of the two factors on probability correct is the product of their individual effects. Multiplicative effects on probability correct could have been manifest as additive, through the following approximation. Multiplicative effects on probability correct correspond to additive effects on the logarithm of probability correct. But the natural log of a relatively large probability *P* is approximately equal to −(1 − *P*). For example, log 0.95 = −0.051. Suppose one process is correct with probability *p*, another is correct with probability *q*, and the probability of a correct response is *r* = *pq*. If one factor changes *p* and another factor changes *q* the factors will have multiplicative effects on probability of a correct response. If *r* = *pq*, then log *r* = log *p* + log *r*. A little algebra shows that if the probabilities are relatively large, the multiplicative effects predict approximately additive effects.

A model in which active/inactive memory set and set size have multiplicative effects was fit to frequencies of correct responses; see Appendix A. Predicted and observed values are quite close in both the Single-Task and Dual-Task Conditions (**Table 5**).

For comparison, a model in which the two factors have additive effects was also fit. For both models the goodness-of-fit statistic, *G*2, has approximately a chi-square distribution with 1 *df*. In the Single-Task condition, for the multiplicative model *<sup>G</sup>*<sup>2</sup> <sup>=</sup> <sup>0</sup>.47 and for the additive model *<sup>G</sup>*<sup>2</sup> <sup>=</sup> <sup>0</sup>.32. The small values of *G*<sup>2</sup> indicate that both models fit very well. Parameters were estimated to minimize *G*2, so the AIC for a model equals *G*<sup>2</sup> plus the



If Active/Inactive Memory Set and Set Size Have Multiplicative Effects in Experiment 2.

number of parameters (see, e.g., Moshagen, 2010). The number of parameters is the same, 3, for each model. The additive model has slightly smaller AIC, *AIC* = 0.15. The additive model is more parsimonious, but negligibly so.

In the Dual-Task condition, for the multiplicative model *<sup>G</sup>*<sup>2</sup> <sup>=</sup> <sup>2</sup>.43 and for the additive model *<sup>G</sup>*<sup>2</sup> <sup>=</sup> <sup>8</sup>.54. The multiplicative model has smaller AIC,*AIC* = 6.11. The multiplicative model fits well and is more parsimonious than the additive model.

Reaction times in the Single-Task Condition and accuracy in both the Single- and Dual-Task conditions are all consistent with the process organization of activation preceding search. (The order of these two processes is not established, but it seems more natural for activation to precede search than the reverse).

Two objections to the multiplicative model for accuracy may be raised. First, the model is fit to averages over participants. But the average of a product does not equal the product of the average of the multiplicands, if the multiplicands are correlated. In response, we note that in our data the correlations are low or moderate. For proportion correct the average correlation between repeated measures across combinations of factor levels is 0.11 in the Single-Task Condition and 0.41 in the Dual-Task Condition. A second objection is that with a multiplicative model for accuracy factor effects are not additive, but significant interactions were not found between set size and activation in the ANOVAs. Further, in the Single-Task Condition, a multinomial tree model with additive effects is more parsimonious (albeit barely) than a multiplicative model. In response we note that the sizes of the interactions predicted by the multiplicative model are quite small for proportion correct: 0.001 for the Single Task Condition and 0.002 for the Dual Task Condition. With such small interactions predicted by the multiplicative model, it is not surprising that an additive model can perform well. Further, a multiplicative model has a natural interpretation: the probability of a correct response is the probability of correct activation followed by correct search given the correct activation.

### **DISCUSSION**

Experiment 2 was informative about our secondary objective, to test the model in which memory set activation precedes search. In the Single-Task Condition, effects on RT of memory set activation and set size were additive, supporting the model. The model was further supported because the equation in which the factors have multiplicative effects on proportion correct fit well. In the Dual-Task Condition, the multiplicative equation also fit effects on proportion correct well.

Information from Experiment 2 about our primary objective, testing whether memory set activation interferes with timing, is complicated by an interaction on TPs between memory set activation and set size.

### *A puzzling result*

For time productions, it is hard to interpret the interaction of memory set activation and set size. When an inactive memory set was searched, it is peculiar that produced intervals were numerically shorter when set size was 6 than when it was 3. The direction is surprising because in the Single-Task Condition, reaction times were significantly longer with higher set size. Increasing set size has consistently lengthened produced intervals previously (e.g., Fortin and Rousseau, 1987; Neath and Fortin, 2005; Fortin et al., 2007). The interaction may have something to do with the high difficulty level of the task. Participants had to memorize two new memory sets on each block. High error proportions led to dropping data from several participants. Difficulty may have led participants to terminate temporal productions too quickly in the inactive memory set condition, when set size was 6.

To pursue this puzzling result, one would want to investigate a wider range of memory set sizes. But using a range of set sizes is not feasible with the paradigm of Experiments 1 and 2. A participant learns two memory sets and the active one is presented again at the start of every trial. The active and inactive memory sets are treated differently, so using more than two set sizes would require considerable counterbalancing. A paradigm treating memory sets more symmetrically would be more suitable.

The paradigm of the two first experiments has been useful. It allowed us to observe different effects for active and inactive memory sets, with systematic effects on reaction time and accuracy data. A clear dissociation was found in Experiment 1, with longer reaction times in the inactive condition, but unchanged time productions. In Experiment 2, evidence from reaction times and errors was consistent with the model in which the two factors selectively influenced sequential processes. However, error proportions were high, time productions in Experiment 2 showed an unusual shortening of produced intervals with increasing set size in the inactive memory set condition, and the paradigm cannot be efficiently used for testing more than two memory set sizes. For these reasons, a related but different paradigm of Conway and Engle (1994) was used in Experiment 3.

## **EXPERIMENT 3**

Experiment 3 tested the effect of activating information in memory on concurrent timing, using a memory task of Conway and Engle (1994, Experiment 4). The participant memorized four memory sets, to a stricter criterion than in our Experiments 1 and 2. Three factors varied: the delay between a cue indicating which set to search and a probe, set size, and presence/absence of the probe in the set. A delay between cue and probe allowed the participant time to activate the appropriate memory set in advance, so it could be searched immediately when the probe appeared. On a trial with no delay, the participant presumably had to activate the memory set in order to search it when the probe appeared.

### **MATERIALS AND METHODS**

Method is as in Experiments 1 and 2 with exceptions described below.

### *Single-task condition*

*Participants.* Ten participants (four men, six women), between 18 and 41 years old (*M* = 24.8; *SD* = 6.68) and recruited through advertisement at Laval University, were given a C\$5 honorarium for one session of about 50 min. Approval for human participants was obtained from the Human Research Ethics Committee at Laval University (CÉRUL). The experiment conforms to relevant regulatory standards. Informed consent was obtained from all participants.

*Stimuli and apparatus.* Responses used the three leftmost keys of a five-key response box. Visual stimuli were presented on a 17 in. monitor about 70 cm in front of the participant. They were in white letters on a black background, Courier New, font size 18 points.

The word pool was 42 four-letter French words (Appendix B), from the high-frequency words in the OMNILEX database (University of Ottawa). Word sets included 3, 4, 5, or 6 different words. For each participant, 18 words were selected randomly from the word pool and assigned randomly to the four sets. A digit, "3," "4," "5," or "6" was presented with each set, the number of words in the set. Each set was presented in a vertical column, centered on the screen, below the digit. Sets of 3, 4, 5, and 6 items were displayed for 40, 50, 60, and 70 s respectively.

*Design and procedure.* In the list-learning phase, the four sets were successively presented in random order. After two presentations of the four sets, the experimenter asked the participants to recall each, identifying the sets with their digits (e.g.: "Please recall the words in the list containing four words."). Recall was correct when the participant recalled the words (in any order) in the set three times successively. The sets were recalled in random order, but the last studied set was never presented first for recall. After testing the four sets, sets with mistaken recall were presented anew, until all sets were recalled three times successively with no error.

The list-search phase then began with a last successive presentation of all sets. Pressing the middle button started the trial with "+," displayed for 1 s, then replaced by a digit above a probe centered on the screen. The digit could be presented simultaneously with the probe (no-delay) or one second before the probe (delay). Participants were asked to indicate as quickly as possible whether the probe word was or not in the list identified by the digit by pressing a left or right key on the response box. One second after the response, the next trial began with "+." There was a single block of 192 trials with no pause.

Set size, delay, and probe presence/absence were determined randomly on each trial, with levels of these factors balanced in the block. If the probe was absent from the set to be searched, it was member of one of the other three sets.

### *Dual-task condition*

*Participants.* Thirteen participants (five men, eight women), between 21 and 38 years old (*M* = 24.5; *SD* = 4.86) completed one session of about 50 min. Approval for human participants was obtained from the Human Research Ethics Committee at Laval University (CÉRUL). The experiment conforms to relevant regulatory standards. Informed consent was obtained from all participants.

*Design and procedure.* Design was as in the Single-Task Condition except that participants were asked to respond to the probe when the tone duration had reached the previously learned target interval (2400 ms).

### **RESULTS AND DISCUSSION**

RTs and TPs averaged over correct memory search responses are in **Table 6** and percent errors in **Table 7**.



Standard deviations in parentheses, time in ms.

#### **Table 7 | Experiment 3 percent errors.**


Standard deviations in parentheses.

### *Single-task condition*

Mean RTs in correct trials and error proportions were calculated at each combination of delay (no-delay, 1 s delay), memory set size (3, 4, 5, or 6 items) and probe (present, absent) factors. These means were input to separate repeated-measure ANOVAs on RTs and error proportions.

RTs were longer in the no-delay than in the delay condition, *<sup>F</sup>*(1, 9) <sup>=</sup> <sup>61</sup>.19, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, *MSE* <sup>=</sup> <sup>41</sup>, 852, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.88. RTs were 261 ms longer in the no-delay condition, which we interpret as the time required to activate the identified memory set. We note that the effect of activation is larger in this experiment than in the previous two. Wickens et al. (1985) noted different sizes of the effect of activation in their two experiments. In our case the difference may be due to four sets in long term memory in Experiment 3, more than in our previous two experiments. RTs were longer in probe-absent trials, *F*(1, 9) = 25.77, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.01, *MSE* <sup>=</sup> <sup>37</sup>, 261, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.74. RTs changed with increasing set size, *F*(3, 27) = 12.86, *p* < 0.001, *MSE* = 53, 949, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.59. In **Table 6** RT tends to increase with Set Size, but drops at set size 6. There is a significant linear trend, *<sup>F</sup>*(1, 9) <sup>=</sup> <sup>30</sup>.64, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, *MSE* <sup>=</sup> 30187, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.77. But there is also a significant quadratic trend, *F*(1, 9) = 14.29, *p* < <sup>0</sup>.01, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.61. We do not know why the decline at highest set size occurs, but it also occurred in three of the four experiments that used this paradigm in Conway and Engle (1994, Figures 3, 4, 6).

No interactions were significant, including that between delay and set size, *F*(3, 27) = 1.49, n.s., *MSE* = 31, 021, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.14. The non-significant interaction between delay and set size is consistent with the model used by Wickens et al. (1981), Wickens et al. (1985), and Conway and Engle (1994), in which the memory set is activated and then searched.

The combined effect of set size and active/inactive is important because the model in which memory set activation precedes memory set search predicts additive effects of these factors on RT. We used the AIC to compare the full model for RT, which has all main effects and interactions of the ANOVA previously conducted, with a reduced model, which omits the interaction of set size and active/inactive and all higher order interactions involving both factors. Analysis was done in R with function lmer in package lme4. Subjects was a random factor; other factors were fixed. Parameters were estimated with maximum likelihood. The reduced model had smaller AIC, *AIC* = 15.19. We conclude that the more parsimonious model does not include interactions involving set size and activation.

The ANOVA on error proportion showed no effect of delay, set size or probe presence/absence, nor any interaction.

### **DUAL-TASK CONDITION**

Effects on time productions of memory search are markedly different from those of memory activation (**Table 6**). The increase in TPs produced by increasing memory set size from three to six, 131 ms, is about two-thirds of the increase produced on RT, 198 ms. On the other hand, the increase in TPs produced by activating a memory set, 24 ms, is less than a tenth of the increase produced on RT, 261 ms.

ANOVAs as in the Single-Task Condition were performed in Dual-Task Condition. Set size had a significant effect on TPs, *<sup>F</sup>*(3, 36) <sup>=</sup> <sup>5</sup>.63, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.01, *MSE* <sup>=</sup> <sup>36</sup>, 318, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.32, an effect consistently observed (Fortin and Rousseau, 1987; Fortin et al., 2007; Rattat, 2010), and interpreted to mean timing slows or pauses when an active memory set is searched. In **Table 6**, TPs tend to numerically increase with set size, with a leveling off or decline at the highest set size. There is a significant linear trend of TP with set size, *F*(1, 12) = 15.71, *p* < 0.01, *MSE* = 26, 890, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.57. The quadratic trend is not significant, but nearly so, *<sup>F</sup>*(1, 12) <sup>=</sup> <sup>4</sup>.19, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.06, *MSE* <sup>=</sup> <sup>38</sup>, 556, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.26. The pattern is like that of the reaction times.

TPs were longer in probe-absent than in probe-present trials, *<sup>F</sup>*(1, 12) <sup>=</sup> <sup>7</sup>.47, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.05, *MSE* <sup>=</sup> <sup>73</sup>, 507, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.38, an effect sometimes observed when memory search is performed concurrently with time production (e.g., Fortin and Rousseau, 1987).

However, the factor corresponding to memory set activation, delay, had no significant effect on TPs, *F*(1, 12) = 0.64, n. s., *MSE* <sup>=</sup> <sup>47</sup>, 166, partial <sup>η</sup><sup>2</sup> <sup>=</sup> <sup>0</sup>.05. Power for rejecting at the 0.05 level the null hypothesis of no effect of activation on TPs was calculated with G∗power3 (Faul et al., 2007). If the true effect is small (Cohen's *f* = 0.10), the power is 0.84. (For power calculation, the average correlation between repeated measurements is 0.983, the non-sphericity correction ε is 1). Power is sensitive to the correlation between repeated measurements, which is high here.

Because the effect of activation on time production is important for the Timing and Complex-Span Hypothesis, we compare the full model, having all main effects and interactions of the ANOVA previously done, with a reduced model omitting activation and all higher order interactions involving activation. Analysis was done in R with function lmer in package lme4. Subjects was a random factor, other factors were fixed. Parameters were estimated with maximum likelihood. The reduced model had smaller AIC, *AIC* = 47.92. We conclude that the more parsimonious model does not include activation or interactions involving it.

There were no other significant effects in this analysis. There were no significant effects in the ANOVA on error proportions.

In this paradigm, activating a memory set and searching the memory set both take time and presumably involve shortterm memory. Nonetheless, only memory search interferes with timing.

### **GENERAL DISCUSSION**

Participants reproduced a time interval concurrently with performing a memory search task. We tested whether activating the memory set interferes with timing. No interference was found in Experiment 1. In Experiment 2, where memory activation was manipulated jointly with memory set size, timing results showed unexpected and opposite effects of memory activation in the low and high memory load conditions. Error data suggested that this complex pattern of disruption of timing may have been produced by extreme difficulty in processing the inactive memory set. This led us to use a related but different paradigm in Experiment 3. This paradigm, borrowed from Conway and Engle (1994), resulted in lower error proportions. Predictions regarding timing productions were confirmed under those conditions: no interference due to activation, while in contrast, time productions lengthened with increasing set size (as in Fortin and Rousseau, 1987 and Fortin et al., 2007). Results overall favor the conclusion that activating a memory set does not interfere with concurrent timing. Results from Experiment 2 suggest this statement must be restricted to conditions where memory search difficulty is moderate. Overall, results support the Timing and Complex-Span Hypothesis, because performance on activating a memory set is not related to Complex-Span (Conway and Engle, 1994).

An interpretation of the time production interference due to memory search in terms of a widely used accumulation model of timing (Gibbon et al., 1984; Zakay and Block, 1996; Brown, 2006; Buhusi and Meck, 2009) is that searching a memory set interferes with at least one timing mechanism requiring attention or memory. In the present study, a target interval is presented to the participant, who estimates its duration and stores it in memory. The model assumes that when the interval is presented pulses are emitted by a "pacemaker" and accumulated. This process requires constant attention. An attention-controlled gate (Zakay and Block, 1996) or switch (Gibbon et al., 1984, see Lejeune, 1998) allows transfer of temporal information from the pacemaker to the accumulator if attention is devoted to time. When the interval ends, the output of the accumulator may be stored in working memory and then transferred to long-term memory in the form of a criterion, a pulse count that will be used later when producing the target interval. In experimental trials, production begins with a key press, and pulses must again be accumulated until the criterion is reached. Accumulation again requires continuous attention, and the production is ended with a second key press when it is judged that the accumulated pulse count corresponds to the criterion. At any moment during accumulation, the current accumulated count must be continuously compared to the criterion in long-term memory.

Attention is critical to hold the gate (or switch) to the accumulator so pulses are accumulated, and memory is required to store output from the accumulator, to store the criterion, and to compare the output with the criterion. Increase of produced intervals with increasing memory set size suggests that searching short-term memory disrupts the accumulation process, delaying the time when the criterion is reached. However, in Experiments 1 and 3 activating a memory set that is in long-term memory did not interfere with any of the timing mechanisms. This suggests that those mechanisms requiring attention or memory are involved in memory search but not in memory activation.

A result in the previous literature on memory search is contrary to the Timing and Complex-Span Hypothesis that a process interferes with concurrent timing if and only if process performance is related to complex span. Because the result is evidence against the hypothesis we propose, we describe the details. The relation between complex span and short-term memory search was investigated by Conway and Engle (1994). Their paradigm for memory search was described here in the introduction to Experiment 3. In their Experiments 1 and 2 the four memory sets to be searched had items in common, while in Experiments 3 and 4 the four memory sets were pairwise mutually exclusive, a difference that lead to different results. In their Experiments 1 and 2, the effect of set size on RT was greater for low- than highcomplex-span individuals on target present trials. This supports the Timing and Complex-Span Hypothesis.

In their Experiments 3 and 4, however, there was no interaction between span and set size on RT or errors. This result is contrary to our hypothesis because in Experiment 3 here, we found greater interference in timing with larger memory sets, despite the memory sets being pairwise mutually exclusive.

From the difference between their first two experiments and their last two, Conway and Engle (1994) concluded that complexspan is related to search of a short-term memory set only when there is competition among the memory sets that are possibly relevant; during search of short-term memory, the executive component of working memory capacity is needed only to inhibit irrelevant information.

If we pursue this reasoning for timing, the reason short-term memory search interferes with concurrent timing when memory sets do not overlap is not because of demand for the executive component. By elimination, the interference must be due to the demand for short-term memory capacity.

The lack of interaction between complex-span and memory set size in Experiments 3 and 4 of Conway and Engle (1994), if it is replicable, requires a modification of our hypothesis. A suitable revision is that if performance of a process is related to complexspan, then the process interferes with concurrent timing.

### **ORGANIZATION OF ACTIVATION AND MEMORY SEARCH**

According to a model proposed by Wickens et al. (1981) memory set activation is carried out before memory set search. The model predicts that a factor selectively influencing activation and a factor selectively influencing search would have additive effects on reaction time (Sternberg, 1969). Non-significant interactions found by Wickens et al. (1981), Wickens et al. (1985), and Conway and Engle (1994) support the model, as do non-significant interactions here in Experiments 2 and 3.

The model also predicts two such factors to have multiplicative effects on the probability of a correct response (Schweickert, 1985). Here a multiplicative model fit accuracy data from Experiment 2 well. (Multiplicative effects were not tested in Experiment 3 because in neither the Single-Task nor Dual-Task Condition did both factors have a significant effect on errors). Conway and Engle (1994) report set size and delay as having additive effects on error probability. We interpret such additivity as occurring because when probability of a correct response is high, multiplicative effects are approximately additive (Schweickert, 1985; Schweickert et al., 2012).

Activation is like moving a pointer to the memory set. Reaction time and accuracy results reveal a clear distinction between activation of a memory set and searching that memory set, and time production results reveal a clear dissociation between them.

### **CONCLUSIONS**

Experiments here on memory set activation support on the whole the Timing and Complex-Span Hypothesis that a process interferes with concurrent timing if and only if process performance is related to complex-span. A result of theoretical importance contrary to the Hypothesis is from Experiments 3 and 4 of Conway and Engle (1994), see the General Discussion. For process organization, data here are consistent with the model of Wickens et al. (1981) in which a memory set is in long term memory is activated and then searched. Our results suggest that when ongoing internal events, such as mental time travel, produce cues that activate information in memory, concurrent timing and complex span are not affected unless the activated information is used.

### **ACKNOWLEDGMENTS**

We are especially grateful to Thomas S. Redick and James M. Broadway, who brought to our attention parallels between processes unrelated to complex span and those not interfering with timing. We thank Motonori Yamaguchi for programs for Experiments 1 and 2 and Andrée-Anne Ouellet for help collecting data in Experiment 3. Portions of this work were supported by AFOSR grant FA9550-06-0383 to Schweickert, and AFOSR grant FA9550-09-0252 to Schweickert and Dzhafarov, and a grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) to Fortin.

### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 17 February 2014; accepted: 01 July 2014; published online: 28 July 2014. Citation: Schweickert R, Fortin C, Xi Z and Viau-Quesnel C (2014) Parallel effects of memory set activation and search on timing and working memory capacity. Front. Psychol. 5:779. doi: 10.3389/fpsyg.2014.00779*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Schweickert, Fortin, Xi and Viau-Quesnel. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## **APPENDIX A**

Experiment 2 Model Fitting: Activating a Memory Set and Its Set Size Have Multiplicative Effects on Probability Correct

Let *pij* be the probability of a correct response when the factor active/inactive memory set is at level *i* and the factor set size is at *j* items. The multiplicative model predicts

$$p\_{\vec{\eta}} = a\_{\vec{\imath}} \mathbf{s}\_{\vec{\jmath}},$$

where *ai* is the probability the memory set is activated correctly and *sj* is the probability the search is carried out correctly. Parameters were estimated with Excel Solver to minimize log likelihood, *G*2, (e.g., Bishop et al, 1975). Because parameter values are not uniquely determined, the probability the active memory set was activated correctly was arbitrarily set to *a*<sup>1</sup> = 0.999999.

With this setting, other parameter values are uniquely determined.

For the Single-Task Condition, estimated parameters were *a*<sup>2</sup> = 0.947181 (inactive memory set activated), *s*<sup>3</sup> = 0.977595, and *s*<sup>6</sup> = 0.953990. For the Dual-Task Condition, estimated parameters were *a*<sup>2</sup> = 0.927115 (inactive memory set activated), *s*<sup>3</sup> = 0.971567, and *s*<sup>6</sup> = 0.941086. Predicted frequencies were calculated with these parameters (**Table 5**).

## **APPENDIX B**

Experiment 3 Memory Set Pool

Midi, Papa, Faim, Jour, Lait, Cinq, Rien, Tête, Auto, Main, Café, Gars, Pain, Bain, Loin, Soif, Prix, Ciel, Film, Sept, Idée, Hier, Prof, Reçu, Aide, Bien, Mère, Fête, Bras, Bébé, Dire, Noir, Fait, Test, Soir, Sexe, Côté, Bleu, Pied, Gens, Date, Mâle

## Processing of sub- and supra-second intervals in the primate brain results from the calibration of neuronal oscillators via sensory, motor, and feedback processes

## *Daya S. Gupta\**

Department of Biology, Camden County College, Blackwood, NJ, USA

### *Edited by:*

James M. Broadway, University of California, Santa Barbara, USA

### *Reviewed by:*

Lihan Chen, Peking University, China Rebecca Schaefer, University of California, Santa Barbara, USA

### *\*Correspondence:*

Daya S. Gupta, Department of Biology, Camden County College, P.O. Box 200, College Drive, Blackwood, NJ 08012-0200, USA e-mail: dayagup@gmail.com

The processing of time intervals in the sub- to supra-second range by the brain is critical for the interaction of primates with their surroundings in activities, such as foraging and hunting. For an accurate processing of time intervals by the brain, representation of physical time within neuronal circuits is necessary. I propose that time dimension of the physical surrounding is represented in the brain by different types of neuronal oscillators, generating spikes or spike bursts at regular intervals. The proposed oscillators include the pacemaker neurons, tonic inputs, and synchronized excitation and inhibition of interconnected neurons. Oscillators, which are built inside various circuits of brain, help to form modular clocks, processing time intervals or other temporal characteristics specific to functions of a circuit. Relative or absolute duration is represented within neuronal oscillators by "neural temporal unit," defined as the interval between regularly occurring spikes or spike bursts. Oscillator output is processed to produce changes in activities of neurons, named frequency modulator neuron, wired within a separate module, represented by the rate of change in frequency, and frequency of activities, proposed to encode time intervals. Inbuilt oscillators are calibrated by (a) feedback processes, (b) input of time intervals resulting from rhythmic external sensory stimulation, and (c) synchronous effects of feedback processes and evoked sensory activity. A single active clock is proposed per circuit, which is calibrated by one or more mechanisms. Multiple calibration mechanisms, inbuilt oscillators, and the presence of modular connections prevent a complete loss of interval timing functions of the brain.

**Keywords: cerebellum, neuronal clock, polysensory processing, parietal, schizophrenia, interval timing**

## **INTRODUCTION**

An important challenge for the modern neuroscience is to understand the mechanistic basis of the interval timing by the brain in the sub- and supra-second range (Wittmann, 1999; Lewis and Miall, 2003; Buhusi and Meck, 2005; Bueti et al., 2008; Coull et al., 2011). A steady output of the scientific literature over past several years has improved our understanding of the timing mechanisms. Old as well as new types of evidence have emerged, implicating multiple regions of the brain in the processing of time intervals (Harrington et al., 1998; Wittmann, 1999; Bueti et al., 2008; Coull et al., 2011; Parsons et al., 2013). Different regions involved in the perception of time intervals include the posterior parietal lobe, frontal lobe, insula, basal ganglia, and cerebellum (Harrington et al., 1998; Bueti et al., 2008; Coull et al., 2011; Teki et al., 2011a). It is generally agreed that multiple mechanisms are responsible for the measurement of time intervals by the brain (Lewis and Miall, 2003; Wittmann, 2009; Teki et al., 2011a). However, a consensus about the mechanisms has not emerged, which is clear from various models that have been proposed to explain the processing of various time intervals by the brain (Buonomano and Merzenich, 1995; Maass et al., 2002; Matell and Meck, 2004; Karmarkar and Buonomano, 2007; Wittmann, 2009; Teki et al., 2011a).

The processing of different time intervals plays an important role in different activities of the brain (Buhusi and Meck, 2005). The sub-second range timing (milliseconds) plays role in the motor control and the speech production (Buhusi and Meck, 2005). The supra-second range (seconds to minutes) interval timing plays role in foraging, decision making, and mental estimation of time, but it tends to be less accurate than the timing of sub-second intervals (Buhusi and Meck, 2005). In an example of a cricket fielder catching a ball during the mid-air trajectory (**Figure 1**), several time intervals are processed within the brain, both at the conscious and subconscious levels. The temporal intervals processed at a conscious level include anticipatory interval for the arrival of ball in the world-centric view and for executing motor movements to catch a ball, both in the supra-second range. Another time interval processed is the perception of the elapsed duration during an act of catching a ball (sub-to supra-second range). The subconscious processing of time intervals helps in the determination of speed of various motor movements, necessary for the successful execution of a task. For example, elapsed duration for the contraction of a muscle will determine the speed of a muscle contraction. Time intervals processed in muscle and joint movements are generally in the milliseconds range. Time intervals in above example

are absolute, duration-based, which contrasts with the beat-based timing, which is relative to a temporal regularity, such as rhythmic beats (Teki et al., 2011a). It is shown that different networks in the brain are involved in both types of interval timing (Teki et al., 2011b).

In contrast to a much longer (24 h) circadian rhythm, which relies on the rhythmic expression of a set of clock genes with specific daily profiles in the suprachiasmatic nucleus and other parts of the brain (Rath et al., 2014), the interval timing in the subto supra-second range mainly relies on neuronal mechanisms. To understand the mechanisms underlying the neuronal clocks, one should first look at the mechanisms of the mechanical and electronic clocks. The mechanical and electronic clocks have inbuilt oscillators, which generate pulses corresponding to a standard unit of time. Both oscillators are based on a standard periodic event, such as the vibration of a quartz crystal or an oscillation of a pendulum. Based on the above analogy to the mechanical and electronic clocks, it is likely that the mechanism of neuronal clocks also includes an oscillator.

## **OUTLINE OF THE PROPOSED NEURONAL CLOCK MECHANISM**

I propose a modular clock mechanism for the processing of the time interval in the sub- and supra-second ranges, which is based on the generic view of timing mechanism promoted by Ivry and Schlerf (2008), called "intrinsic models" (Ivry and Schlerf, 2008). Intrinsic models assume that "there is no specialized system for representing temporal information in the brain, asserting that time is inherent in neural dynamics" (Ivry and Schlerf, 2008). The proposed neuronal clock is formed by two main modules (see the schematic in **Figure 2**). The first module is formed by a neuronal oscillator. Neuronal oscillators generate temporally regular activity from the firing of the action potentials. I propose that the oscillators are built within different circuits across the brain, consistent with the intrinsic nature of timing mechanisms (Ivry and Schlerf, 2008). Temporal information is processed by the modular neuronal clocks, when the activity of the neuronal oscillator excites the proposed frequency modulator (FM) neurons (**Figure 3**) in the second module, which are endowed with special conductances, or are connected by synapses exhibiting plasticity. Special conductance properties, connections or even the nature of the input, such as recurrent excitation can produce modulation in the activity of FM neurons. The state of the circuit in the second module, determined by the frequency, or the rate of the change of frequency is proposed to code the time intervals in the brain.

Two important circuits, likely to contain the second module, forming neuronal clocks, include (i) the dorsal stream responsible for sensorimotor tasks, connecting the posterior parietal cortex and the motor, premotor and prefrontal areas of frontal cortex (Kaas et al., 2011), and (ii) cerebro-cerebellar loops that are responsible for the motor, and non-motor functions, for example working memory, executive tasks, and emotion (Strick et al., 2009; Bostan et al., 2013). The state of the circuit in the second module, following modulation in the activities of FM neurons, will code for various temporal parameters, such as the perception, reproduction and production of time intervals, and the speed of a motor movement.

The primary role of the oscillators in the modular clock mechanism is to represent the external physical time. In order to accurately represent the external physical time intervals, the oscillators are calibrated via tasks that involve a direct or indirect interaction with the external world. Since the oscillators are inbuilt, their intrinsic regular rhythm will be superimposed with fluctuations caused by feedback processes, as well as the sensory inputs, involved in the normal functioning of a circuit. In the next section, I will discuss (a) how the external time is represented by neuronal oscillators within circuits in the brain and (b) various types of regular neuronal activities that could potentially play role as the neuronal oscillators in the modular clocks.

## **REPRESENTATION OF THE PHYSICAL TIME IN THE CENTRAL NERVOUS SYSTEM**

The proposed neuronal clock mechanism outlined above incorporates inbuilt oscillators, which generate temporally regular activity representing a unit of time. Accordingly, a temporal unit interval, called neural temporal unit, between two adjacent spikes or bursts of spikes from the output of an oscillator, is proposed to encode the external physical time information required for processing time intervals in the sub- to supra-second range in the brain. Also note that the current model shares the requirement for an oscillator or a pulse generator with a pacemaker-accumulator model (Wittmann, 1999). However, there are significant differences between two models. An important difference is the inbuilt nature of the proposed oscillator circuit, which is in contrast to a separate temporal pacemaker in the pacemaker-accumulator clock model (Wittmann,1999). The temporal pacemaker produces pulses with stable frequency, which is fed into an accumulator to record its number over a time period, and it is then compared with memory data (Wittmann, 1999). Furthermore, the inbuilt neuronal oscillators in the current proposal are subjected to fluctuations caused by the feedback processes and sensory inputs, which contrasts with the stable frequency of the temporal pacemaker

proposed in the pacemaker-accumulator clock model (Wittmann, 1999).

Note that the neural temporal unit will not be represented by a single value in oscillators across the brain. Instead, the neural temporal units will be represented by different values in different oscillators. This will allow mechanistic flexibility for the interaction between separate clock mechanisms (see the discussion in Evidence Supporting the Role of Neural Temporal Units). A characteristic feature of the neural temporal unit is "intrinsic regularity" during a particular task. Note that the proposed intrinsic regularity is not the same as the precise regularity of mechanical clocks. Instead, it conforms to a behavior resulting from the intrinsic regularity, dependent on the single neuron properties. The regular behavior of the proposed neuronal oscillator is further expected to exhibit statistical properties, which is not discussed in the current manuscript.

According to the current anatomical and physiological data, the following classes of circuits are likely to provide the functionality of oscillators in the proposed modular clock.

(a) Pacemaker neurons (**Figure 4A**): Pacemaker neurons have a common presence in the nervous system (Llinas, 1988). Regular rhythmic activity of pacemaker neurons is believed to emerge from special network connectivity and membrane conductances, such as a low threshold calcium conductance (Llinas,1988). Suprachiasmatic nucleus, known for its role in the diurnal rhythm, contains pacemaker neurons (Schaap et al., 2003). Pacemaker activity is also observed *in vivo* and *in vitro* in deep cerebellar nuclei, which form the main output regions of the cerebellum (Jahnsen, 1986; Raman et al., 2000). Since the cerebellum plays an important role

in cognitive measurements of time intervals (Lewis and Miall, 2003), it is likely that its pacemaker activity projects to form oscillators in other parts of the brain with which it has connections, for example, the posterior parietal cortex (Amino et al., 2001), also known to become co-active with the cerebellum in cognitive timing tasks (Lewis and Miall, 2003). In a separate example, dopamine neurons with slow pacemaker activity are present in the substantia pars compact (Grace and Onn, 1989; Kang and Kitai, 1993), which project to the medium spiny neurons of the striatum, forming synaptic connections near regions receiving excitatory inputs from the neocortex (Purves et al.,2001). The oscillator-type mechanism, resulting from the pacemaker activity of nigrostriatal dopamine neurons, is consistent with other types of evidence in favor of its role in timing mechanisms (Coull et al., 2011).

(b) Tonic sensory input (**Figure 4B**): Tonic activity of slowly adapting peripheral receptors projecting into brain could also serve as an oscillator. The highly sensitive Merkel's disks (Purves et al., 2001), which are peripheral sensory receptors, are good candidates as a source of regular tonic input for various oscillator mechanisms. Tonic activity resulting from the stimulation of proprioceptors in tendons and muscles, project to the cerebellum and lateral intraparietal (LIP) area in the posterior parietal cortex (Prevosto et al., 2011), which could double up as an oscillator for clock mechanisms in circuits in the cerebellum and posterior parietal cortex. Role of the regular proprioceptive activity, acting as an oscillator, is likely to be combined with its role in motor movements in the cerebellar circuits.

(c) Regular synchronized activation and inhibition of ensemble of neurons (**Figure 4C**): Recording during specific tasks have

**FIGURE 3 |The schematic illustration of the representation of physical time in neuronal circuits in (A); the frequency modulation in (B) and (C); and an ultrastructural basis of frequency modulation in (D).** The schematic in **(A)** shows the regular firing activity of a neuronal oscillator. The interval between two adjacent spikes during a regular activity is defined as the neural temporal unit, which is proposed to represent the physical time within the neuronal circuits. The value of the neural temporal unit is characteristically the same in a given circuit during a particular phase of a given activity. **(B)** Shows that the inter-spike interval

shown ensembles of neurons undergoing synchronized changes of membrane potential, which are fleeting in both location and time (Kahana, 2006; Akam and Kullmann, 2010; Buschman et al., 2012; Buzsaki and Wang, 2012). These synchronized changes in membrane potential form the basis of dynamic electromagnetic gamma oscillations (10–50 Hz) detected in different parts of the brain (Kahana, 2006; Akam and Kullmann, 2010; Buschman et al., 2012; Buzsaki and Wang, 2012). It is further believed that synchronized changes in the membrane potential of ensemble of neurons represent excitatory and inhibitory post-synaptic changes (Sarko et al., 2013). Synchronization of the excitation and inhibition increases the chances of firing of neurons connected together in an ensemble during a particular phase of cycle (Fries et al.,2007), leading to bursts of spikes at regular intervals (Sarko et al., 2013; schematically represented in **Figure 4C**). The inevitable result of the synchronization of the activity of neurons connected together in ensembles is the temporal regularity of synchronized bursts. I propose that temporally regular bursts of spikes, generated by synchronized activation and inhibition of an ensemble of neurons during a cognitive task, could also function as an oscillator in a neuronal clock mechanism. Experimental evidence arguing this potential mechanism is provided by studies that had shown synchronous modulation of a large number of neurons in primate brains during various timing tasks (Durstewitz, 2004; Lebedev et al., 2008; Schneider and Ghose, 2012).

progressively decreases in an FM neuron with ascending activity. **(C)** Shows the descending activity of an FM neuron. Note the progressive increase in the inter-spike interval. **(D)** An electron micrograph of a section from a rat spinal cord at a lower thoracic level (unpublished data from work done by the author at University of Louisville) shows a dendritic spine receiving inputs from two morphologically different synapses (indicated by red asterisks). Two different types of synapses can exhibit different types of plasticity in the same neuron, which can provide an ultrastructural basis for the modulation of the activity of neurons.

A key mechanistic requirement of a timing circuit is the ability to mark the onset and offset of a time interval. This could be achieved by incorporating a mechanism to start and stop a timer. Gamma oscillations are shown to be phase locked to or reset by different sensory stimuli (Eckhorn et al., 1988; Llinas et al., 1991; Joliot et al., 1994; Gray and Viana Di Prisco, 1997; Sannita, 2000; Fernandez-Ruiz and Herreras, 2013). Reset of an active synchronized ensemble of neurons by sensory stimuli would provide a mechanism to mark the onset or offset of time duration. Evidence suggesting this type of mechanism is provided by a study in monkeys involving self-timed saccading tasks, which showed that adjacent timing intervals were independent, consistent with the reset before the onset of each task (Schneider and Ghose, 2012). In this study, the reset of timing mechanism is believed to occur by internally generated cues (Schneider and Ghose, 2012).

Lebedev et al. (2008) have previously argued that distributed networks of neurons are responsible for the interval timing in a self-timed, delayed motor task, which was confirmed by the multielectrode recording from multiple areas of the brain in Macaque monkeys (Lebedev et al., 2008). They showed that the average rate of the change in frequency of neural activities obtained from simultaneous multi-electrode recording correlated with the delay period in a self-timed, delay motor task (Lebedev et al., 2008). The detection of the simultaneous modulation of neurons in multiple areas can be explained by their

during each spike burst.

connection to a synchronized activity (**Figure 4C**). In addition to playing role as an oscillator, an ensemble of neurons with synchronized activity may also act as a feeder circuit between an oscillator and a circuit of FM neurons. Moreover, there are two types of information available for processing in ensembles of neurons with synchronized activities: (i) the interval between bursts and (ii) the activity of neurons during the bursts.

into the brain, resulting from the tonic sensory stimulation, also provides

## **EVIDENCE SUPPORTING THE ROLE OF NEURAL TEMPORAL UNITS IN THE INTERVAL TIMING**

The processing of neural temporal units in neuronal circuits is also inferred from psychophysics experiments. In a task designed to compare the time intervals between two sound tones of a given frequency with a standard time interval, the benefits of training disappear when the standard time interval is changed (Wright et al., 1997). But the benefit of training is transferred when only the frequency of sound tones is altered (Wright et al., 1997). In this particular example, proposed oscillator activity for the clock mechanism is generated by the regular evoked bursts of action potentials in the brain after listening to regular beats of a given frequency. When the frequency of beats is changed, it would help to switch to a different proposed oscillator circuit, which would cause the benefits of the training to disappear. However, when the frequency of the tone was altered without changing the beat frequency, the same oscillator circuit would remain, continuing the benefits of the training. Thus, this study reveals an interval represented by regular beats, which plays role in timing tasks in above experiments, similar to the postulated neural temporal unit in the current model.

In another psychophysics experiment (Becker and Rasmussen, 2007), participants were first required to listen to an adapting rhythm of various beat frequencies. After listening to the adapting rhythm, participants listened to a fixed rhythm. After listening to the fixed rhythm, subjects were required to reproduce the fixed rhythm by a finger tapping task. An aftereffect of the adapting rhythm on the reproduction of the fixed rhythm was observed when the interval between adapting beats was closer to that of the fixed rhythm. This is explained by the proposed oscillator circuit, which is established by the longer lasting adapting rhythm with a periodicity closer to that of the fixed rhythm. In another result in the same study, the delivery of sound waves for the adapting and test rhythms from opposite ears produced similar aftereffects (Becker and Rasmussen, 2007), which indicates that there is a single neuronal clock mechanism for this particular task, localized to the right hemisphere, a likely assumption based on a past study (Harrington et al., 1998).

### **THE PROCESSING OF TIME INTERVALS FROM THE "NEURAL TEMPORAL UNITS"**

"Neural temporal unit," defined as the interval between two adjacent regular spikes or spike bursts, (**Figure 3**) is proposed to represent the "dimension of time" in neuronal circuits processing various types of temporal information, such as the time interval, speed, and rate. Activity from the inbuilt oscillator is proposed to feed into the second module, where it produces modulations in the activity of FM neurons. The modulations in the activity of FM neurons help in coding different "time intervals" by generating changes in state-dependent parameters of the circuit in the second module, represented by the frequency change and the rate of the change of frequency or their combination.

Several past studies have demonstrated neurons undergoing modulation, with ascending or descending activity in the motor, premotor and parietal cortex during motor tasks, involving mental measurement of time intervals, and comparing lengths of time intervals (Leon and Shadlen, 2003; Durstewitz, 2004; Lebedev et al., 2008; Schneider and Ghose, 2012). Using simultaneous multi-electrode recordings, Lebedev et al. (2008) showed that the average rate of the change of the frequency of neurons with ascending activity correlates with the time intervalsfor motor tasks lasting 2.5–4.5 s (Lebedev et al., 2008). By recording activities of neurons with descending activity in LIP area, Schneider and Ghose (2012) showed that the production of inter-saccade interval in the range of 800 –1200 ms was positively correlated with the variations in the frequency, analyzed in 100 ms bins (Schneider and Ghose, 2012). Another study, by Mita et al. (2009), recorded activities in medial motor areas, in particular the pre-supplementary motor area in tasks requiring the minimum waiting periods of 2,4, and 8s(Mita et al., 2009). They found neurons, with graded activity with exponential fit, during timing task, which showed selectivity to the duration of time intervals (Mita et al., 2009). These studies show that there are populations of neurons, undergoing modulations that are distributed across different parts of the brain, which have role in the processing of time intervals in the subto supra-second range. Further note that the modulated frequencies, detected in different studies involving timing tasks, range from 10 to 30 spikes/s (Lebedev et al., 2008; Schneider and Ghose, 2012), which is consistent with the connection of the neurons undergoing modulation with synchronized activities in the brain in the gamma range (20–40 Hz), believed to be important for information exchange (Buzsaki and Wang, 2012).

Short-term plasticity is one of several potential mechanisms, which can produce modulation in the activity of neurons in the central nervous system (Regehr, 2012). As depicted in **Figure 3**, neurons exhibiting short-term plasticity with excitation or depression, after excitation by the activity of oscillator circuits via synaptic inputs, can give rise, respectively, to an ascending (**Figure 3B**) or a descending activity (**Figure 3C**). Short-term plasticity, associated with facilitation or depression, has been demonstrated in mammalian cortical neurons (Gil et al., 1999). Theoretical studies have also shown that ascending and descending type of modulation in populations of neurons may be derived, respectively, from the properties of the network connections and the stochastic behavior of the neuronal populations (Gavornik and Shouval, 2011; Simen et al., 2011).

### **READING THE ACTIVITY OF FM NEURONS**

The activity of FM neuron is read by downstream circuits to give rise to time intervals required for various tasks and decisionmaking processes. A network of FM neurons in the second module of the proposed clock is under the influence of a single oscillator during a given task. The requirement for a single active oscillator circuit is based on the argument that multiple timing mechanisms for a single task or function will interfere with the homeostasis. The activity of FM neurons will be read by downstream circuits in order to produce temporal characteristics of a task, or a time interval for a delayed motor task, or production of a time interval. The proposed FM neurons are a mixed group of neurons with either ascending or descending activities. Both types of activities are likely to play role in various interval timing tasks. For example, a descending activity, directly or indirectly via a connection with an inhibitory interneuron, can produce the disinhibition. Disinhibition will occur when an inhibitory presynaptic neuron with a descending activity reaches a stage such that the post-synaptic neuron is not sufficiently inhibited. A disinhibition after a time interval encoded by the decay dynamics of an FM neuron can help in the activation of select neurons in a temporally specific manner. The activation of a motor unit by an FM neuron may help in a timed motor action. The activation of a post-synaptic neuron can also occur by an excitatory neuron with an ascending activity. Alternately, an excitation of a post-synaptic neuron with inhibitory activity can produce a timed-inhibition. The timed inhibition of a motor unit can inhibit activity of antagonist muscles, which can co-ordinate with activation of a synergistic muscle in timed motor actions.

Excitation of a post-synaptic neuron by another neuron with an ascending activity will produce behavior of a threshold oscillator (Borresen and Lynch, 2012). Theoretical considerations show that various logical operations, such as AND, OR, and NOT can be implemented by threshold oscillators with two inputs (Borresen and Lynch, 2012). Two logical states for neuronal circuits are "ON" state or the active state and "OFF" state or the inactive state of a neuron. The logical operations

can help in determining the ON and OFF states of different neurons, which may control motor units or other neurons in the nervous system, helping in the decision-making processes. Together the decaying and ramping dynamics, and the logical operations would explain the timed motor behavior in animal experiments. Logical operations will also play a role in the muscle coordination by determining the activation (ON state) of synergistic muscles or the inhibition (OFF state) of antagonistic muscles.

## **CALIBRATION OF THE PROPOSED MODULAR CLOCK MECHANISM**

An important question that is not adequately addressed by various models is how neuronal clocks are calibrated. The accuracy of neuronal clocks will depend on how closely time intervals measured reflect time intervals of the external physical events. Various abnormalities of the central nervous system functions will result from the lack of calibration of neuronal clocks due to the inability to meet specific temporal requirements of external tasks. It is now believed that the timing intervals for motor and sensory tasks are calibrated by feedback mechanisms (Parsons et al., 2013). But how it is done is not well understood (Parsons et al., 2013).

### **ROLE OF MOTOR PROCESSES IN THE CALIBRATION MECHANISM**

An important function of the cerebellum is to produce a smooth control of the motor movements, which suggests its involvement in the feedback processes controlling the motor movements. Since an important role of a feedback mechanism is to maintain the normal range of a function, its failure in control of motor movement will produce an increase in the range of a movement. Consistent with the preceding, a past study has shown a four to fivefold increase in the range (representing 95% of throws) of finger opening and release times in skilled overarm throwing in patients with cerebellar lesions (Timmann et al., 1999). Furthermore, the role of the feedback processes, controlling precise movements, in transferring information about the external time into the neuronal circuits in the brain can be deduced from the discussion of an example of a cricket fielder catching a ball (**Figure 1**). To catch a ball, especially in a fast trajectory, the fielder is required to execute a complex set of temporally precise motor movements, which would recruit feedback mechanisms in the cerebellum. Missing the ball's trajectory, even by a few fractions of a second, will result in a failure to execute the task. In other words, the feedback processes must tightly couple the motor actions to the requirements of the task parameters, such as the speed of a ball. The speed (-Distance/-Duration) of a ball contains information about time in an external three-dimensional surrounding. Therefore, the successful execution of a task will result in the transfer of information about "physical time intervals," such as those based on the speed of external objects, into the neuronal circuits in the brain. Since most changes needed for the smooth control of motor movements occur over sub-second durations, it indicates that the cerebellum will play a role in representing shorter time durations. Consistent with this expectation, it has been shown that the brain uses networks involving the cerebellum for tasks involving shorter intervals (below 1 s; Breukelaar and Dalrymple-Alford, 1999; Ivry and Spencer, 2004; Jantzen et al., 2004). Another study showed adverse effects of the cerebellar lesions on the temporal estimation, reproduction, and production tasks (Gooch et al., 2010). Cerebellar lesions in the same study were also associated with an increase in the variability of temporal tasks (Gooch et al., 2010). The increase in the variability in temporal tasks suggests a disruption of feedback processes. Based on the preceding discussion, I propose that the feedback processes occurring in the cerebellum provide a binding property between the oscillators in the cerebellum and the cortical areas, which is responsible for maintaining the accuracy of neuronal timing mechanisms in the cortex. This binding property, present between oscillators in the cerebellum and cortical circuits, is due to the synchronous effects of feedback processes, which leads to the calibration of timing mechanisms in the cortex (**Figure 5**). The binding property is likely to be a synchronous, but a temporally sharp change at the level of the cerebellar and the cortical oscillator circuits, produced by feedback processes controlling motor movements, matching the temporal parameters of the external task. The sharp synchronous changes will help in defining or focusing temporally sharp points on time axis within different circuits in the cortex and cerebellum in synchronicity with the external temporal events, thus, transferring the external temporal information into the circuits in the brain. It is also posited that there is a direct serial connection between oscillators in the cerebellum and the cortical areas, which would be required for the flow of information between oscillators in the cerebellum and the cortical areas. Note that the flow of external physical time information from the cerebellum to the cortical areas (**Figure 5**) would require that the neural temporal units maintain their regular interval during their transmission. The pacemaker activity has been observed *in vivo* and *in vitro* in deep cerebellar nuclei (Jahnsen, 1986; Raman et al., 2000), which may represent the source of temporal units being transmitted to various clock mechanisms of the brain. Furthermore, any frequency modulation during the flow of information from an oscillator in the cerebellum to an oscillator in a cortical area is likely to disrupt the proposed binding property. A review of recent literature also finds that the output from the cerebellum projects to multiple nonmotor areas in the prefrontal and posterior parietal cortex (Strick et al., 2009), providing the anatomical substrate for the binding property between the cerebellar and the cortical nonmotor circuits. The binding property can be confirmed by recording correlated activities in the nonmotor areas of the cortex and the cerebellum during motor movements.

It is argued above that external time information is represented within the neuronal circuits with the help of feedback processes, responsible for the tight coupling between the task execution and its temporal requirements. This leads us to the next question: which neurophysiological parameters help in coding the physical time as a result of the successful execution of a task? The "frequency" of neuronal firing is a good candidate for this role. Changes of activities such as the rate of muscle contraction or angular change at a joint mostly result from changes in the frequency of firing of motor neurons. Increased rate of a change in motor movements would represent shorter time durations, as the rate of change is inversely related to the time duration of occurrence of a defined change. Accordingly,

The oscillator circuits in the cerebellum and the cortex are connected to each other via the thalamic relay. Feedback processes, controlling motor movements, are shown to affect oscillator mechanism in the cerebellum. A common type of the feedback loop in the motor control involves inputs from proprioceptive pathways and outputs from motor neurons. The error produced during a motor task will be measured with the help of proprioceptive inputs, and the error correction will be accomplished by the motor output. The feedback process controlling motor movements affect the oscillators in the cerebellum and cortex, which are connected to each other via a binding property, producing synchronous changes at both levels. This helps to represent physical time intervals in circuits in the brain during an interaction with external physical event or surroundings. The irregular sensory evoked activities as oscillator circuits, synchronous with the external events, helping to label the external activity on a physical time scale within circuits in the brain. Sources of the oscillator activity in the cerebellum (as discussed in the main text) include proprioceptive inputs and pacemaker neurons. Sources of the oscillator activity in the posterior parietal cortex (as discussed in the main text) are the proprioceptive inputs and polysensory processing. Different regions of the cortex receive projections from the cerebellum via thalamic relay (indicated by solid arrows), which allow cerebellum to engage with different regions of the cortex in a modular fashion during a clock mechanism. The broken arrows indicate that the reciprocal connections entering the cerebellum are less developed compared to the outgoing connections from the cerebellum.

a higher frequency of a neuronal activity in the nervous system will encode a shorter time duration, over which a defined activity occurs. Furthermore, as an increase in the activity of motor neurons is commonly accompanied by a general increase in the rate of the firing of neurons in various upstream circuits, a proportional increase will be seen in neuronal oscillators in the cortical and cerebellar circuits. An increased activity of oscillator circuit, according to various models, such as recurrent feedback excitation (Gavornik and Shouval, 2011), and facilitation type of short-term plasticity (Regehr, 2012) is likely to enhance the rate of the change of the activity of the proposed FM neurons. The preceding claim is supported by a study, which reported an increased rate of the change in the activity of neurons, undergoing positive modulation, when animals reported shorter anticipatory time intervals in motor tasks (Lebedev et al., 2008).

### **ROLE OF SENSORY PROCESSES IN THE CALIBRATION MECHANISM**

The successful execution of motor tasks, emphasized above for their role in timing mechanisms, also depends on various sensory functions, for example, cortical representations of the spatial coordinates of the surroundings, spatial coordinates of auditory stimuli, eye-centric, and body-centric visual fields. LIP area in the posterior parietal cortex area has been shown to contain body centric; and its adjacent area 7a is shown to contain world-centric representations of visual field (Andersen,1997; Snyder et al.,1998). Neurons in LIP area also code auditory stimuli in eye-centric coordinates (Andersen, 1997). Various sensory representations in the parietal cortex can play role in feedback mechanisms in sensory processes, which can serve to calibrate oscillators in a manner discussed above for the motor processes (**Figure 5**). Consistent with the proposed role in the calibration of oscillator circuits, the left parietal lobe has been shown to be associated with the perceptual and motor temporal predictability (Coull et al., 2011). Along these lines, it is noteworthy that areas in the posterior parietal cortex, including LIP, receive proprioceptive inputs directly from the relay nuclei in the thalamus (Prevosto et al., 2011), providing a tonic input to the posterior parietal cortex, resulting from the regular bursts from the stimulation of proprioceptors, which could serve as an oscillator. A significant number of di-synaptic connections from the spinal cord dorsal column nuclei project to the posterior parietal cortex, which are the shortest possible sensory pathways to the posterior parietal cortex, and hence the most direct method of the delivery of proprioceptive inputs (Prevosto et al., 2011). Thus, the posterior parietal cortex is likely to play the role, alongside with the cerebellum, as the oscillator module, helping in the calibration in the proposed modular clock model (**Figure 5**).

Although, the role of sensory processes in transferring external temporal information via direct interaction has been emphasized, it has been argued earlier that we are able to anticipate changes in events that we cannot imitate, such as lashing waves or other inanimate events (Schubotz, 2007). The ability to anticipate can also help us in internalizing the temporal information from external events, such as falling leaves during fall, stones rolling down a mountain slope or movements of vehicles in complex traffic situations. Such ability will help in internalizing temporal information from the external events without a task execution.

The representation of external time into the neuronal circuits can be achieved by rhythmic external activities producing sensory stimulation, such as waves lashing a beach. Rhythmic external physical events, such as the waves lashing shores at a regular rate, will help to encode information about time represented within the periodicity of sounds and sights of lashing waves. This information would calibrate the oscillator activities within circuits processing rhythmic sensory inputs by directly or indirectly influencing their activities. A simple example of a periodicity coding time information will be the activation of an auditory neuron at two time points by consecutive single sound tones. In this example, the time interval between two sound tones is represented internally in the neuronal circuits by the time interval separating two evoked activities in auditory areas. In addition to providing information about the temporal periodicity, the periodic sensory inputs can modify or become an oscillator module for a neuronal clock mechanism.

Non-regular external sensory events also appear likely to play a role in coding the temporal information in neuronal circuits by producing evoked activities at irregular intervals resulting in simultaneous sharp changes in the firing activity at variable intervals, enabling the binding between different sensory circuits, similar to the effect of feedback processes during a motor activity. Such temporally non-regular events may include the sight of an object going up in the air to fall down. Temporally non-regular event can produce synchronous evoked activities in areas of cortex with different representation of same sensory modality leading a binding of oscillators in different parts of cortex, which could play a role in the calibration of timing mechanisms. A certain kind of activity may be more likely to undergo primary processing in one or the other visual field. For example, various characteristics of a speeding car will be analyzed best by organizing the visual information in the worldcentric representation in the parietal cortex. In comparison, a cricket ball being sought by a running fielder would be processed best by retrieving information from the head-centric cortical representation. Accordingly, the timing mechanism for ball movement in the head-centric area of the parietal cortex will calibrate the timing mechanism in the world-centric area 7a by virtue of the binding property, enabled by synchronously evoked activities in both cortical representations. In addition, the multisensory processing circuits in the posterior parietal cortex (Andersen, 1997), may also mediate the binding of oscillator circuits of different modalities, resulting in the calibration of the timing circuits controlling multimodal tasks.

## **THE ROLE OF BASAL GANGLIA IN THE MODULAR CLOCK MECHANISM**

Past animal studies and clinical data have implicated the basal ganglia and nigrostriatal circuits in interval timing tasks. One study showed that Parkinson's disease subjects (withdrawn from levodopa therapy for 12–24 h) under-estimated time intervals in a verbal estimation task and over-produced time intervals, suggesting the role of dopamine and nigrostriatal circuits in an "internal clock"(Pastor et al., 1992). Another study with subjects suffering from Parkinson's disease found impaired discrimination of time intervals in 1 s range, without impairment in the 12–42 s range (Riesen and Schnider, 2001). In contrast to the human studies, the animal studies present a more robust evidence of a substantial role of the dopamine and nigrostriatal circuit in the timing mechanisms (Coull et al., 2011). A theory to explain the role of the basal ganglia in interval timing, based on the coincidence detection was proposed earlier (Matell et al., 2003). With the help of a discussion of recent scientific literature below, I argue that the basal ganglia may play role as an oscillator module, helping in the calibration of the modular clocks (**Figure 6**).

Striatum forms part of multisynaptic, cortical–striatal– pallidal–thalamic–cortical loops (**Figure 6**), which are responsible for various functions depending on the cortical area involved, They include the motor movements (motor cortex), planning and working memory (prefrontal cortex), and limbic functions (cingulate cortex; Middleton and Strick, 2000; Purves et al., 2001). An *in vivo* recording of the medium spiny neurons, which are the main computational units in the striatum, has shown two states, an UP state (a depolarized state) and a DOWN state (a hyperpolarized state; Murer et al., 2002). Action potentials can only fire during UP state, driven by the excitatory cortical

**FIGURE 6 |The schematic representation of basal ganglia structures in formation of a proposed calibration module.** In this schematic, the role of the basal ganglia circuit in interval timing is illustrated in the context of the current modular clock model. An anatomical loop, formed between the basal ganglia structures, thalamus, and different cortical areas is depicted. The anatomical loop is proposed to serve as a substrate for feedback mechanisms. The striatum, which contains medium spiny neurons with stable UP and DOWN states, receives dopamine inputs from the nigral neurons with pacemaker activity. The rhythmic input of

inputs (Murer et al., 2002). Thus, it is argued that the regular spaced UP states associated with regular bursts of action potentials will serve as an oscillator in the modular clock. In addition, the pacemaker activity of nigrostriatal neurons (Grace and Onn, 1989; Kang and Kitai, 1993), will release fluctuating quantities of dopamine, which would help in modulating oscillator pattern, produced by the firing of the striatal medium spiny neurons. The multisynaptic, cortical–striatal–pallidal–thalamic– cortical circuit is an anatomical loop which connects the cortex, basal ganglia and thalamus. Since many negative feedback processes are represented by loop-like pathways, cortical–striatal– pallidal–thalamic–cortical circuit is a likely anatomical substrate for a feedback loop with a role in cortical information processing. Although it is not clear if the thalamic connections return to the same cortical region, it is a possibility that an anatomical loop may be completed by the cortical association fibers. On the basis of the above discussion, combined with the evidence from different studies discussed below, I posit that cortical–striatal–pallidal–thalamic–cortical circuit would serve as the anatomical substrate for the feedback mechanisms to play a role in the calibration of the proposed basal ganglia clock (**Figure 6**).

A study that simultaneously recorded single neuron activities from the primary motor and sensory cortex together with the putamen during a visual motor task, found spatial and temporal dopamine can help in the maintenance of the oscillations, resulting from the stable UP and DOWN states of the neurons in the striatum. The oscillations in loop are calibrated by the feedback processes proposed within the anatomical loop formed by the basal ganglia structures. The role of the feedback mechanism is also suggested by other types of evidence, discussed in the main text. The calibrated oscillation activity from the anatomical loop, formed by the basal ganglia structures can project to various parts of brain in a modular fashion to participate in different clock mechanisms.

task-related firing of neurons across these regions (Santos et al., 2014). The correlated firing behavior is consistent with the accurate transmission of information (Santos et al., 2014), which is necessary for a feedback mechanism. Another study showed coherence during the simultaneous recordings of the electrocorticogram and local field potential in the basal ganglia (Magill et al., 2004). Coherent slow waves (∼1 Hz) and spindle frequency (7–12 Hz) oscillations recorded in the cortico-basal ganglia circuits switched to a coherent, medium to high-frequency (15–60 Hz) activity after sensory stimulation by pinching hindpaws at 15 s intervals with serrated forceps (Magill et al., 2004). The state-dependent coherence in the activity is consistent with information processing during a feedback mechanism. A direct involvement of the cortical–striatal–pallidal–thalamic–cortical circuit in the feedback mechanisms is further indicated by asynchronous activity after the onset of movements observed in basal ganglia during the simultaneous recording of multiple neurons in monkeys (Jaeger et al., 1995).

The nigral dopaminergic inputs to striatal spiny neurons may play a role in the feedback loops by maintaining the activity in cortical–striatal–pallidal–thalamic–cortical circuits. Accordingly, the loss of nigrostriatal dopamine input to striatal circuits would lead to the disruption of the feedback mechanism (**Figure 6**). This is indicated by the increased variations in timing tasks in Parkinson's disease (Pastor et al., 1992). Also note that the role of the

basal ganglia circuit in motor control is likely to be different from its role in the calibration mechanism in the modular clock. This is indicated by a recent study, examining the effects of the bilateral high-frequency stimulation of the basal ganglia, used widely for the treatment of the motor symptoms of Parkinson's disease, on the duration and beat based tasks of the sub-second range (Cope et al., 2014). This study found no significant effect on any perceptual timing task studied (Cope et al., 2014), which could be due to the inability of bilateral high-frequency stimulation of the basal ganglia to significantly influence the proposed feedback processes in the cortical–striatal–pallidal–thalamic–cortical circuits.

## **THE ROLE OF CEREBELLAR CIRCUITS IN THE MODULAR CLOCK MECHANISM**

Past studies have shown that the brain uses networks involving the cerebellum for tasks involving sub-second intervals (Breukelaar and Dalrymple-Alford, 1999; Ivry and Spencer, 2004; Jantzen et al., 2004). In addition, the lesions of the cerebellum increase variability on various timing tasks, which include the temporal estimation, reproduction, and production tasks (Gooch et al., 2010). The above is consistent with the proposed role of the cerebellum as a module responsible for the calibration of neuronal clocks, for timing intervals in the sub-second range. It is also argued that the binding between oscillators in the cerebellum and cortical areas, as a result of feedback processes controlling motor movements, is responsible for the calibration of the modular clocks in cortical areas (**Figure 5**). The pacemaker activity has been observed *in vivo* and *in vitro* in deep cerebellar nuclei, forming the main output regions of the cerebellum (Jahnsen, 1986; Raman et al., 2000), which may represent an oscillator, projecting to another oscillator circuit in the cortical areas. An inter-connection or binding between oscillators would help in transferring information about physical time from the cerebellar circuits to the cortex during tasks requiring interaction with the external world (**Figure 5**). Furthermore, the changes that occur during a motor control are commonly in the sub-second range, which supports the role of the cerebellum in the sub-second timing.

## **PROCESSING TIME INTERVALS WITHOUT ANY EXTERNAL TASKS OR CUES**

Since the processing of time intervals, in the absence of cues, does not involve any specific tasks, the calibration of clock mechanism will not be accomplished by a successful task completion. Instead, the mechanism for interval timing in the absence of cues would be linked to the sensory perception of the passage of time. This is based on the observed relationship between the sense of passage of time and the subjective estimation of time intervals. For example, a subjective report of the fast passage of time is often associated with the over-estimation of time intervals. The sensory perception of the flow of time is likely to be a consequence of the sensory detection of the temporal order, when there is a minimum temporal separation between adjacent multimodal or monomodal stimuli, for example by 20–30 s for auditory stimuli (Poppel, 1994, 1997; Pastore and Farrington, 1996). During the conscious state, there is a task-independent processing of multimodal stimuli, such as auditory, proprioceptive and visual, associated with the perception of the flow of time in the posterior parietal cortex, which

will generate oscillator-type activities. I, furthermore, posit that the task-independent polysensory processing will act as an oscillator in the modular clock in the absence of external cues. This agrees with a study in primates (Schneider and Ghose, 2012), in which the production of inter-saccade intervals in the range of 800–1200 ms in absence of any external cues, was positively correlated with the variations of the frequency of neurons with descending activity in LIP area in the posterior parietal cortex, believed to be an important area for the polysensory processing in the brain (Andersen, 1997; Bremmer et al., 2001; Macaluso and Driver, 2001).

A potential mechanism of calibration in the absence of cues would be the feedback process based on the cortical–striatal– pallidal–thalamic–cortical loop. The anatomical evidence, supporting the role of cortical–striatal–pallidal–thalamic–cortical loop in the calibration of clock circuits comes from a study using the retrograde transneuronal transport of rabies viruses to examine the subcortical inputs to anterior intraparietal area in the posterior parietal cortex, which found labeling of a substantial number of neurons in the substantia pars reticulata (Clower et al., 2005), one of the output structures of the basal ganglia (Middleton and Strick, 2000). Another potential method of the calibration of clocks in a cue-independent mode is a pre-stored pattern of activity in the premotor region, which is commonly co-activated with the inferior parietal region in fMRI studies during timing tasks, without specific instructions to time (Coull and Nobre, 2008).

## **SENSORY PROCESSING IS COUPLED WITH THE PERCEPTION OF TIME INTERVALS**

Sensory experience of external events involves simultaneity of perception of different modalities, which results from the polysensory processing in the parietal cortex, involving gain modulation (Andersen, 1997). Since many timing tasks are multimodal in nature, the polysensory processing is likely to help in the transfer of the external time information into the nervous system. The involvement of polysensory processing in timing mechanisms comes from a study, in which subjects compared the interval between stimuli of same (monomodal) or different (polymodal) modalities with a standard time interval (Harrington et al., 2011). During a multimodal timing task, comparing time interval between auditory and visual sensory stimuli, there was increased activation of the parietal frontal network (Harrington et al., 2011). In contrast, the monomodal timing task showed greater activation of the striatum (Harrington et al., 2011). This suggests that timing mechanism, involving cues of different sensory modalities, is likely to be a part of polysensory processing circuits in the posterior parietal lobe. Thus, it is not surprising that networks in the brain, involved with the measurement of time intervals in most studies, include the posterior parietal region (Amino et al., 2001), which contains several important polysensory processing areas, as well as various representations of the visual and auditory fields (Andersen, 1997). Functional imaging studies have shown that the polysensory processing areas in humans and primates are present in the superior temporal sulcus and intra parietal sulcus (Bremmer et al., 2001; Macaluso and Driver, 2001).

PET studies have shown that the tasks requiring the detection of the visual auditory asynchrony activate a large network in the brain that includes the posterior parietal, prefrontal, and cerebellar areas (Bushara et al., 2001), which suggests a relationship between the polysensory and temporal processing. Moreover, past studies have shown that the visual and auditory response fields often overlap in the parietal cortex, and also show gain modulation (Andersen, 1997). Gain modulation, which produces a non-linear effect on the perception of another modality, is likely to play a role in the multimodal timing tasks.

### **SYNOPSIS OF THE PROPOSED MODULAR CLOCK MODEL**

The current model postulates "neuronal temporal units," which represent the external time information in neuronal clocks in the brain. "Neuronal temporal unit," defined as the interval between two adjacent spikes or spike bursts resulting from the regular activity of neuronal oscillators, is calibrated, so that it represents physical time accurately within the neuronal circuits. The module containing the neuronal oscillator is responsible for the calibration of the clock. The mechanisms for the calibration of neuronal oscillators include (a) feedback processes involving motor movements, (b) binding between different circuits that result from the synchronous effects of the feedback processes or sensory processes, and (c) input of temporal periodicity resulting from a regular sensory stimulation. The calibration of the neuronal clocks involves a successful transfer of the temporal information from the external physical events into the neuronal circuits, via the successful execution of various tasks or functions. The transfer of external physical time information may sometimes involve two levels of oscillator circuits. For example, during motor tasks, there is a proposed involvement of two oscillators, which help in the transfer of external physical time information. The first-level oscillator is present in the calibration module, such as the cerebellum (**Figure 5**). The second-level oscillator is inbuilt within various circuits in the brain. The second-level oscillator (inbuilt) is in a binding state with oscillators in one of several calibration modules. The ability of an inbuilt oscillator to bind to the oscillators in different calibration modules is responsible for the modular nature of the clock. The effect of feedback and sensory processes on the oscillator in the calibration module would produce synchronous effects on various inbuilt oscillators due to the binding, which is proposed to lead to the representation of physical time in various brain circuits. Note that the external information can be also imported into neuronal circuits without a direct interaction with the external world (Schubotz, 2007).

No single algorithm for the processing of the neuronal temporal units is proposed in the current model. As discussed earlier, the output of oscillator circuits is likely to be processed by several possible mechanisms to produce a ramping or a decaying-type modulation in the activity of FM neurons. The frequency or the rate of the change of the frequency, resulting from the modulation in the activity of FM neurons, codes the time intervals in the brain circuits. This is in line with the prevailing view that many mechanisms may be responsible for computing time intervals (Lewis and Miall, 2003; Wittmann, 2009; Teki et al., 2011a). Different networks of brain are found to be active in different interval timing tasks depending on nature, motor vs. perceptual, or the duration sub-second vs. supra-second (Wiener et al., 2010a). According to the current model, the differences in the mechanism of interval timing in different tasks result from the modular nature of the proposed clock.

### **DIFFERENTIAL ACTIVATION OF NETWORKS IN THE BRAIN IN DIFFERENT TIMING TASKS**

Timing tasks are classified as (a) explicit interval timing, when subjects report duration by speaking, or performing delayed or prolonged motor task; and (b) implicit interval timing, when regular pattern of sensory stimuli or motor responses is used to predict non-temporal parameters, such as, velocity of object trying to land (Coull et al., 2011). Explicit timing is commonly associated with the activation of the basal ganglia, and the co-activation of the prefrontal, premotor, and cerebellar areas seen in more contextdependent tasks (Coull and Nobre, 2008). This is consistent with the current proposal for the role of the basal ganglia in the modular clock for a cue-independent timing. On the other hand, an implicit perceptual timing task recruits the inferior parietal and the premotor area (Coull and Nobre, 2008), suggesting the importance of the visual and spatial information processed in the posterior parietal cortex. Another fMRI study had shown the activation of the left cerebellar hemisphere in addition to the right hemispheric pre-supplementary motor area, frontal pole, and inferior parietal cortex in an explicit interval timing task requiring the discrimination of sub second intervals (Lewis and Miall, 2003). Also, a meta-analysis of neuroimaging studies showed activation of the left inferior parietal cortex in implicit timing tasks (Wiener et al., 2010b). Furthermore, a fMRI study, involving the direction (spatial) and velocity (spatial and temporal) judgment tasks, showed co-activation with parietal regions only during the velocity (spatial and temporal) judgment task, but not during direction only task (O'Reilly et al., 2008), which underscores the "engagement" of the cerebellum only during temporal tasks. Anatomical studies have shown that the cerebellar outputs also target multiple nonmotor areas, especially the prefrontal and posterior parietal cortex, implicated in interval timing (Strick et al., 2009; Bostan et al., 2013), which provides the anatomical basis for the co-activation of cortical regions with the cerebellum during different timing tasks. The differential activation of brain networks during different timing tasks suggests the modular nature of the neuronal clocks, which is consistent with the currently proposed clock model (**Figure 5**).

### **MODULAR NATURE OF THE NEURONAL CLOCK MODEL AND ITS IMPLICATION**

Consistent with the modular nature of the neuronal clocks, I propose that one or several calibration mechanisms would serve in a single clock mechanism. This feature of the modular clock model introduces redundancy in the clock mechanism of the brain. In individuals with the lesions of the cerebellum, other calibration mechanisms, mainly sensory, will take over the function of calibrating neuronal clocks. Since mechanisms based on the sensory processes calibrate or internalize time intervals of longer durations in comparison to the motor processes, the cerebellar lesions will result in greater variations during the processing of temporal intervals. Evidence for the preceding is provided by a study, which showed that the cerebellar lesions disrupted the precise timing, indicated by increased variations (Timmann et al., 1999; Ivry and Spencer, 2004). Furthermore, the multiplicity of calibration mechanisms and multiple oscillator circuits built inside various circuits spread across the brain would explain why no neurological or psychiatric illness is characterized by a complete deficiency of the perception of time intervals.

### **MULTIPLE CALIBRATION MODULES AND THE IMPLICATIONS**

The discussion of different circuits in the current manuscript also indicates that the posterior parietal areas, pre-motor area and basal ganglia circuit (cortical–striatal–pallidal–thalamic– cortical circuits) also are likely to serve as calibration modules in addition to the cerebellum. An earlier fMRI study had found significant activations in a cerebellar network during absolute, duration-based timing and a striato–thalamo– cortical network during relative, beat-based timing (Teki et al., 2011b). Since processing the absolute duration will require input of physical time into brain circuits, the role of cerebellum in processing absolute durations is consistent with its role in the motor tasks, which help to produce representation of the physical time in the brain circuits. However, the proposed cortical-striatal-pallidal-thalamic-cortical circuits are closed loops, with no likely direct interaction with the external events. Accordingly, the internal homeostasis of cortical information processing may be the primary function of the feedback processes in the cortical-striatal-pallidal-thalamiccortical circuits. With no direct representation of physical time, basal ganglia circuits may mainly play role in the beatbased timing, which is relative to a rhythm. This is consistent with activation of basal ganglia during beat-based timing tasks (Teki et al., 2011b).

### **SUB-SECOND VS. SUPRA-SECOND INTERVAL TIMING**

While the cerebellum is implicated in the sub-second interval timing tasks, the imaging and clinical data implicate the basal ganglia in both, the sub- as well as the supra-second interval timing tasks (Malapani et al., 1998; Wiener et al., 2010a). A metaanalysis study showed that different networks are activated in sub-second vs. supra-second timing tasks (Wiener et al., 2010a). Many of the differences reported in this study are likely to be related to the differential activation of the second module, associated with various tasks. Furthermore, because of the modular nature of the proposed neuronal clock, a particular calibration module, for example, the cerebellum may become differentially co-activated within different networks of the brain during various sub-second tasks. In an example of the second module activation, sub-second perceptual tasks demonstrate the highest activation likelihood in the left inferior frontal gyrus, which is an important cortical area for the language production (Wiener et al., 2010a).

### **A COMMON LINK BETWEEN THE SCALAR PROPERTY OF THE INTERVAL TIMING AND INCREASED VARIATIONS ON TIMING TASKS IN CEREBELLAR LESIONS?**

The scalar property of the interval timing, that is, the variations in interval timing increases with its increasing duration (Piras and Coull, 2011), may be linked empirically to engaging different calibration mechanisms in tasks of different durations (**Figure 5**). This can be explained with the help of an example of a stochastic process, which assembles anticipatory time intervals with a random number of temporal units of a standard length, to produce the best match with a task requirement. During the production of time intervals with temporal units of a longer standard length by such process, one is likely to find an increase in variations. This type of random process may also explain the presence of greater variations in the production of time intervals of greater length, when a modular clock uses a different calibration mechanism, such as the sensory processes, instead of the cerebellar mechanisms, which is more suitable for the shorter time intervals. For similar reasons, the cerebellar lesion leads to an increase in the variability of temporal tasks (Gooch et al., 2010). Following the cerebellar lesions, other calibration modules would be responsible for calibration, which help in creating representations of longer intervals in the brain. This would result in increase in the variations, as argued above with the help of a random assembly process.

### **ABNORMAL INTERVAL TIMING IN SCHIZOPHRENIA**

In a recently published study, schizophrenic patients were found to underestimate anticipatory time intervals during the tracking of the movement of visual stimuli under cognitively challenging conditions (Peterburs et al., 2013). Common basis for the cognitive defect and impairment of interval timing in schizophrenia can be explained with the help of this model. The likely target for common pathological changes would be various circuits controlling cognitive functions with inbuilt clocks, for example, cortical–striatal– pallidal–thalamic–cortical circuit. Damage to above circuits would contribute to cognitive symptoms, in addition to the impairment of the interval timing functions. Furthermore, a disruption of the general feedback processes involved in motor control, playing a role in the calibration of inbuilt clocks in the proposed model, would explain a high prevalence rate of motor symptoms observed in schizophrenia (Walther and Strik, 2012). A high prevalence rate of the motor symptoms in schizophrenia suggests an involvement of the cerebellum. Anatomical and physiological studies have shown that the cerebellum projects to myriad neocortical areas, such as the prefrontal and parietal cortex, and contributes to the cognition and visuospatial reasoning (Strick et al., 2009; Bostan et al., 2013). Therefore, the proposed inbuilt clocks, calibrated by the cerebellar processes, could be involved in schizophrenia, producing cognitive as well as motor symptoms.

### **FUTURE DIRECTIONS**

Emerging evidence indicates that the "correlated noise" between neurons and its structure help pools of neurons in guiding their behavior (Nienborg and Cumming, 2010). Although the present model does not to assign a clear role to the "correlated noise," it may play an important role in decoding the activities of FM neurons, helping in the processing of different time intervals, and the decision-making processes. Mounting evidence also suggests that stochastic processes play an important role in information processing in the brain (Xing et al., 2012). Correlated noise and population dynamics could play role at different levels in this model, which include (a) the feedback processes influencing the oscillator circuits, (b) modulation of FM neuron activity by oscillator circuits, and (c) decoding of activity of FM neurons by circuits involved in various tasks. Future study of the proposed modular clock mechanisms would also address the role of the population dynamics of neurons or synaptic connections, or both in complex modulations seen in various timing tasks.

### **ACKNOWLEDGMENTS**

The author is deeply indebted to members of ResearchGate community (ResearchGate Corporation, Cambridge, MA 02142, USA) for thought provoking discussions, helping to write this paper, especially to the contributors to a Question and Answer topic: Can anyone tell "How our Brain Process Time?" The author is also grateful to the teachers and students of Camden County College, Blackwood, New Jersey, USA for their support.

### **REFERENCES**


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 24 March 2014; accepted: 09 July 2014; published online: 01 August 2014. Citation: Gupta DS (2014) Processing of sub- and supra-second intervals in the primate brain results from the calibration of neuronal oscillators via sensory, motor, and feedback processes. Front. Psychol. 5:816. doi: 10.3389/fpsyg.2014. 00816*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Gupta. This is an open-access article distributed undertheterms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Perceptual inequality between two neighboring time intervals defined by sound markers: correspondence between neurophysiological and psychological data

#### *Takako Mitsudo1 \*, Yoshitaka Nakajima2, Hiroshige Takeichi <sup>3</sup> and Shozo Tobimatsu4*

<sup>1</sup> Psychophysics Laboratory, Department of Informatics, Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan

<sup>2</sup> Department of Human Science/Research Center for Applied Perceptual Science, Faculty of Design, Kyushu University, Fukuoka, Japan

<sup>3</sup> Computational Engineering Applications Unit, Advanced Center for Computing and Communication (ACCC), RIKEN, Saitama, Japan

<sup>4</sup> Faculty of Medical Sciences, Kyushu University, Fukuoka, Japan

### *Edited by:*

James M. Broadway, University of California, Santa Barbara, USA

### *Reviewed by:*

James M. Broadway, University of California, Santa Barbara, USA Makiko Sadakata, Radboud University Nijmegen, Netherlands

### *\*Correspondence:*

Takako Mitsudo, Department of Informatics, Faculty of Information Science and Electrical Engineering, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, Japan e-mail: mitsudo@cog.inf. kyushu-u.ac.jp

Brain activity related to time estimation processes in humans was analyzed using a perceptual phenomenon called auditory temporal assimilation. In a typical stimulus condition, two neighboring time intervals (T1 and T2 in this order) are perceived as equal even when the physical lengths of these time intervals are considerably different. Our previous event-related potential (ERP) study demonstrated that a slow negative component (SNCt) appears in the right-frontal brain area (around the F8 electrode) after T2, which is associated with judgment of the equality/inequality of T1 and T2. In the present study, we conducted two ERP experiments to further confirm the robustness of the SNCt. The stimulus patterns consisted of two neighboring time intervals marked by three successive tone bursts. Thirteen participants only listened to the patterns in the first session, and judged the equality/inequality of T1 and T2 in the next session. Behavioral data showed typical temporal assimilation. The ERP data revealed that three components (N1; contingent negative variation, CNV; and SNCt) emerged related to the temporal judgment. The N1 appeared in the central area, and its peak latencies corresponded to the physical timing of each marker onset. The CNV component appeared in the frontal area during T2 presentation, and its amplitude increased as a function of T1. The SNCt appeared in the right-frontal area after the presentation of T1 and T2, and its magnitude was larger for the temporal patterns causing perceptual inequality. The SNCt was also correlated with the perceptual equality/inequality of the same stimulus pattern, and continued up to about 400 ms after the end of T2. These results suggest that the SNCt can be a signature of equality/inequality judgment, which derives from the comparison of the two neighboring time intervals.

**Keywords: temporal assimilation, equality perception, N1, contingent negative variation, slow negative component**

## **INTRODUCTION**

Cognitive time management is an essential function in human life. Adequate time estimation is necessary for our normal social functioning, such as movement, speech, and the prediction of timing (e.g., when a traffic light changes). Indeed, the time perception literature has argued that varieties of human behavior may rely on the perception of time in the seconds-to-minutes range (for reviews, see Matell and Meck, 2000; Buhusi and Meck, 2005). Interestingly, temporal judgments at sub-second timing sometimes lead to various types of perceptual distortions (i.e., illusions) in several modalities. We previously found some phenomena related to auditory and visual temporal perception in short time intervals by means of psychophysical measurements (Nakajima et al., 1991, 2014; Sasaki et al., 1998; Arao et al., 2000; Miyauchi and Nakajima, 2005, 2007; ten Hoopen et al., 2006). One is the perceptual phenomenon referred to as "temporal assimilation" (Nakajima et al., 2004; Miyauchi and Nakajima, 2007). Consider the case where three successive tone bursts (of 20 ms) are used to create two neighboring empty time intervals (T1 and T2), with durations of 120 and 200 ms, respectively. When individuals hear this type of temporal pattern, they often perceive the two intervals as almost equal, despite the physical temporal difference. This temporal assimilation occurs asymmetrically, within a range of −80 ms ≤ T1 − T2 ≤ +50 ms. Although this phenomenon is robust, its underlying neural mechanisms are poorly understood.

Here, we delineate the characteristics of human auditory temporal processing related to auditory temporal assimilation, by conducting electroencephalographic (EEG) measurement. EEG has high temporal resolution, and is suitable to extract brain responses relevant to the perception of milliseconds-toseconds intervals, from different brain areas at the same time. Previous research in time perception has reported some eventrelated potential (ERP) components that could be attributed to differences in the performance of time estimation tasks (Gibbon et al., 1997; Macar and Vidal, 2003). The transient evoked component, such as N1, is known to be modulated by the parameters of stimulus properties. It has been suggested that the characteristics of these components also vary, depending on attention allocation (e.g., Lange et al., 2003; Okamoto et al., 2007; Gontier et al., 2013; Picton, 2013) to sensory signals. When one compares the lengths of two neighboring time intervals, T1 and T2, the participant might judge the lengths of two intervals by focusing on the temporal location of the tone marking the end of T1 (and simultaneously marking the beginning of T2). Therefore, the N1 latency to the second tone burst that separate the two intervals might reflect a specific allocation of temporal attention marking the separation of the two intervals.

Another ERP component is the contingent negative variation (CNV; Walter et al., 1964). Numerous studies have revealed relationships between the CNV and the processing stages of time intervals; not only in the seconds-to-minutes range, but also in the sub-second range. It is related to the memorization of time intervals (Pouthas et al., 2000; Pfeuty et al., 2003a,b), duration reproduction (Macar et al., 1999), and accumulation processes (Pouthas et al., 2000; Montfort and Pouthas, 2003; Pfeuty et al., 2003b). CNV modulation has also been observed in our previous study (Mitsudo et al., 2009). We recorded ERPs while participants were judging the equality/inequality of T1 and T2. The CNV component appeared in the frontal area during T2 presentation, and its amplitude increased as T1 was lengthened.

In addition to these components, our previous study found an ERP component that might be related to the estimation of the equality/inequality of time intervals. A slow negative component (SNCt) appeared in the right frontal areas when participants were engaged in temporal judgments, and the component was larger for stimuli that were associated with subjective inequality between T1 and T2. Equality perception, including auditory temporal assimilation, seemed to correlate with smaller SNCts.

In the present study, we examined the characteristics of these three types of ERP components, N1, CNV, and SNCt, related to temporal assimilation. By presenting time intervals marked with three successive sounds and by recording ERPs from the central and the frontal areas simultaneously, we were able to extract the brain responses in several stages of temporal judgment. We thus aimed to explore the mechanisms of human temporal perception more systematically in the present paradigm. Our particular interest was in the SNCt related to the judgment of equality/inequality to be made after T2. We compared the magnitudes of the SNCt after T2 between conditions in which equality/inequality judgments were and were not required. ERPs were recorded while participants judged the equality/inequality of two neighboring time intervals in the experimental session. ERPs were also recorded during participants' passive listening to the stimuli in the control session. We first conducted an experiment employing the same stimulus patterns as in our previous study, but with new participants. About 1 year later, we conducted another experiment with the same participants, in which the stimulus patterns were reversed in time. If the SNCt in our previous study had really reflected the brain mechanism of equality/inequality perception, it should appear in a different group of participants and for different set of stimulus patterns, and should be larger for the stimulus patterns in which the subjective inequality between T1 and T2 had been dominant, i.e., when T1 − T2 < −80 ms or T1 − T2 > 50 ms.

## **MATERIALS AND METHODS PARTICIPANTS**

Thirteen healthy volunteers with normal hearing [Mean age 20.8 (SD = 3.2) years in Experiment 1, 1 male and 12 females] participated in both Experiments 1 and 2. None of them were musically trained except in ordinary school classes. Informed consent was obtained from each participant after an explanation of the purpose and procedures of the experiment, which were approved by the Ethics Committee of the Graduate School of Medical Sciences, Kyushu University.

## **APPARATUS AND STIMULI**

The experiments were conducted in an electromagnetically shielded soundproof room (Yamaha Music Cabin, SC-3 or SC-5). The background noise was kept below 30 dBA. Stimuli were synthesized with J software (with a sampling frequency of 44.1 kHz) run on a Dell Dimension 4500C personal computer. They were presented diotically from an AV tachistoscope (Iwatsu, IS-703) via a low-pass filter (NF DV8FL with a cutoff frequency of 8 kHz), an amplifier (Stax SRM-313), and headphones (Stax SR-303, STAX). All stimulus patterns consisted of two neighboring time intervals marked by three successive pure-tone bursts of 1 kHz and 20 ms with rise and fall times of 5 ms. We labeled the three markers S1, S2, and S3. The sound pressure level of these tone bursts was 77 dBA. This level was measured as the level of a continuous tone of the same amplitude with a precision sound level meter (Node 2075), mounted on an artificial ear (Brüel and Kjær 4153). In Experiment 1, we used seven standard stimulus patterns, in which T1, defined as the inter-onset interval between the first and second marker, varied from 80 to 320 ms in 40-ms steps, whereas T2, the inter-onset interval between the second and third marker, was fixed at 200 ms. In Experiment 2, the same apparatus was used, and the stimulus patterns were reversed in time; T1 was fixed at 200 ms, and T2 varied from 80 to 320 ms. In each of these experiments, we used four dummy stimulus patterns to prevent the participants from memorizing or noticing the fixed 200-ms duration. Indicating the neighboring time intervals as T1|T2 ms, the dummy patterns were 140|140, 260|260, 200|80, and 200|320 ms in Experiment 1, and the same patterns were reversed in time in Experiment 2 (**Figure 1**).

## **PROCEDURES**

We first conducted Experiment 1. Experiment 2 was conducted about 1 year later with the same participants. Each experiment consisted of an experimental session and a control session. The task in the experimental session was to judge whether the durations of T1 and T2 were equal or unequal by pressing one of the two buttons on a unit held with both hands. The task in the control session was to listen passively to the stimuli and to press one of the two buttons, chosen at the participant's own will, without making a judgment. For both the experimental and the control sessions, the seven standard stimuli and four dummy stimuli were

presented. One trial started with a dummy presentation followed by an experimental presentation. The combination of stimuli presented in the dummy presentation and the experimental presentation was randomized. Each of the seven standard stimuli were presented 100 times in random order in the experimental presentations, while the dummy presentations, in which the dummy and the standard stimuli were employed, alternated with the experimental presentations to avoid participants memorizing or noticing the fixed intervals in the standard stimuli (**Figure 1**). The sessions were divided into 10 blocks of 40 trials and 10 blocks of 30 trials (i.e., 7 standard stimuli × 10 blocks × 10 trials). ERPs were recorded only in the experimental presentations. Interstimulus intervals (ISIs) were varied randomly between 3 and 5 s. Each participant first performed the control session and then the experimental session on four separate days in total.

### **ERP RECORDINGS**

ERPs were recorded from 19 scalp locations (Fp1, Fp2, F7, F8, Fz, F3, F4, Cz, C3, C4, Pz, P3, P4, T3, T4, T5, T6, O1, and O2; international 10-20 system) referred to an electrode at the nose tip, using EEG-1100 (Neurofax, Nihon Koden). Horizontal and vertical electro-oculograms (EOGs) were also recorded using four electrodes placed over the outer canthi and in the superior and inferior areas of the orbit. The electrode impedance was kept below 5 k-. The ERP and EOG data were band-pass filtered between 0.27 and 300 Hz, and sampled at a rate of 683 Hz. For the ERP analysis, each stimulus epoch began 500 ms prior to, and continued 1000 ms after, the onset of the first marker (**Figure 1**). The participant was instructed to close their eyes and yet to stay alert. Trials that included artifacts, defined as waves for which voltage exceeded ±100μV at one or more electrodes, were excluded from the analyses.

## **RESULTS**

### **BEHAVIORAL DATA**

**Figure 2** shows the results of the equal/unequal judgments. We assessed the equal response ratio defined as the proportion of trials in which participants judged the two time intervals as equal. After an inverse sine transformation, response ratios in Experiments 1 and 2 were subjected to ANOVA (T1 − T2: −120, −80, −40, 0, +40, +80, +120 ms)1 . There were significant main effects of T1 − T2 [Experiment 1: *<sup>F</sup>*(6, 84) <sup>=</sup> <sup>46</sup>.89, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.76, and Experiment 2: *<sup>F</sup>*(6, 84) <sup>=</sup> <sup>44</sup>.60, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.75]. Dunnett's *post-hoc t*-test was performed for each experiment to check whether the equal response ratios obtained from 6 stimulus patterns (T1 − T2 = −120, −80, −40, +40, +80, +120 ms) differed from that for the stimulus pattern of physically equal time intervals (T1 − T2 = 0 ms). The response ratios differed significantly from that obtained for T1 − T2 = 0 ms when T1 − T2 was −120, +80, or +120 ms both in Experiment 1 (200|200 vs. 80|200: *p* < 0.001, 200|200 vs. 280|200: *p* < 0.001, 200|200 vs. 320|200: *p* < 0.001) and in Experiment 2 (200|200 vs. 200|80: *p* < 0.001, 200|200 vs. 200|120: *p* < 0.001, 200|200 vs. 200|320: *p* < 0.001). In both experiments, T1 was perceived as equal to T2 when the difference between T1 and T2, T1 − T2, was in an asymmetrical range from −80 to 40 ms. The asymmetrical temporal assimilation indeed occurred.

### **ERP DATA**

ERPs were obtained by averaging the EEG waveforms for each of the seven stimulus patterns. **Figure 3** shows grand averaged ERP

<sup>1</sup>Because the two experiments were done about 1 year apart, we analyzed the behavioral data from each experiment separately.

waveforms of 13 participants elicited in the condition where T1 = T2 = 200 ms in Experiment 1. N1 appeared maximally at the central area (Cz). A CNV-like component appeared at the frontal area (Fz) during the stimulus presentation. The SNCt emerged in the right-frontal area at approximately 300 ms after the first marker and lasted until 400 ms after the third marker. The SNCt amplitudes in the experimental session were greater in the right-frontal areas than those in the left corresponding areas. These components were observed in all stimulus patterns and in Experiment 2 as well.

## *N1*

First, we analyzed the ERP components at the central electrode (Cz) to check the spatiotemporal characteristics of the ERP components related to the temporal judgment. We focused on a transient negative component (N1) at Cz, corresponding to the three sound markers' onsets (N1S1, N1S2, and N1S3), and checked the attentional effects on the N1S2 that separated the two intervals. In the analysis, all sound marker onsets were located relative to the timing of each marker onset. We selected mirror-pairs of stimulus patterns where T1 and T2 were perceived as nearly equal in the behavioral results (see **Figure 2**), even though the physical durations of these intervals were different (i.e., 160|200, 200|160, and 240|200, 200|240). The differences of N1 peak latency to each of the three sound markers in the experimental session were measured from the baseline. A 3 (N1: N1S1, N1S2, and N1S3) × 2 (Experiment: 1 and 2) ANOVA with repeated measures was performed for each pair of stimulus patterns to check whether or not N1 latencies varied related to equality perception. The Greenhouse-Geisser correction was applied to the ANOVA when the sphericity assumption was violated in the dependent measures. The Bonferroni correction for multiple post-hoc comparisons was applied when required. The η<sup>2</sup> *p* (partial eta-squares) were calculated for the quantitative comparison of effect sizes. Given that the N1 latency to each sound marker reflects attentional effects on the sensory signals, the N1 peak might be shifted to the temporal point where the two time intervals are assimilated. **Table 1** and **Figure 4** show the means (and SDs) of the peak N1 latencies for N1S1, N1S2, and N1S3 of each stimulus pattern in the experimental session, measured relative to each marker onset. For the mirror-pair of 160|200 and 200|160, there was a main effect of N1 [*F*(2, 24) = 6.26, *p* < 0.01, η2 *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.34]. The effect of Experiment [*F*(1, 12) <sup>=</sup> <sup>0</sup>.13, *n.s.*, <sup>η</sup><sup>2</sup> *p* = 0.01] and the interaction of N1 and Experiment [*F*(2, 24) = 2.33, *n.s.*, η<sup>2</sup> *<sup>p</sup>* = 0.15] were not significant. Peak latencies for N1S2 and N1S3 extended gradually, and the differences in N1 peak latencies for each mirror-pair did not appear. For the mirror-pair of 240|200 and 200|240, the main effects of N1 [*F*(2, 24) = 3.93, *p* < 0.05, η<sup>2</sup> *<sup>p</sup>* = 0.25] was significant. Neither the main effects of Experiment [*F*(1, 12) <sup>=</sup> <sup>0</sup>.34, *n.s.*, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.03] nor the interaction of N1 and Experiment [*F*(2, 24) <sup>=</sup> <sup>1</sup>.78, *n.s.*, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.12] were significant. As in former patterns, the differences of N1 peak latencies for each mirror-pair did not appear.

## *CNV*

The characteristics of neural activity during stimulus presentation were examined. The CNV difference waves at the frontal electrode (Fz) in the experimental session and the control session were calculated over two successive 100-ms time windows (TWcnv) from the onset of the second marker to the onset of the third marker (i.e., T2) of Experiment 1. The corresponding TWs in Experiment 2 were also calculated (i.e., from the onset of the second marker to the time windows 200 ms after the second marker)2 . The CNV difference waves at Fz in each of the seven standard stimuli in Experiment 1 (80|200, 120|200, 160|200, 200|200, 240|200, 280|200, and 320|200) and Experiment 2 (200|80, 200|120, 200|160, 200|200, 200|240, 200|280, and 200|320) were integrated within each TWcnv on Fz for each participant. The CNV difference waves in the frontal area should increase as T1 was lengthened in Experiment 1, if they accompanied the process of memorizing the lengths of T1. **Figure 5** shows the amplitude differences of the CNVs between the experimental session and the control session in TWcnv1 and TWcnv2. The CNV difference waves of TWcnv1 and TWcnv2 in Experiment 1 were fitted to a linear regression curve. Adjusted R-squared and Spearman's ρ were calculated to check whether the CNV amplitude differences in TWcnv1 and TWcnv2 changed as a function of preceding time intervals (i.e., T1). The length of T1 and the averaged CNV differences for TWcnv1 and TWcnv2 were positively correlated (TWcnv1:*R*<sup>2</sup> <sup>=</sup> <sup>0</sup>.58, TWcnv2:*R*<sup>2</sup> <sup>=</sup> <sup>0</sup>.84) and significant (TWcnv1:Spearman's ρ = 0.82, *p* < 0.02, TWcnv2: Spearman's ρ = 1.00, *p* < 0.01). In Experiment 2, the CNV difference waves of TWcnv1 and TWcnv2 did not change, as expected from the present results of Experiment 1 and our previous results (Mitsudo et al., 2009; Experiment 2).

<sup>2</sup>We set TWcnv1 and TWcnv2 to 200 ms after the second marker for both Experiments 1 and 2. In the cases of T2 = 80, 120, 160 ms, the difference waves of these three stimulus patterns would not reflect the memory functions of preceding intervals, because these difference waves were assumed to contain the component after the third marker, and they could not be separated from the SNCt.

**condition where T1 = T2 = 200 ms (physically equal) in Experiment 1.** The two neighboring time intervals were perceived as equal. Red lines represent the ERPs obtained when the participants made equal/unequal

listened to the stimuli passively. N1 peak latencies were clearly observed at Cz for both experimental and control tasks. The CNV emerged in the frontal areas and the SNCt emerged in the right-frontal areas.

**Table 1 | N1S1, N1S2, and N1S3 latencies of each stimulus pattern.**


## *SNCt*

We focused on six frontal electrodes (Fp1, Fp2, F7, F8, F3, and F4) where a post-stimulus SNCt emerged. Because any temporal comparison must have taken place only after the participant had a chance to perceive both of the neighboring time intervals, the ERPs corresponding to the judgment were expected to appear after the third marker (Paul et al., 2011). To examine the SNCt, the stimulus epoch up to 400 ms after the end of the third marker was divided into four time windows (TWSNCts) of 100 ms: TWSNCt1 to TWSNCt4. We calculated the SNCt difference waves by subtracting the mean SNCt amplitudes in the control session from those in the experimental session. The SNCt difference waves were integrated within each TWSNCt on all of the 19 scalp electrodes. The integrated values of six frontal electrodes (Fp1, Fp2, F7, F8, F3 and F4) were used for further statistical analyses. The means (SDs) of the SNCt difference waves are shown in **Table 2**.

We first divided the ERPs into two groups: those obtained in the conditions where equal judgments dominated (i.e., T1 − T2 = −80, −40, 0, +40 ms) and those obtained in the conditions where unequal judgments dominated (i.e., T1 − T2 = −120, +80, +120 ms). **Figure 6** shows the color maps of the brain activity corresponding to equal- and unequaldominant stimulus patterns in Experiment 1 (**Figure 6**, left figure) and Experiment 2 (**Figure 6**, right figure), in which the SNCt difference waves up to 400 ms after the third marker were averaged. A remarkable difference between these two groups was observed in the frontal area. A three-way (4 time windows (TWs) × 2 laterality × 2 equality) repeated-measures ANOVA was performed over left- (Fp1, F7, and F3) and right- (Fp2, F8, and

equal in behavioral data even though the lengths of T1 and T2 were

physically different.

F4) frontal electrodes, to check for the effects of laterality and equal/unequal judgment in each TW. In Experiment 1, the main effect of equal/unequal judgment was significant [*F*(1, 12) = 5.95, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.03, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.33]. Multiple comparisons with the Bonferroni correction revealed that the SNCt in the unequal-dominant stimulus patterns was significantly larger than that in the equaldominant stimulus patterns (equal vs. unequal: *p* = 0.03). The effect of laterality was not significant [*F*(1, 12) <sup>=</sup> <sup>0</sup>.003, *n.s.*, <sup>η</sup><sup>2</sup> *p* = 0.00]. The interaction between equal/unequal judgment and laterality was significant [*F*(1, 12) <sup>=</sup> <sup>4</sup>.90, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.04, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.29]. Multiple comparisons showed that the SNCt in the right-frontal area was larger in unequal-dominant stimulus patterns than in equal-dominant stimulus patterns (equal vs. unequal: *p* = 0.011). In Experiment 2, the main effect of laterality was significant [*F*(1, 12) <sup>=</sup> <sup>10</sup>.38, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.007, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.46]. The SNCt in the right frontal area was significantly larger than that in the left (right vs. left: *p* = 0.007). The effect of equality was not significant [*F*(1, 12) <sup>=</sup> <sup>0</sup>.49, *n.s.*, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.04]. The interaction between laterality and TWs was significant [*F*(1.53, <sup>1</sup>.84) = 4.61, *p* = 0.03, η2 *<sup>p</sup>* = 0.28]. Multiple comparisons indicated that the neural activity derived from the right-frontal electrodes was larger than that derived from the left-frontal electrodes between 0 to 400 ms after the onset of the third marker (TW1: *p* = 0.02, TW2: *p* = 0.008, TW3: *p* = 0.009, and TW4: *p* = 0.004).

To investigate the relationship between the SNCt and the judged equality/inequality, we performed a new type of selective averaging of the ERP data. Trials in which participants responded "equal" or "unequal" were averaged separately. We took the data obtained when T1|T2 = 280|200 and 200|280 for the following reasons. First, in these patterns, the temporal differences between T1 and T2 were both physically 80 ms. Second, behavioral data showed that the perception for these temporal patterns had some ambiguity: these patterns caused both "equal" and "unequal" judgments to substantial amounts, although the "unequal" judgment dominated for 280|200, while the "equal" judgment dominated for 200|280. Waveforms of T1|T2 = 280|200 and 200|280 were divided and averaged selectively in terms of "equal" and "unequal" responses. The SNCt difference waves were calculated by subtracting the mean SNCt amplitudes in "equal" responses from those in "unequal" responses. As in the stimulus-based analysis, the SNCt difference waves were integrated within each TWSNCt on all of the 19 scalp electrodes, and the integrated values of 6 frontal electrodes (left: Fp1, F7, and F3, and right: Fp2, F8, and F4) were used for the analyses.

**Figure 7** shows response-based selective averaging ERPs of T1|T2 = 280|200 and 200|280, obtained from three frontal electrodes of right (Fp2, F8, and F4) and left (Fp1, F7, and F3) in Experiments 1 and 2 [**Figure 7**, 1(a)–4(a)]. "Unequal" responses were averaged: 531 trials in Experiment 1, and 424 trials in Experiment 2. "Equal" responses were averaged as well: 293 trials in Experiment 1, and 639 trials in Experiment 2. We compared the averaged waveforms of the left and right SNCt between "equal" and "unequal" judgments of T1|T2 = 280|200 and 200|280, by conducting a paired *t*-test at each time point. The figures below each waveform are the results of *p*-values from the paired *t*-test (*df* = 12, *p* < 0.05) between "equal" and "unequal"

The upper graph shows the results of Experiment 1, while the lower graph of Experiment 2. The waveforms **A** and **B** are indicated to

1, the CNV amplitudes in the experimental task increased as T1 was lengthened.

response-based averaging [**Figure 7**, 1(b)–4(b)]. The results of the paired *t*-test showed that the averaged waveforms in the rightfrontal electrodes were large when participants judged two time intervals as subjectively "unequal" both in Experiments 1 and 2. This tendency was observed both in 280|200 (an unequaldominant stimulus pattern) and 200|280 (an equal-dominant stimulus pattern). The ERP differences between "equal" and "unequal" judgments started at 520 ms, approximately 40 ms after the third marker in Experiment 1 and at 480 ms, immediately after the third marker in Experiment 2, corresponding to the SNCt.

We further conducted a two-way (4 time windows (TWs) × 2 laterality) repeated-measures ANOVA for each of T1|T2 = 280|200 and 200|280, to check for effects of laterality in each TW. For both in Experiments 1 and 2, the main effects did not reach significance either in the time windows [Experiment 1: *<sup>F</sup>*(1.26, <sup>15</sup>.2) <sup>=</sup> <sup>0</sup>.42, *n.s.*, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.03, and Experiment 2: *<sup>F</sup>*(1.8, <sup>21</sup>.5) <sup>=</sup> <sup>0</sup>.41, *n.s.*, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.03], or in the laterality [Experiment 1: *<sup>F</sup>*(1, 12) <sup>=</sup> <sup>0</sup>.59, *n.s.*, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.05, and Experiment 2: *<sup>F</sup>*(1, 12) <sup>=</sup> <sup>3</sup>.2, *n.s.*, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.21].

## **DISCUSSION**

The purpose of this study was to examine the characteristics of three types of ERP components, N1, CNV, and SNCt, which can be associated with temporal equality/inequality. Behavioral results showed that assimilation took place in an asymmetrical time range of −80 ≤ (T1 − T2) ≤ +50 ms. This result agrees with


**Table 2 | The means (SDs) of the SNCt difference waves categorized by the equal-dominant and the unequal-dominant stimulus patterns at left (Fp1, F7, and F3) and right (Fp2, F8, and F4) frontal electrodes.**

Values are in μV.

previous psychophysical findings (Nakajima et al., 2004; Miyauchi and Nakajima, 2007; Mitsudo et al., 2009), and demonstrates the robustness of this asymmetrical tendency. The N1 appeared corresponding to the physical onset of each marker, and the CNV appeared as a function of T1. These two components appeared during the presentation of the stimuli. The SNCt appeared at the right-frontal brain area 0–400 ms after the third marker, also for the new stimulus patterns in Experiment 2. The reproducibility suggests that this component is related to the equality/inequality perception of time intervals. Our results also show that the analysis of several different ERP indices is necessary to understand temporal processing, from stimulus detection to decision making related to the equality/inequality perception.

### **AUDITORY EVOKED ONSET RESPONSES CORRESPONDING TO THE THREE TEMPORAL MARKERS**

Peak latencies of N1 in response to the sound markers appeared corresponding to the physical timing of each marker onset (**Figure 4** and **Table 1**). The differences in N1 peak latencies between two mirror-pairs for each sound marker, N1S2, N1S1, and N1S3, did not exhibit the statistical differences between the pairs of stimulus patterns. The N1 peak latencies to each sound marker appeared constantly at approximately 100 or 110 ms after stimulus onset. The specific allocation of temporal attention would not reflect on the timing of these sensory signals. This is somewhat against of previous studies reporting the enhancement of N1 amplitude (e.g., Hillyard et al., 1973; Lange et al., 2003) and the shortening of N1 latency (Okamoto et al., 2007) caused by attention—a recent report has suggested that the N1 is more closely related to the temporal structures of stimulus patterns than the CNV (Kononowicz and van Rijn, 2014). In our study, N1 responses that separated the intervals did not exhibit any latency shortening related to temporal judgment. The N1 in the present case must have been related to neural activity that was timelocked to the onsets of the stimuli, but not to the attention that affects equality/inequality judgment.

## **THE CNV ACTIVITY OBSERVED OVER THE FRONTAL SITE AFTER THE SECOND MARKER**

In Experiment 1, the CNV amplitudes in the experimental task increased as T1 was lengthened. In Experiment 2, in which T1 was always fixed at 200 ms, the CNV kept the same amplitude during the same temporal windows as in Experiment 1 (**Figure 5**). When the first interval (T1) was varied from 80 to 320 ms in Experiment 1, the information regarding the duration of T1 was probably retained in subsequent time windows to compare it with the second interval (T2). Thus, the memorized duration of T1 was reflected to the EEG changes in TW1 and TW2, and the CNV amplitudes increased linearly as a function of T1. In contrast, when the first intervals were fixed (at 200 ms in the current study) in Experiment 2, the preserved information of T1 should have been constant across all the stimulus conditions. This explains the fact that the CNV amplitudes did not change. Previous ERP studies adopting temporal judgment tasks have reported that the CNV amplitudes became larger when perceived time length was estimated as longer (Pfeuty et al., 2003b; Le Dantec et al., 2007; Mitsudo et al., 2012; Gontier et al., 2013). The CNV, which was defined as the brain activity up to 200 ms after the onset of the second marker in the present study, must have changed depending on the memorized duration of the intervals. The CNV observed over the frontal site was assumed to be related to the memorization of time intervals, which is in line with previous studies where the CNV amplitudes were larger when subjectively judged time was longer (Macar et al., 1999; Pfeuty et al., 2003a,b; Le Dantec et al., 2007; but see different views; Kononowicz and van Rijn, 2011, 2014; van Rijn et al., 2011).

### **BRAIN ACTIVITY DERIVED FROM EQUAL-DOMINANT/UNEQUAL-DOMINANT STIMULUS PATTERNS**

The SNCt emerged most prominently around the right-frontal electrodes. The SNCt observed in our previous study appeared clearly in a new set of data recorded from different participants for increased stimulus patterns. When the ERPs were divided for equal-dominant and unequal-dominant stimulus patterns, the SNCt derived from the right-frontal brain area was larger in the unequal-dominant stimulus patterns (**Figure 6**). Previous studies that examined the EEG signatures of temporal discrimination argued that the decision processes in temporal discrimination were reflected in a component that appeared after the presentation of both durations to be compared (Gontier et al., 2009; Paul et al., 2011). The SNCt, which involved brain activities after T1 and T2, was considered as an index of decision processes regarding the subjective temporal judgments. This is in accord with the notion that the right-frontal brain area plays a crucial role in the perception of time (Pfeuty et al., 2003a,b; Rubia and Smith, 2004; Hairston and Nagarajan, 2007).

## **NEURAL CORRELATES OF PERCEPTUAL EQUALITY/INEQUALITY**

The most interesting finding in the current study was that the SNCts at the right-frontal electrodes showed larger activities

when the brain processed perceptual inequality, rather than equality, of the two neighboring time intervals, which was revealed in T1|T2 = 280|200 and 200|280 (**Figure 7**). Even if participants listened to the same stimulus pattern, the SNCt in the right-frontal electrodes was different when they judged two neighboring time intervals as "equal" or "unequal." This tendency was observed both in 280|200 (an unequal-dominant stimulus pattern) and 200|280 (an equal-dominant stimulus pattern). The magnitude of the SNCt changed corresponding to the behavioral responses even to physically identical stimulus patterns. Previous studies reported that the slow positive component that is considered to be related to the decision processes appeared in the prefrontal cortex during duration discrimination and it emerged within 500 ms after the stimulus offset (Gontier et al., 2009; Paul et al., 2011); it is very likely that the SNCt was also related to the decision processes regarding the equality/inequality of two neighboring time intervals.

Earlier studies have documented that the brain attenuates its activities when a temporal task is performed more efficiently (Casini and Macar, 1996). The magnitudes of SNCt can be connected to the economic information processing in the brain (Nakajima et al., 2004). When the successively presented sounds are assumed to create regular time intervals, the brain is probably able to save its activity. This may result in the low SNCt amplitude at the right-frontal areas in the equal responses.

According to a psychophysical model of unilateral temporal assimilation (Nakajima et al., 2004), the perceived difference between T1 and T2 could be reduced by cutting the processing time for T2 after the offset of the third marker. If this model works in the present experimental paradigm, the whole of the processing, including the detection of the markers, basically continues about 80 ms after the third marker's onset. In T1|T2 = 200|280, in which unilateral temporal assimilation (time-shrinking) probably occurred, the SNCt differences appeared almost immediately when the third marker was presented. This may show the process to reduce the processing time. In T1|T2 = 280|200, in which temporal assimilation usually would not occur, the SNCt differences of perceptual equality/inequality started about 40 ms after the third marker. This suggests that the brain activation corresponding to "unequal" perception appeared within 80 ms after the stimuli. Hence, the model is likely to explain the processing of temporal assimilation in the brain. The SNCt continued up to about 400 ms after the end of T2, and was established as a signature of equality/inequality judgment caused by the comparison of the two neighboring time intervals.

Previous literature has reported that the right dorsolateral prefrontal cortex is involved in tasks of cognitive time estimation (Rubia and Smith, 2004), especially in comparison of time intervals (Rao et al., 2001). The SNCt, which is related

**FIGURE 7 | Averaged waveforms of 13 participants obtained from 3 right-frontal (Fp2, F8, and F4) and 3 left-frontal (Fp1, F7, and F3) electrodes in Experiment 1 [1(a) and 2(a)] and Experiment 2 [3(a) and 4(a)].** Black lines represent the ERPs for participants' unequal perception, while red lines those for participants' equal perception. The figures below each waveform are the results of paired t-tests (p < 0.05, two-tailed) between "equal" and "unequal" response-based averaging waveforms [1(b), 2(b), 3(b), and 4(b)]. The time interval of significance was identified when a paired t-test reached significance. The open circles in each figure represent

p-values in terms of the comparison between "equal" and "unequal" waveforms at each time point. The horizontal black lines in the figure represent the significance level (p < 0.05). Both in Experiment 1 and Experiment 2, the SNCt in the right-frontal electrodes was large when participants judged two time intervals as subjectively "unequal." The ERP differences between "equal" and "unequal" perceptions started within 80 ms after the stimulus in T1|T2 = 280|200 (an unequal-dominant stimulus pattern), while immediately after the third marker in T1|T2 = 200|280 (an equal-dominant stimulus pattern) in Experiment 2.

to the equal/unequal judgments, emerges most prominently around the right-frontal electrodes (i.e., Fp2, F8, and F4). This suggests that the right dorsolateral prefrontal cortex could be a generator of the SNCt (**Figures 3**, **6**). Other imaging techniques, such as magnetoencephalography, should be introduced in order to clarify the spatio-temporal characteristics of this component. Brain activity related to the perceptual equality/inequality of neighboring time intervals thus appeared clearly, and "equal" judgments and "unequal" judgments corresponded to different ERP patterns—for the same stimulus patterns.

### **ACKNOWLEDGMENTS**

We are grateful to Naruhito Hironaga for his constructive opinions on an earlier version of our manuscript. Also we would like to express our gratitude to James M. Broadway and Makiko Sadakata for their valuable suggestions for improving our manuscript. This study was partly supported by a grant from the Ono Acoustics Research Fund to Takako Mitsudo.

### **REFERENCES**


theoretical evaluation. *Front. Integr. Neurosci*. 5:91. doi: 10.3389/fnint.2011. 00091

Walter, W. G., Cooper, R., Aldridge, V. J., McCallum, W. C., and Winter, A. L. (1964). Contingent negative variation: an electric sign of sensori-motor association and expectancy in the human brain. *Nature* 203, 380–384. doi: 10.1038/203380a0

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 May 2014; accepted: 06 August 2014; published online: 23 September 2014.*

*Citation: Mitsudo T, Nakajima Y, Takeichi H and Tobimatsu S (2014) Perceptual inequality between two neighboring time intervals defined by sound markers: correspondence between neurophysiological and psychological data. Front. Psychol. 5:937. doi: 10.3389/fpsyg.2014.00937*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Mitsudo, Nakajima, Takeichi and Tobimatsu. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Interval discrimination across different duration ranges with a look at spatial compatibility and context effects

## *Giovanna Mioni 1,2\*, Franca Stablum2 and Simon Grondin1*

<sup>1</sup> École de Psychologie, Université Laval, Québec, QC, Canada

<sup>2</sup> Department of General Psychology, University of Padova, Padova, Italy

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

#### *Reviewed by:*

Joseph Glicksohn, Bar-Ilan University, Israel Emi Hasuo, Kyushu University, Japan

### *\*Correspondence:*

Giovanna Mioni, École de Psychologie, Université Laval, Pavillon Félix-Antoine-Savard, 2325, rue des Bibliothèques, Québec, QC G1V 0A6, Canada e-mail: mioni.giovanna@gmail.com

In the present study, a time discrimination task was used to investigate the effect of different contexts for intervals varying from 400 to 1600 ms. A potential time-space interaction was controlled, and participants used both manual responses (Experiments 1 and 2) and vocal responses (Experiment 3). Three ranges of durations were employed (short, middle and long), and within each range condition, three standard values were used (400, 700, and 1000 ms; 700, 1000, and 1300 ms; and 1000, 1300, and 1600 ms). Within each range, standard intervals were randomized (Experiments 1 and 3) or remained constant (Experiment 2) within a block of trials. Our results suggest that context influences time discrimination performances only when the temporal range under investigation is below 1300 ms and the temporal intervals varied within blocks. In the case of temporal intervals longer than 1300 ms, participants presented a tendency to respond "long" independently of the procedure used. Moreover, our results suggested that performances in a discrimination task are mainly influenced by the fact of varying standard durations within blocks, and not much by the time-space compatibility.

**Keywords: time discrimination, context effect, spatial compatibility, temporal intervals, manual responses, verbal responses**

### **INTRODUCTION**

There are different signs in the time perception literature revealing the vulnerability of psychological time; indeed, different contexts move the output of temporal mechanisms in different directions. Amongst the contexts exerting influences on time estimation, there is the fact of dealing with different temporal intervals within a given investigation. An extensive literature shows that the duration of an event is not solely experienced on the basis of its temporal properties: attention (Zakay, 1998; Brown and Boltz, 2002; Grondin and Rammsayer, 2003; see also Zakay and Block, 1996), arousal and emotional levels (Angrilli et al., 1997; Mella et al., 2011; Droit-Volet et al., 2013; Grondin et al., 2014), and stimulus context (Barnes and Jones, 2000; McAuley and Jones, 2003; Jones and McAuley, 2005) can all affect the experience of time. Additionally, the time scale of the stimulus and the task used to measure participants' subjective duration have a great influence on the mechanisms involved in temporal processing (Gil and Droit-Volet, 2011; Mioni et al., 2014b).

When investigating time perception, a major concern is related to the temporal range under investigation. In the field of time perception, researchers have mainly used intervals in the range of 100 ms to a few seconds (Grondin, 2001, 2010). This temporal range is particularly important in humans because it involves processes from motor control, speech generation, playing music, and dancing to more complex processes like learning and decision making (Buhusi and Meck, 2005).

A general tendency in timing literature, mainly in neuroscience researches, is to emphasize a distinction between intervals above and below 1 s, which is based on differential pharmacological effects (Rammsayer, 2008) and on patient studies with various brain damages (see Ivry and Spencer, 2004; Meck, 2005; Mioni et al., 2014a; Piras et al., 2014). Moreover, researchers claimed that processing of smaller intervals is more sensory based, or benefits from some automatic processing, whereas the processing of longer intervals requires the support of cognitive resources (also see Lewis and Miall, 2003; Hellström and Rammsayer, 2004). Even if this "1-s" transition period remains somewhat arbitrary, there is certainly some turning point on the time continuum given the benefit one should expect from adopting an explicit counting strategy for processing long temporal intervals (Grondin et al., 2004; Grondin and Killeen, 2009a,b). Indeed, there are empirical reasons to believe that this transition occurs circa 1.2 s (Grondin et al., 1999), at least for the processing of auditory time intervals, the Weber fraction for time increasing for intervals longer than 1.3–1.5 s (Gibbon et al., 1997; Grondin, 2012, 2014).

However, processing temporal intervals cannot be independent from methodological issues. For example, in a typical time discrimination task, participants are required to judge the relative durations of two temporal intervals successively presented (first "standard" and second "comparison"). Presenting intervals successively induces some bias in the perceived duration of intervals, this effect being known as the time-order error (TOE). A positive TOE is observed when the first stimulus presented is over-estimated whereas a negative TOE is observed when the first stimulus is under-estimated, compared to the second. Researchers have explained the TOE as the result of a response bias or of a perceptual effect (Allan, 1977; Hellström, 1985, 2003; Eisler et al., 2008). Providing participants with information about correct responses (feedback) in time discrimination tasks eliminates the tendency for judging the second duration as longer than the first (Jamieson and Petrusic, 1975, 1978). Such a result is consistent with the view that TOE is a reliable perceptual effect and that practice with feedback leads participants to adopt biased decision criteria to overcome this effect.

Moreover, and as revealed by Vierordt's law (Vierordt, 1868; Lejeune and Wearden, 2009), when short and long intervals are presented within the same experimental context, shorter intervals tend to be overestimated and longer intervals are underestimated. The point in between, for which there is no constant error, is called the *indifference point*. The estimated value of the indifference point indeed depends on the durations used in the experiment (Eisler et al., 2008; Lejeune and Wearden, 2009).

Jones and McAuley (2005) tried to give a comprehensive explanation of temporal performance using time discrimination tasks. They reported interesting results describing local and global context effects. The authors pointed out that the temporal context systematically affects the perception of a temporal interval. The authors used an experimental setting in which a series of brief tones are presented, which determine an isochronous sequence of inter-onset intervals (Base IOIs). The Base IOI sequence precedes the two final pairs of stimuli (standard and comparison) that have to be compared. In this paradigm, the rate of the Base IOI sequence alters the perceived duration of the standard interval, producing a *local context effect* (Barnes and Jones, 2000; McAuley and Jones, 2003). The alteration of the perceived duration occurs because the local context sequence induces an internal periodicity that distorts participants' perception of the standard IOI in the direction of the Base IOI (over- or under-estimation according to the Base IOI). Barnes and Jones (2000) also reported that the rate of other sequences within the same session affects the perceived duration of the standard IOI, producing a global context effect (see also McAuley and Jones, 2003; Large, 2008).

Another, more recent line of investigation rather explains temporal performance from a time-space compatibility perspective (*mental time line*; Ishihara et al., 2008). These studies describe an association between temporal duration and the spatial position of the response keys on the keyboard: specifically, the congruity between spatial and temporal information along the "mental time line" may facilitate manual responses, which may yield a *spatial–temporal association of response codes* (STEARC) effect (Ishihara et al., 2008). Short temporal durations are associated with left space, and long temporal durations are associated with right space. The time-space interaction follows the idea that time, space, and numbers are processed by a common system (Walsh, 2003), a magnitude mechanism that codes information according to a quantitative representation, usually outlined as a left-to-right orientation continuum. Specifically for the temporal domain, in the context of time discrimination tasks, participants are presented with pairs of temporal intervals (standard duration presented first and comparison duration presented second) and have to judge if the second interval presented is longer (or shorter) than the standard. For time discrimination task, the presentation order is critical in the experimental setting. Higher accuracy is expected when the duration of the comparison stimulus is short and the "short" response ispositioned on the left side, compared to the condition in which the duration of the comparison stimulus is short and the "short" response is positioned on the right side. Opposite performance patterns are expected when the duration of the comparison is long; in fact, in this case, higher accuracy is expected when the right response key is associated with the "long" response, compared to the condition in which the "long" response is positioned on the left side (Conson et al., 2008; Ishihara et al., 2008; Bonato et al., 2012). Therefore, it is possible that the negative or positive TOE, often observed in time discrimination task, is partly caused by time-space compatibility rather than a memory or perceptual process related to the temporal interval under investigation.

In the present study, we first wanted to know if using different temporal contexts affects duration discrimination. In particular, we focused on "1-s" temporal interval and we included it within different temporal contexts or as the longest, the medium or the shortest standard temporal interval. In this way, it was possible to determine if participants' performance depends specifically on the temporal interval used or if it is modulated by the context within which this interval is included. Our use of context effect is inspired more by the work related to Vierordt's law than by the context effect suggested by Jones and McAuley's (2005). In fact, in Jones and McAuley's (2005) studies the influence of context on temporal performance was induced by the presentation of sequences of brief tones. In the present study, participants performed a time discrimination task in which the 1-s interval was included in blocks of trials, which include different temporal intervals (longer or shorter than 1-s temporal intervals).

A second aim of the present study was to investigate the time-space compatibility in order to understand the influence of response lateralization on temporal performances. For this purpose, participants were asked to respond manually (pressing a designed lateralized response key on the keyboard) or to respond orally. It was then possible to determine if the preference in responding short or long depends on the compatibility between temporal intervals (short-long) and position of the response keys (left-right), i.e., time-space compatibility.

## **EXPERIMENT 1**

### **METHODS**

### *Participants*

Fifty-six students from the University of Padova (Italy) were randomly assigned to one of three experimental groups: 20 participants in Group 1 (*M* = 21.70 years; *SD* = 0.97) for which standard durations lasted 400, 700, and 1000 ms; 18 participants in Group 2 (*M* = 21.56 years; *SD* = 1.50) for which standard durations lasted 700, 1000, and 1300 ms, and 18 participants in Group 3 (*M* = 22.00 years; *SD* = 1.84) for which standard durations lasted 1000, 1300, and 1600 ms. This and the following studies, were conducted in accordance with the Department of General Psychology guidelines, and all participants completed an informed consent form.

### *Materials*

Each participant was tested in a quiet room at the Department of General Psychology of the University of Padova, (Italy). Each test session lasted approximately 20 min. All stimuli were presented on a 15-inch PC monitor and participants were seated at a distance of approximately 60 cm. The stimulus marking intervals to be discriminated was a gray dot centrally presented on a white background (**Figure 1**). We used E-Prime® 2.0 to program and implement the tasks.

### *Procedure*

Participants were instructed to complete a duration discrimination task by judging the relative duration of two time intervals successively presented. The first visual stimulus on the computer's screen marked the standard interval and the second one the comparison interval. For each group, three standard intervals were used. In Group 1 (short standard durations), the standards lasted 400, 700, and 1000 ms; in Group 2 (middle standard durations) they lasted 700, 1000, and 1300 ms; and in Group 3 (long standard durations) they lasted 1000, 1300, and 1600 ms. For each standard interval, one of two comparison stimuli was presented: ±25% with respect to the standard value (**Table 1**). Participants were seated at 60 cm from the computer screen and they were instructed to press two distinct keys: "B" if the second was shorter than the first one ("B" referred to the Italian word "Breve" = short) or "L" if the second was longer than the first one ("L" referred to the Italian word "Lungo" = long). For half of the participants the label with the letter "B" was placed over the letter "A" on the left of the keyboard and the label with the letter "L" was placed over the letter "L" on right of the keyboard; for the other half of the participants the label with the letter "B" was placed over the letter "L" on the left of keyboard and the label with the letter "L" was placed over the letter "A" on right of the keyboard. Twelve pairs of stimuli (standard—comparison) were presented within each block, and the standard durations were randomized within blocks. Stimuli sequences consisted of two gray circles separated by a 500-ms inter-stimulus interval; the next sequences of stimuli were presented 1000 ms after the participant's response. There were three blocks of trials in the experimental session. A practice phase was included at the beginning of the session in order to clarify the instructions and to familiarize participants with the task. One presentation of each pair of stimuli (standard—comparison)

**Table 1 | Summary of standard and comparison temporal intervals used in Experiments 1–3.**


was included in the practice phase. Participants were instructed to be accurate and fast in their responses, and no feedback was provided.

### *Statistical analyses*

Data were analyzed in terms of accuracy (proportions of correct responses) and perceived duration (proportion of "long" responses). For accuracy, an analysis of variance (ANOVA) according to a 3 (Group1—Short, Group 2—Middle, Group 3—Long) × 2 (Response key Short-left, Short-right) × 3 (Standard duration short, middle, and long) × 2 (Comparison short and long) design was conducted, with the Standard duration and Comparison being within-subject factors. An ANOVA on the proportions of "long" responses according to a 3 (Group1—Short, Group 2—Middle, Group 3—Long) × 2 (Response key Short-left, Short-right) × 3 (Standard duration short, middle, and long) design was conducted, the last factor being within-subjects.

To further investigate the effect of response key and spatial compatibility on time perception, we considered the responses as "congruent" when the comparison duration was short and the short response key was placed on the left side of the keyboard; and we considered the responses as "incongruent" when the comparison duration was short and the short response key was placed on the right side of the keyboard. Other ANOVAs were then conducted on proportions of correct responses according to a 3 (Group1—Short, Group 2—Middle, Group 3—Long) × 2 (Response key Congruent short-left, Incongruent short-right) × 2 (Standard duration short and long), the last factor being withinsubjects. All significant analyses were followed by *post-hoc* analyses performed with a Bonferroni correction to reduce the Type I error rate, and the effect size was estimated with partial eta squared (η<sup>2</sup> *p*).

### **RESULTS AND DISCUSSION**

### *Proportions of correct responses*

The mean proportions of correct responses as a function of groups, standard durations and comparisons are reported in **Figure 2A**. The ANOVA revealed a significant effect of group [*F*(2, 53) <sup>=</sup> <sup>5</sup>.44, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.007, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.179] and standard duration [*F*(2, 106) <sup>=</sup> <sup>15</sup>.27, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.234]. The analysis also showed a significant group × standard duration [*F*(4, 106) = 2.49, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.048, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.090] and standard duration × comparison interactions [*F*(2, 106) <sup>=</sup> <sup>64</sup>.49, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.563].

Moreover, the group × standard duration × comparison interaction [*F*(4, 106) <sup>=</sup> <sup>4</sup>.93, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.166] was also found (**Figure 2A**). *Post-hoc* analyses revealed similar patterns of performance in Group 1 and Group 2: participants were more accurate when the standard duration was short (standard-short = 400 ms in Group 1, and = 700 ms in Group 2) and when the comparison interval was shorter than the standard. No effect of comparison (standard-short vs. standard-long) was observed for the middle standard durations (standard-middle = 700 ms in Group 1 and = 1000 ms in Group 2) whereas, when the standard duration was long (standard-long = 1000 ms in Group 1 and = 1300 ms in Group 2), better performances were observed when the comparison was longer than the standard. In the case of Group 3, better performances were observed only when the standard was long (1600 ms) and when the comparison interval was longer than the standard.

No effect of comparison (*p* = 0.431) or response key (*p* = 0.104), as well as no interaction effect (all *p*s > 0.05), were significant.

### *Proportion of "long" responses*

The mean proportion of "long" responses as a function of groups and standard durations are reported in **Figure 3A**. The ANOVA revealed a significant effect of standard duration [*F*(2, 106) = 54.90, *p* < 0.001, η<sup>2</sup> *<sup>p</sup>* = 0.523], and the group × standard duration interaction [*F*(4, 106) <sup>=</sup> <sup>4</sup>.53, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.002, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.153]. *Posthoc* analyses revealed that participants in all groups had a tendency in responding "long" when the standard duration was the longer presented. Participants of Group 1 (short = 400 ms) responded "long" less often when the standard duration was short than participants of Group 3 (short = 1000 ms). No effect of group (*p* = 0.180) or of response key (*p* = 0.870), or other interactions, were significant (all *p*s > 0.05).

### *Time-space compatibility (congruent vs. incongruent)*

The analyses conducted to investigate the effect on accuracy of time-space compatibility (congruent vs. incongruent) revealed a significant effect of standard duration [*F*(1, 56) = 11.59, *p* < 0.001, η<sup>2</sup> *<sup>p</sup>* = 0.188] and a significant interaction between standard duration × response key [*F*(1, 56) = 5.36, *p* = 0.025, η2 *<sup>p</sup>* = 0.097] (**Figure 4A**). *Post-hoc* analyses showed that participants were less accurate when the standard duration was short and the response key was on the right side but no effect of response key was found for long responses. No effect of group (*p* = 0.891) or of response key (*p* = 0.227), or other interactions, were significant (all *p*s > 0.05).

The results showed different performance patterns, depending on the temporal intervals under investigation. In the case of Group 1 and Group 2, results showed a positive TOE when the standard intervals were the shortest (400 ms in Group 1 and 700 ms in Group 2) and the comparison was shorter than the standard; whereas a negative TOE was observed when the standard interval was the longest of the experimental setting (1000 ms for Group 1 and 1300 for Group 2) and the comparison was longer than the standard. In the case of Group 3, a general tendency in responding "long" produced a negative TOE independently of the standard duration (**Figure 2A**).

These results are consistent with the idea that there is some transition regarding the temporal processes operating with different duration ranges in the vicinity of 1200–1300 ms (Grondin et al., 1999). Further analyses of the influence of the position of the response key on temporal performance revealed lower performances when the standard duration was short and the response key was in the incongruent condition (short-right).

### **EXPERIMENT 2**

In order to further distinguish the effect of context from the effect of temporal interval, the standard duration was kept fixed within blocks in the present experiment.

## **METHODS**

### *Participants*

As in Experiment 1, 56 students (*M* = 22.18 years; *SD* = 1.72) from the University of Padova (Italy) were randomly assigned to one of three experimental groups: 20 participants in Group 1 for which standard durations lasted 400, 700, and 1000 ms; 18 participants in Group 2 for which standard durations lasted 700, 1000, and 1300 ms, and 18 participants in Group 3 for which standard durations lasted 1000, 1300, and 1600 ms. All participants provided informed consent to complete the study and none took part in Experiment 1.

### *Procedure and materials*

The experimental setting was similar to the one used in Experiment 1 with one key difference: the standard durations did not vary within blocks (**Figure 5**). In each block, participants were always presented with the same standard duration and the comparison interval was ±25% compared to the standard (**Table 1**). The presentation order of the blocks was randomized. As in Experiment 1, there was a practice phase and no feedback; participants were instructed to be accurate and fast in their responses. The response keys were counterbalanced between participants. Finally, the designs of the statistical analyses are the same as the ones described in Experiment 1.

## **RESULTS AND DISCUSSION** *Proportions of correct responses*

The mean proportions of correct responses as a function of groups, standard durations and comparison intervals are reported in **Figure 2B**. The ANOVA revealed a significant effect of standard

duration [*F*(2, 106) <sup>=</sup> <sup>6</sup>.27, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.003, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.111] and of comparison interval [*F*(1, 53) <sup>=</sup> <sup>12</sup>.42, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.199]. No main effect of group (*p* = 0.360), response key (*p* = 0.710) or other interactions were significant (all *ps* > 0.05).

### *Proportions of "long" responses*

The mean proportions of "long" responses as a function of groups and standard durations are reported in **Figure 3B**. The ANOVA revealed only a significant standard duration effect [*F*(2, 106) <sup>=</sup> <sup>3</sup>.03, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.050, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.054] indicating that participants pressed "long" more frequently when the standard duration was the longest presented, compared to the shortest (*p* > 0.05 for all other effects).

### *Time-space compatibility (congruent vs. incongruent)*

The analyses conducted on accuracy to investigate the effect of time-space compatibility (congruent vs. incongruent) (**Figure 4B**) revealed only a significant standard duration effect

[*F*(1, 53) <sup>=</sup> <sup>19</sup>.02, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.276] indicating that participants were more accurate when discriminating long temporal intervals. No other main or interaction effects were significant (all *ps* > 0.05).

Results from Experiment 2 showed high accuracy in all three Groups and a tendency in responding long when the standard duration was long and the comparison was longer than the standard. No effect of response key was observed, suggesting that this effect depends on the context in which the temporal interval is presented and is not specifically related to the single duration employed.

### **EXPERIMENT 3**

Pieces of scientific literature consistently suggest that there is an interaction between time and space (Ishihara et al., 2008; Bonato et al., 2012). In order to disentangle the specific effect of time-space compatibility from the effect of the temporal intervals employed, participants in Experiment 3 were asked to give vocal responses.

### **METHODS**

### *Participants*

As in Experiments 1 and 2, 51 students (*M* = 23.16 years; *SD* = 2.57) from the University of Padova (Italy) were randomly assigned to one of three experimental groups: 18 participants in Group 1 for which standard durations lasted 400, 700, and 1000 ms; 17 participants in Group 2 for which standard durations lasted 700, 1000, and 1300 ms, and 16 participants in Group 3 for which standard durations lasted 1000, 1300, and 1600 ms. All participants provided informed consent to complete the study and none took part in Experiments 1 or 2.

### *Procedure and materials*

The material and procedure were exactly as in Experiment 1 (see **Figure 1**), except that participants were instructed to respond vocally by saying "Breve" ("Breve" = "short" in Italian) or "Lungo" ("Lungo" = "long" in Italian), instead of pressing the response keys.

### **RESULTS AND DISCUSSION**

Given that participants were instructed to give vocal response, no effect of response key is taken into consideration. Data were analyzed in terms of accuracy (proportions of correct responses), and perceived duration (proportion of "long" responses). For accuracy, an ANOVA according to a 3 (Group1—Short, Group 2—Middle, Group 3—Long) × 3 (Standard duration short, middle, and long) × 2 (Comparison short and long) design was conducted, with the Standard duration and Comparison being within-subjects factors. An ANOVA on the proportions of "long" responses according to a 3 (Group1—Short, Group 2—Middle, Group 3—Long) × 3 (Standard duration short, middle, and long) design was conducted, the last factor being within-subjects.

### *Proportions of correct responses*

The mean proportions of correct responses as a function of groups, standard durations and comparison intervals are reported in **Figure 2C**. The ANOVA revealed a significant effect of group [*F*(2, 48) <sup>=</sup> <sup>19</sup>.85, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.453] and of standard duration [*F*(2, 96) <sup>=</sup> <sup>7</sup>.72, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.139]. The analyses also revealed a significant standard duration × comparison interval interaction [*F*(2, 96) <sup>=</sup> <sup>63</sup>.21, *<sup>p</sup>* <sup>&</sup>lt; <sup>0</sup>.001, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.568] and a significant group × standard duration × comparison interval interaction [*F*(4, 96) <sup>=</sup> <sup>4</sup>.59, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.002, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.161] (**Figure 2C**).

Post-hoc analyses revealed that in Group 1 and Group 2, participants were more accurate when the standard duration was short (standard = 400 ms in Group 1 and standard = 700 ms in Group 2) and the comparison interval is shorter than the standard. An opposite pattern of performance is observed when the standard durations are long (standard = 1000 ms in Group 1 and standard = 1300 ms in Group 2). In this case, better performances are observed when the comparison is longer than the standard. No effect of comparison interval was observed for the middle standard duration (standard = 700 ms in Group 1 and = 1000 ms in Group 2). In the case of Group 3, better performances are observed only with the middle and long standard durations (1300 and 1600 ms) when the comparison interval is longer. No main effect of comparison (*p* = 0.176) or other interactions were found (all *ps* > 0.05).

### *Proportions of "long" responses*

The mean proportions of "long" responses as a function of groups and standard durations are reported in **Figure 3C**. The ANOVA revealed a significant effect of standard duration [*F*(2, 96) = 62.77, *p* < 0.001, η<sup>2</sup> *<sup>p</sup>* = 0.567], as well as the group × standard duration interaction [*F*(4, 96) <sup>=</sup> <sup>4</sup>.14, *<sup>p</sup>* <sup>=</sup> <sup>0</sup>.004, <sup>η</sup><sup>2</sup> *<sup>p</sup>* = 0.147] indicating that participants in Group 3 had a greater tendency to respond "long" for short standard intervals (Group 3 = 1000 ms) than participants in Group 1 (Group 1 = 400 ms). No main effect of group was found (*p* = 0.340).

The results observed in Experiment 3 are consistent with the ones reported in Experiment 1. In both experiments, the standard durations were randomly presented within blocks. Whether vocal or manual responses are used, participants showed a positive TOE when the standard intervals was shorter (400 ms in Group 1 and 700 ms in Group 2) and the comparison was shorter than the standard; a negative TOE was observed when the standard interval was the longest of the experimental setting (1000 ms for Group 1 and 1300 for Group 2) and the comparison was longer than the standard. In the case of Group 3 a general tendency in responding "long" produced a negative TOE independently of the standard duration was observed.

Participants' performance was not modulated by the assignment of response keys, thus suggesting that processing duration in time perception did not involve spatial representation. This finding would not follow predictions of time-space compatibility, but would rather be consistent with alternative interpretation frameworks. In particular, the present data would be consistent with a sequential order system that represents items with respect to other items, via an inter-item association or on the basis of their ordinal position in the sequence without an absolute, spatially-defined reference (Marshuetz, 2005).

## **GENERAL DISCUSSION**

The present study was conducted for testing the effect of various temporal contexts and the effect of time-space compatibility on duration discrimination. In particular, we focused on 1-s temporal interval considering that below and above this duration, different processes (automatic vs. controlled) would be at play (Lewis and Miall, 2003; Hellström and Rammsayer, 2004; Rammsayer, 2008).

### **TEMPORAL CONTEXTS**

Results from Experiment 1 showed that participants' accuracy (percentage of correct responses) on time discrimination task depended on the stimulus duration and context. Interestingly, participants in Group 1 and Group 2 had a similar pattern of performance. In both cases, participants of these groups had a preference in responding "short" when the standard interval was the shortest, and in responding "long" when the standard duration was the longest and, this was true independently of the standard duration used. In fact, let's consider the standard intervals used for Group 1 and Group 2 (**Table 1**). Both groups had 700 and 1000 ms as standards, but in the case of Group 1, 700 ms was the middle and 1000 ms was the longest standard interval, and for Group 2, 700 ms was the shortest and 1000 ms was the middle standard interval. In the case of Group 1, no effect of standard-comparison was observed for the 700 ms standard interval (middle standard) and when the standard was 1000 ms (longest standard), participants had the tendency of responding "long." But in the case of Group 2, participants had a tendency in responding "short" when 700 ms was presented (shortest standard) and no effect of standard-comparison was observed for 1000 ms (middle standard). A different pattern of performance was observed in Group 3, in which participants had the tendency of responding long, and this pattern occurred for the middle and the longest standard durations (Group 3 middle = 1300 ms and longest = 1600 ms).

Although it is difficult to totally discard the possibility that the temporal performances observed in Groups 1 and 2 are influenced by a perceptual phenomenon like the TOE (Hellström, 1977, 1978), a cognitive interpretation is viable. Over several trials, participants might have created anchor duration, i.e., a memory representation issued from the averaging of the shorter and the longer temporal intervals presented. It is posited that this anchor duration may exert influence (some weight) on the discrimination process at the moment of task (for a given single trial). Stimuli are partly classified as a function of the "anchor," with stimuli below the "anchor" tending to be assigned as short and stimuli above the "anchor" as long (see also Oshio et al., 2006).

However, this anchor hypothesis does not explain the performance observed in Group 3. In fact, participants in Group 3 generally responded long independently of the duration of the standard interval presented. Such a result is compatible with the hypothesis stipulating that there are distinct systems for processing duration above or below 1-s, one for longer temporal intervals (in the range of seconds) and one for short temporal intervals (in the range of milliseconds). It is as if the amount of information to be processed exceeds the capacity of the system, just like there is a limited capacity of processing in working memory (Miller, 1956; Cowan, 2001), a limitation that could be compensated by re-organization of information processing with the creation of chunks of information. Actually, a spontaneous way for re-organizing temporal information (too long intervals) is to use strategies such as explicit counting or tapping (Grondin et al., 1999, 2004). In sum, the results of Experiment 1 suggest that, for duration between 400 and 1300 ms, temporal performance is influenced by the context, but for standard durations longer than 1300 ms, other processes seem to be involved and they would attenuate or erase the anchor effect reported with briefer intervals.

Experiment 2 showed that temporal performance is not only related to the range of the temporal intervals under investigation, but also to the experimental procedure used. In fact, without a randomization of trials, there is no anchor effect for short temporal intervals (Group 1 and Group 2). Interestingly, participants in Group 3 rather showed the same pattern of performance (preference in responding long) in Experiments 1 and 2. Moreover, for Group 3 in these two experiments, the accuracy level was about the same. This fact provides additional support to the idea that a different temporal information system would contribute to the processing of longer temporal intervals.

### **TIME-SPACE COMPATIBILITY**

In the present study, we also tested whether time-space compatibility may influence a duration discrimination performance. A recent line of research explains temporal performance from a time-space compatibility perspective (Ishihara et al., 2008; Bonato et al., 2012). These studies suggest that humans do not process time and space separately, but represent time as space. Time flows using a spatial organization or a "mental time line." In the present study, time-space compatibility would be expressed by an association between temporal duration and the spatial position of the response keys on the keyboard: specifically, short temporal durations would be associated with left space, and long temporal durations with right space. Such an association should lead to shorter reaction times and higher accuracies in the congruent (i.e., short-left) than in the incongruent (i.e., short-right) condition (Vallesi et al., 2008).

Results from our study tend to show at first sight that participants' performance could be modulated by time-space compatibility. In fact, in the case of Experiment 1, participants were less accurate when response keys were placed in the incongruent position (long-left and short-right), but only when the standard duration was the shortest. No effect of position of response keys on time discrimination performance was observed when the standard was the longest.

However, in the case of Experiment 2, there was no sign of a time-space compatibility effect on time discrimination performances. Participants were generally more accurate when discriminating long standard intervals, with no effect of position of the response keys. Results from Experiments 1 and 2 indicate the importance of comparing duration intervals with each other. In brief, the potential influence of space compatibility in time discrimination was only observed in Experiment 1, i.e., when the standard duration varied within blocks. Therefore, it is the implicit comparison of temporal intervals and not the duration of the interval itself that seems to be the key factor underlying the response bias (time-space compatibility) (Vicario, 2011).

Interestingly, in Experiment 3, in which no manual response was involved, participants showed exactly the same pattern of performances as the one observed in Experiment 1. In fact, participants had a tendency to respond "short" when the shortest standard duration was presented and "long" when the longest standard duration was presented. Participants' performance was not modulated by assignment of response keys, thus suggesting that processing of duration in time discrimination did not involve necessarily spatial representation. In other words, considering that the effect obtained in Experiment 1 (1) disappeared in Experiment 2 without the randomization of standard durations within blocks, but (2) occurred in Experiment 3 with this randomization but without the assignment of responses keys, what could have look like a time-space compatibility effect in Experiment 1 is indeed due to another factor, namely, varying standard durations within blocks.

It is difficult to compare the present findings with previous results, or to evaluate directly the impact of our findings on the material available in the literature. Previous studies used auditory stimuli and different experimental methods [see (Bonato et al., 2012) for a review]. It remains possible that a time-space effect is involved in other temporal performances, but this might be caused by specific methodological and/or strategic conditions rather than by a general and stable cognitive effect.

## **CONCLUSION**

In conclusion, we have found a complex interaction between context, time-space compatibility and temporal range under investigation. Our results suggest that context influences time discrimination performances only when the temporal range under investigation is below 1300 ms and the temporal intervals vary within blocks. In the case of temporal intervals longer than 1300 ms, participants presented a tendency to respond "long" independently of the method used to present the standard temporal intervals (intervals varying within blocks or between blocks). Overall, these findings indicate that distinct temporal processes might be at play above and below 1300 ms.

### **REFERENCES**


process of humans. *J. Exp. Psychol. Hum. Percept. Perform.* 25, 993–1004. doi: 10.1037/0096-1523.25.4.993


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 23 May 2014; accepted: 21 June 2014; published online: 08 July 2014. Citation: Mioni G, Stablum F and Grondin S (2014) Interval discrimination across different duration ranges with a look at spatial compatibility and context effects. Front. Psychol. 5:717. doi: 10.3389/fpsyg.2014.00717*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Mioni, Stablum and Grondin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Why studying intermodal duration discrimination matters

## *Simon Grondin\**

École de psychologie, Université Laval, Québec, QC, Canada \*Correspondence: simon.grondin@psy.ulaval.ca

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

*Reviewed by:* Richard A. Block, Montana State University, USA

**Keywords: time perception, duration discrimination, psychological time, sensory modalities, attention**

A critical issue in the field of time perception is whether or not explicit judgments about time are processed by some internal clock mechanism. A subsequent issue is whether or not this clock, if any, is central (i.e., is the same for a large range of durations, for whatever way of marking the intervals to be processed). There are several ways of marking time, including the use of signals delivered from different sensory modalities. In other words, do we have sensory specific representations of time, or is there an amodal—central—mechanism (Bueti, 2011)? This fundamental question is addressed here with an emphasis on the discrimination of brief empty time intervals. More specifically, intermodal intervals are of interest, an intermodal interval being marked by two brief and successive stimuli delivered from different sensory modalities.

The interest for the effect of modalities on perceived duration and sensitivity to time has grown recently. Researchers have reported that intervals marked by auditory signals are perceived as longer than time intervals marked by visual signals (Walker and Scott, 1981; Wearden et al., 1998; Penney et al., 2000; see Grondin, 2003), but this issue received recent attention in a context where auditory and visual signals marking time could be presented simultaneously (Gamache and Grondin, 2010; Hartcher-O'Brien et al., 2014). The relative duration of intermodal intervals also received attention. For instance, intervals marked by an audio-visual sequence are perceived as longer than intervals marked by a visuo-auditory sequence (Grondin and Rousseau, 1991; Grondin et al., 1996). Moreover, some intermodal experiments emphasized the role of markers' length on perceived duration (Grondin et al., 2005; Kuroda et al., 2014), with both the lengthening of the first and second marker resulting in longer perceived duration (Grondin et al., 1996).

Recently, Mayer et al. (2014) conducted an investigation involving intermodal intervals lasting from 100 to 900 ms, with combinations of auditory, visual and tactile (A, V, T) stimuli. They observed that when a sound serves as the first marker, in either an AV or AT sequence, duration is perceived as longer than in conditions where a sound serves as the second marker, as in a VA or TA sequence; but reported no ordering effect when tactile and visual signals were used together (TV vs. VT). Mayer et al. interpreted their results in terms of sensory latency (see also Grondin, 1993; Grondin et al., 1996), arguing that the summative distortion pattern they observed (by opposition to a multiplicative effect) is consistent with the hypothesis that there exists a central timekeeping mechanism, common for the processing of any intervals, independently of their markers' modality (see also Hartcher-O'Brien et al., 2014, for a similar conclusion). However, when intermodal and intramodal intervals are randomized from trial to trial, the overall interpretation in terms of sensory latency is more disputable (Grondin and Rousseau, 1991). For instance, for the discrimination of intervals circa 250 ms, Grondin and Rousseau (see their Table 6) reported a condition where the second marker of an interval was tactile, and the first was T, V, or A. They reported that an AT interval was perceived as much longer than TT and VT intervals. This could have been interpreted as if the A signal was detected more rapidly when serving as the first marker. However, when the second marker is always visual and the first one A, V, or T, it is not the AV intervals that are perceived as the longest, but the TV ones. In other words, an explanation based exclusively on latencies finds serious limitations when both intra- and intermodal intervals are compared.

Even more critical from a theoretical perspective is the question relative to the discrimination levels (sensitivity) reached with intermodal conditions. The recent data reported by Mayer et al. (2014) are also interesting as they describe the discrimination levels. In the VA and AV conditions, the Weber fractions are roughly the same, and vary from 30% at 0.1 s to slightly above 20% at 0.9 s. The results are essentially the same when auditory and tactile stimuli combinations are used, with the exception that performances are generally better when the auditory marker is presented first, especially at 0.1 s (above 40% in TA). With visual and tactile signals, the Weber fraction varies roughly between 25 and 31%, with the discrimination being usually better when the visual signal is presented first, especially at 0.1. For the discrimination of intervals lasting about 250 and 1000 ms, Rousseau et al. (1983) reported about the same performance levels in the VA and AV conditions. Also, for intervals lasting 1000 ms, Grondin (2003) reported about the same discrimination levels in AT and TA conditions, and in TV and VT conditions; at 250 ms, performance were slightly better in AT than in TA, and were slightly better in VT than in TV.

The stability of the Weber fraction over time reported by Mayer et al. (2014) is a bit surprising considering the data reported by Grondin (1996) for intermodal intervals lasting 0.125, 0.25, 0.5, 1, 2, or 4 s. In this study, AV and VA intervals were used. There were 36 sessions (3 per experimental condition) lasting about 30 (at 0.125 s) to 65 min (at 4 s). As reported in **Figure 1**, the performances in both conditions were roughly the same and, most importantly, the pattern over time was the same: there is an important and monotonic decrease of the WF from 0.1 s (much higher than 30%) to 1 s (circa 10%). Indeed, it is well-known that the WF is higher with briefer intervals, a fact that is accounted for by the generalized form of Weber's law (Grondin, 2001). In this experiment by Grondin (1996), explicit counting was not refrained, which should explain the low Weber fractions with longer intervals.

Indeed, the real interest is not that much in the intermodal comparisons *per se*, but in the comparison of intra- and intermodal intervals. It is well-established that the discrimination of time intervals is much easier with auditory than with visual markers (Grondin et al., 2001, 2008; Grondin, 2005). If this auditory vs. visual difference is due to the sensory noise associated with the signals marking an interval, marking an empty interval with one auditory signal and one visual signal (AV or VA) should lead to performance levels in-between the ones involving two auditory (AA) and two visual (VV) signals. In Grondin (1993), the Weber fraction for the same range of durations tended to increase with briefer intervals (0.125 s)—especially when using visual signals—varying between 4 and 8% in AA, and circa 12% in VV. An interpretation in terms of variability (or latencies) belonging to the signals themselves would predict a performance level (WF) between 8 and 12%. This result is far from the Weber fraction above 30% reported by Grondin (1996) for intermodal intervals with the same method.

This intra- vs. intermodal difference challenges another hypothesis. Using transcranial magnetic stimulations (TMS) over the primary auditory cortex, Kanai et al. (2011) observed that time discrimination is impaired not only when auditory signals mark time, but also when visual signals do. However, only the performance in the visual condition is impaired when TMS is used over the primary visual cortex. This finding suggests that in timing tasks, the auditory cortex has a supramodal role: the lower performance level in vision than in audition would be due to the need to transfer the visual signals into an auditory code (Kanai et al., 2011). If such is the case though, having one auditory signal (AV or VA conditions) instead of none (VV) should lead once again to performance levels in-between the ones involving two auditory and two visual signals. However, clearly, for very brief intervals (<1 s) and when A and V signals are used, discrimination is severely impaired in AV and VA conditions compared with AA and VV intramodal conditions (Rousseau et al., 1983; Grondin and Rousseau, 1991; Grondin et al., 2005).

Note however that for intervals lasting 1.6 s, the large difference between the threshold value in AA and AV conditions is washed out when an explicit count of numbers is used for completing the task (Grondin et al., 2004). This reduction could certainly be attributed to the efficiency of using sub-intervals (smaller chunks of information), assuming that the counting process remains error free. However, the hypothesis that the efficiency of counting is actually due to the translation of visual signals into an auditory code cannot be discarded.

Recent EEG data, and more specifically the amplitude of the contingent negative variations recorded at fronto-central electrodes, revealed a basic difference between the AA condition and other modality conditions (Gontier et al., 2013; Hasuo et al., 2014). There seems to be something specific to auditory time perception. Moreover, an attentional component would also be at the heart of the intra- vs. intermodality differences (Gontier et al., 2013).

In brief, the different perceived durations and discrimination levels observed in the different intra- and intermodal conditions is a challenge for the single-clock hypothesis. An interpretation based only on sensory latencies (Mayer et al., 2014) would not be sufficient to account for the intra- vs. intermodal difference. Indeed, it would be difficult to explain the variance observed in all intra- and intermodal conditions, for both perceived duration and discrimination levels (Grondin, 1998), on the basis of the variance due to the clock process, and to the addition of non-temporal noise (sensory latencies or attention switching). However, before concluding that there is some modalityspecific temporal processing instead of a central clock, it remains necessary to understand the real impact of all the possible interactions amongst the sources of non-temporal noise (how a stimulus in

one modality impacts the detection of the attention to a stimulus to be delivered in other modalities, what is the role of prior entry in intramodal conditions, ... ). Another avenue is the possibility to have hierarchical model involving a level with modality-specific temporal processing and modality-independent processing system at another level (Stauffer et al., 2012).

### **ACKNOWLEDGMENTS**

The author's work is supported by a research grant awarded by the Natural Sciences and Engineering Council of Canada. Thank you to Rick Block for his comment on a previous draft of this article.

### **REFERENCES**


*Psychol.* 149, 134–141. doi: 10.1016/j.actpsy.2013. 11.013


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 10 May 2014; accepted: 02 June 2014; published online: 01 July 2014.*

*Citation: Grondin S (2014) Why studying intermodal duration discrimination matters. Front. Psychol. 5:628. doi: 10.3389/fpsyg.2014.00628*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Grondin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## It's time to take the psychology of biological time into account: speed of driving affects a trip's subjective duration

## *Hedderik van Rijn\**

Experimental Psychology, University of Groningen, Groningen, Netherlands \*Correspondence: hedderik@van-rijn.org

### *Edited by:*

Claire Zedelius, University of California Santa Barbara, USA

### *Reviewed by:*

Hugo Merchant, Universidad Nacional Autónoma de México, Mexico Wilbert Zarco, The Rockefeller University, USA

**Keywords: interval timing, real-world tasks, perceived speed, driving, time perception, temporal cognition, time dilation, subjective time**

The last decades have seen a surge in research into interval timing (for recent reviews, see Merchant et al., 2013; Wittmann, 2013; Allman et al., 2014; van Rijn et al., 2014), with some work focussing on the more abstract mechanisms underlying interval timing (e.g., Taatgen et al., 2007) or the role of cognitive faculties such as memory and decision processes on interval timing tasks (e.g., Taatgen and van Rijn, 2011; Shi et al., 2013), but a large proportion of the work focuses the neural substrates of human (e.g., Kononowicz and van Rijn, 2011; Wiener et al., 2012; Kononowicz and Van Rijn, 2014) and animal (e.g., Díaz-Mataix et al., 2013; Bartolo et al., 2014; Cheng et al., 2014) timing processes. Based on this work, we are getting closer to unraveling the biological mechanisms underlying interval timing.

Interestingly, although an accurate sense of the passing of time at short timeframes (e.g., less than a couple of seconds) is an important building block in many cognitive tasks, many papers including some of our own—still use rather hackneyed examples such as the timing involved in deciding whether to brake when a traffic light turn yellow to stress the importance of interval timing in every days tasks. Although timing is obviously involved in such real world tasks, the complexities of these tasks are far removed from the simple paradigms using in interval timing studies. For one, in many real-world timing tasks, the temporal stimulus has a direct relevance for the person doing the timing, whereas in most laboratory experiments the participant is asked to time an external stimulus a distinction which can be compared to a first vs. a third-person perspective on time. These discrepancies make it difficult to generalize from the highly specific experiments in the lab to the psychology of timing as observed in the real world (see also Matthews and Meck, 2014), with a possible exception for studies on target interception (for a review, see Merchant and Georgopoulos, 2006). Although literature does list a number of papers in which interval timing aspects are being studied in the real world (e.g., Ten Bosch et al., 2005; Miller and Fu, 2007), those studies are often not focused on the mechanisms underlying timing, or require large databases with naturalistic data. To ensure that interval timing does not follow the path of some other fields of science where no one apart from the researchers active in that field remember why a particular phenomenon was interesting enough to study—we should study interval timing not just in artificial tasks that are specifically created to test a particular phenomenon, but also in tasks that have a clear analog to complex, real-life interval timing tasks.

To demonstrate the viability of this approach, below I will discuss a simple experiment based on a prototypical interval-timing paradigm (e.g., Kononowicz and Van Rijn, 2014) set in the context of the evaluation of the speed of a car from a first-person perspective (e.g., as driver or co-driver).

Theories of human time perception typically assume a clock that provides temporal information to decision processes (van Rijn et al., 2011). Although the exact formulation of this clock is still subject of discussion (see for a review, van Rijn et al., 2014), most theories assume that the information emitted by this clock is relatively stable over time. However, both endogenous (e.g., neurochemical fluctuations, Coull et al., 2011) and exogenous manipulations (e.g., contextual changes, van Rijn and Taatgen, 2008; Lui et al., 2011; or manipulations of expectancy, e.g., Tse et al., 2004; see Grondin, 2001, for an extensive review) affect interval timing. For example, if the display duration of a moving stimulus has to be estimated, a positive correlation is found between speed and the estimated display duration (e.g., Brown, 1995; Kline and Reed, 2012, see also Roelofs and Zeeman, 1951–1952), if a stimulus is perceived to move toward an observer it is perceived as having a longer duration than when the same stimulus is presented as a static image or is perceived to be moving away (e.g., van Wassenhove et al., 2008; New and Scholl, 2009; Wittmann et al., 2010), or if the environment in which a temporal stimulus is presented moves faster, the duration of the temporal stimulus is overestimated (e.g., Mate et al., 2009) compared to static or slower moving environments. The interpretation of types of studies into the subjective dilation of time can be roughly summarized as faster movement, or movement toward rather than moving away from the observer, yielding a faster ticking internal clock, resulting in the subjective duration of the stimulus lengthening. These effects are most likely driven by early visual processes that detect the (number of) changes in a display (Droit-Volet and Wearden, 2002), possibly as early as the primary visual cortex (Kanai et al., 2006). Typically, these types of studies ask participants to estimate the duration of stimuli that move on the screen, demonstrating that the subjective perception of time can be affected by perceiving third-person movement. Here we address the question whether simulated first-person perspective movement also affects the estimation of time. If these effects generalize to first-person perspective, this might have direct consequences for naturalistic, real-life settings such as the subjective evaluation of driving speed and speed limits. That is, faster speeds result in faster movement of the scenery, which might result in a lengthening of subjective time. To offset these effects, drivers might be tempted to drive faster, giving rise to more speed violations and a potential positive correlation between absolute speed limit and dissatisfaction with these limits—demonstrating that modulations of interval timing processes can have significant real world implications.

The study was designed to test whether time dilation effects due to moving stimuli generalized to first-person perspective, and what the consequences are on the perceived duration of (short) drives. Using the driving simulator Distract-R (http:// cog.cs.drexel.edu/distract-r/; see Salvucci, 2009), we recorded a video clip of several minutes of a car driving at 100 km/h, see Panel A of **Figure 1**. Participants were acquainted with the standard duration of 2.5 s by presenting them five unique 2.5 s segments of the video. After this presentation, participants were trained on reproducing this interval during 30 reproduction trials. Hereto unique segments of the video where started, and participants were asked to indicate when

the just perceived duration had passed by pressing a key, after which feedback (see Kononowicz and van Rijn, 2011, for details) was provided. The experimental phase consisted of 200 temporal generalization trials, in which participants were presented video segments of either 2.3, 2.4, 2.6, or 2.7 s and which they had to categorize as either shorter or longer than the learned duration. Critically, the video segments where either taken from the recorded video, or from video that was slowed down to represent 50 or 75 km/h, or that was sped up to 125 or 150 km/h. Video segments were randomly assigned to speed conditions. **Figure 1B** shows the average proportions of long responses (38 individuals participated, 3 removed for not following instructions, all participants associated with the University of Groningen, ethical approval #12242-NE, Ethical Committee Psychology) separately for each simulated car-speed condition (solid lines). The five lines indicate that participants are sensitive to the duration manipulation, with shorter durations less often categorized as long than longer durations. Moreover, the curves are vertically ordered in line with the depicted speed, indicating that video clips depicting faster speeds were more likely to be categorized as long. A binomial linear mixed effect model (glmer from the lme4 package 1.1–6 in R 3.0.2 using logit as link-function and estimating fixed-effects parameters and random effects in a linear predictor using maximum likelihood), with centered fixed effects for speed (i.e., expressing speed as −50, −25, 25, and 50) and for duration (−0.2, −0.1, 0.1, 0.2) and a random intercept, and separate random slopes for speed and duration per participants, confirmed the effects of duration and speed (βspeed = 0.0187, *SE* = 0.0025, *z* = 7.573 and βduration = 3.5432, *SE* = 0.4569; *z* = 7.754 respectively, *p*s < 0.0001), the interaction between duration and speed was not significant (βspeed <sup>×</sup> duration = 0.0064, *SE* = 0.0052, *z* = 1.251, *p* = 0.21), nor was the intercept of the model (βintercept = −0.0756, *SE* = 0.0779, *z* = −0.971, *p* = 0.33). Indeed, model comparisons indicate that the addition of the interaction was not warranted [χ<sup>2</sup> (1) = 1.55, *p* = 0.2129].

This initial model assumes that there is a linear effect of speed. Because **Figure 1** suggests a nonlinear mapping of speed on probability long responses, we also conducted an analysis with speed as a factorial variable (reference: 100 km/h). Although this increases the complexity of the model, the fit is sufficiently improved to select this more complex model [χ<sup>2</sup> (21) = 50.684, *p* = 0.0003]. As before, the inclusion of the interaction is not warranted [χ<sup>2</sup> (4) = 3.3126, *p* = 0.5069]. The estimated effect size of the intercept does not deviate from zero (β = 0.10391, *SE* = 0.09179, *z* = 1.132, *p* = 0.258), indicating that no change in speed does not significantly affect the subjective perception of time. The estimates for the other four speed conditions are in the expected direction (β−<sup>50</sup> = −1.28323, *SE* = 0.20088; β−<sup>25</sup> = −0.72023, *SE* = 0.11840; β<sup>25</sup> = 0.41170, *SE* = 0.09786; β<sup>50</sup> = 0.53864, *SE* = 0.11022; |z| > 4.2, *p* < 0.0001), and the estimated duration effect is similar to the previous model (β = 3.56750, *SE* = 0.46144, *z* = 7.731, *p* < 0.0001). The estimated psychometric curves resulting from this last model are plotted in **Figure 1B** as dotted lines. Note that these lines are extended beyond the direct measured data based on the estimates of the binomial linear mixed effect model.

The circles drawn around the intersections between the fitted speed-specific psychometric functions and the p(long) = 0.5 line represent the point of subjective equality (PSE), that is, at what point in the duration of the movie clip (would have) felt as long as the standard duration of 2.5 s. Note that as the 50% decrease condition resulted in few long categorizations, the estimated PSE is extrapolated from the estimated psychometric function. On the basis of these data, the inset depicts the dilation as a function the speed of the video. At 100 km/h, participants' estimations are quite accurate, with a dilation of only 1.0%. For the 50% increase in speed condition, the subjective lengthening relative to the baseline condition of 100 km/h is 7.2%, and for the 25% increase in speed 6.2%. For the 25% decrease in speed, the subjective shortening is 6.8%, and for the 50% decrease in speed, the extrapolated estimate is a subjective shortening of 15%. Over the four conditions, the average effect of a 10% increase or decrease of speed on time is a dilation or contraction of 2.4%, indicating that about one fourth of the objective advantage of increased speed limits is canceled out by subjective lengthening of time. These results demonstrate that the effect of the perception of movement on interval timing extends to movement in first-person perspective, and can also be observed in naturalistic, yet well controlled conditions.

These results are in line with the thirdperson perspective studies on the effect of movement on time. That is, the faster the speed of the car in the video, the more context changes were perceived in the video, aligning nicely with the hypothesis that the number of visual changes drives temporal dilation effects. Although tested in a different context, this explanation find corroborative support in a study (Antonson et al., 2009) on the effect of landscapes on preferred speed in car simulators, with landscapes richer in details (i.e., forests compared to open spaces) associated with slower preferred speeds. Although this effect is typically explained by other, higher-level factors, our study suggests that the rich detail landscapes might cause internal time to run faster due to the higher number of changes, causing participants to drive slower to keep their subjective speed at comfortable levels. A similar finding that links the subjective perception of speed with interval timing is reported in Rudin-Brown (2004) who has shown that eye height of a driver affects preferred speed, with drivers seated higher preferring faster speeds caused by the subjectively slower movement of the outside world. As suggested by a reviewer, an elegant test that could provide further links between laboratory tasks and task-settings with higher external validity is to compare conditions with meaningful semantic visual context versus phase-scrambled movies, which would make the current experiment better comparable to laboratory studies in which semantically irrelevant movement is provided. Comparing these results will allow us to assess directly whether the first-person perspective in a meaningful context affects subjective interval timing.

To summarize, this study shows that the temporal dilation effects observed in labstudies on interval timing, including but not limited to the phenomena discussed earlier (Eagleman, 2008), have real world consequences: if a driver is used to driving at 100 km/h, and is suddenly allowed to drive 130 km/h, the dilation of time will result in an internal experience of approximately 123 km/h. Compensating for this subjective discrepancy will cause speeding, whereas adherence to the speed limit will cause the driver to perceive a discrepancy with the enforced limits and his or her internal evaluation of speed. Moreover, this study also shows that generalizing findings from the lab is possible, and that appealing examples can be found that demonstrate the relevance for interval timing in real-world settings.

### **ACKNOWLEDGMENTS**

The author would like to thank Jorn Schotanus and Jeroen van den Born for assistance with data collection and Rogier van Rijn for assistance with creating the stimulus materials.

## **REFERENCES**


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 June 2014; accepted: 28 August 2014; published online: 16 September 2014.*

*Citation: van Rijn H (2014) It's time to take the psychology of biological time into account: speed of driving affects a trip's subjective duration. Front. Psychol. 5:1028. doi: 10.3389/fpsyg.2014.01028*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 van Rijn. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Images of time: temporal aspects of auditory and movement imagination

## *Rebecca S. Schaefer\**

Department for Psychological and Brain Sciences, SAGE Center for the Study of the Mind, University of California, Santa Barbara, Santa Barbara, CA, USA \*Correspondence: rebecca.schaefer@sagecenter.ucsb.edu

### *Edited by:*

Simon Grondin, Université Laval, Canada

### *Reviewed by:*

Tsuyoshi Kuroda, Kyushu University, Japan

**Keywords: auditory imagery, motor imagery, temporal processing, expertise**

Research on mental imagery has shown that when we imagine something, the related neural processes overlap with those related to actually perceiving or performing that same percept or action (Kosslyn et al., 2001). Although visual imagery has long been the dominant modality for the investigation of sensory imagery, involvement of modality-specific brain regions (i.e., visual areas being implicated in visual imagery, and so on) has now also been reported for auditory, olfactory and tactile imagery (Halpern, 2001; Plailly et al., 2012; Schmidt et al., 2014). Movement or action imagery has informed theories of action representation (Jeannerod and Decety, 1995) and has recently gained interest in the context of mental practice for expert skill acquisition such as sports or surgery (Cocks et al., 2014) and movement rehabilitation (Malouin and Richards, 2010). Increasingly, the neural underpinnings of imagery have become clearer, and both modality-specific and modalityunspecific neural activations related to imagery have been found (Daselaar et al., 2010; Zvyagintsev et al., 2013). However, the growing body of neuroimaging literature on imagination has yet to include an account of temporal imagery. Although the reproduction of time intervals has been a research topic of interest, the contribution of temporal imagery—namely internal timekeeping or creating the temporal aspects of imagery in other modalities—is largely unexplored. As such it is unclear whether, similar to other perceptual modalities, imagery for time shares cerebral substrate with brain networks involved in time perception or regularity detection. Specifically for actions or sounds, the temporal structure of the imagined stimulus or action is crucial in conjuring a faithful image, suggesting that in scientific findings of auditory and motor imagery, temporal imagery is included. The temporal patterns we create internally arguably lie at the basis of any self-paced movement, be it a fast sprint, a musical performance, or an easy walk in the park. Here, I argue that the shared components between movement and auditory (specifically music) imagery may offer a window into timing and temporal skills, which may carry cognitive importance beyond movement or auditory functions.

Whereas the reciprocal influences of rhythmic sound and movement have been shown, with rhythmic movement supporting auditory acuity (Su and Pöppel, 2012) and rhythmic sound supporting movement efficiency (Bood et al., 2013), few studies have looked at the extent to which imagery can functionally replace perception or action and show similar interactions, which may point to shared processing in their temporal structure. Music imagination was reported to affect finger tapping similarly to perceived music (Repp, 2001) and has been used to support movement in clinical settings (Satoh and Kuzuhara, 2008). Anecdotal observation from music imagery experiments (such as described in Schaefer et al., 2011b), suggests that tapping a finger along to music imagery supported accurate imagery timing, and auditory cues are reported to increase motor imagery vividness (Heremans et al., 2009). Furthermore, motor timing skills and anticipatory music imagery ability appear to be related (Pecenka and Keller, 2009; Keller and Appel, 2010). These findings suggest that the links between motor and music processing may extend into imagined stimuli or actions, and the contribution of the current piece is to bring together findings from temporal processing, music and motor research, and to assert that firstly, the commonalities between music and motor imagery may reveal more about internally driven timing mechanisms, and secondly, one might speculate that these internal timing patterns are related to expertise, and as such may be seen in the context of findings on temporal skills and cognitive abilities (Madison et al., 2009; Holm et al., 2011).

## **IMAGERY FOR TEMPORAL PATTERNS**

Auditory patterns and movement sequences share a strong dependence on temporal structure. Specifically in music and speech sounds, a fine-grained temporal ordering of acoustic features forms a meaningful percept, and precise perceptual skills are required to decode or extract information from these signals. Similarly, skilled movement sequences are intricately ordered, and often need to be concatenated through extensive practice. When sounds or complex movements are imagined, their temporal structure crucially needs to be integrated in the image, creating an imagined temporal pattern. The direction of time in imagined sounds or movement sequences is non-reversible, although single elements can be imagined in reverse; for instance, you may imagine single movement elements in reverse, or a construct working memory task that makes use of sensory imagery through the mental reversal of melodies, see Zatorre et al. (2009), although this changes the identity of the sound or movement, and likely requires increased imagery ability and domain-specific expertise.

The discourse on the temporal structure of imagery is most developed for the motor domain, where researchers aim to understand and increase the efficacy of the mental practice of skilled movement (for reviews, see Guillot et al., 2012; Smith and Wakefield, 2013). Although the timing of an imagined action will not necessarily convey information about the content or vividness of the imagery, the assumption is that it at least gives an indication of whether the temporal structure of the imagined action is similar to the actual action. Among the factors that can affect the timing of motor imagery are the type, complexity and duration of the imagined action and the age, expertise level and imagery strategy of the subject (Guillot et al., 2012). Although there is still little evidence to suggest that accurately timed mental practice actually improves motor learning, the timing of movement imagination has been shown to impact the timing of subsequent motor performance (Debarnot et al., 2011). Again, changing the speed of a complex movement changes its nature (although much more subtly than when reversing a sequence), and the ability to manipulate imagery tempo may only be available to movement experts. Smith and Wakefield (2013) further argue that as movement rhythms are crucial to success in sports, the rhythm of imagined actions should also receive more consideration, rather than recording only their overall timing. However, good paradigms to assess imagined movement rhythm need to be developed.

The temporal structure of auditory imagery has not received as much specific research attention, but given that rhythmic patterns are integral to music, the imagery of temporal structure is perhaps even more crucial to music imagination. Accordingly, musical experts were reported to show increased temporal acuity for imagery of tonal scales (Janata and Paroo, 2006), and temporal precision of musical imagery enhances music performance, especially in ensemble playing (Keller, 2012). However, even for non-experts, being able to imagine music unfolding in time and being able to track or scan its progression when hearing the music assumes the maintenance of an internal pulse or timekeeper (Povel and Essens, 1985) that dictates the pace of the (internal) music. As for movement, this process can be controlled, since, to a certain extent, we can imagine the same music at a very fast or slow tempo. However, after repeated listening, the tempo of music is stored in long-term memory, which was reported to be surprisingly accurate (Levitin and Cook, 1996). Thus, the timekeeper allows imagination processes to activate representations that unfold accurately over time. However, it is not yet clear how this timekeeper is implemented in the brain.

## **THE BRAIN BASIS OF IMAGINED TEMPORAL PATTERNS**

Commonalities between the brain activity patterns of (auditory) rhythm perception and movement have long been interpreted as related to temporal structure (Ivry, 1996; Schubotz et al., 2000), implicating the cerebellum, striatum, premotor cortex (PMC), and (pre-) supplementary motor cortex (SMA). These areas are also part of a network of areas identified in a meta-analysis of 41 studies of interval timing tasks, where conjunctions for perceptual and motor timing tasks were reported in the bilateral SMA, PMC, parietal areas, bilateral inferior frontal gyrus (IFG), striatum, right insula and bilateral posterior cerebellum for sub-second intervals and in the bilateral SMA, left precentral gyrus, right cingulate gyrus, right IFG and bilateral insula for supra-second intervals (Wiener et al., 2010). Specifically for auditory perceptual timing, distinct patterns were found for duration- and beat-based timing processes (Teki et al., 2011), showing absolute, duration-based timing tasks to mostly activate cerebellar and brainstem areas, and relative, beatbased timing to activate the basal ganglia, PMC, (pre-)SMA and dorsolateral prefrontal cortex. Comparing these results to reports of imagined music and imagined movement reveals considerable anatomical overlap, with temporal processing being a common inference in the interpretation of these activations during imagination. Findings from music imagery provide consistent reports of activation in the secondary auditory cortex, IFG and (pre-)SMA, as well as cerebellar and striatal areas (Halpern and Zatorre, 1999; Leaver et al., 2009; Herholz et al., 2012). A recent meta-analysis of 75 movement imagery studies reported that when combining all types of movement, the areas that are most commonly found to be active during movement imagery include SMA, bilateral precentral gyrus, IFG, parietal areas, striatum, and cerebellum (lobule VI) (Hétu et al., 2013). Collectively, it appears that there is considerable overlap in the brain areas activated by music and motor imagery, and similar brain area networks are implicated in timing tasks. However, the striking apparent similarity between brain networks engaged in motor and perceptual timing, auditory imagination and imagined actions is of course no indication that each of these functions depends on identical neural mechanisms.

The investigation of temporal structure is inherently hampered by the fact that these structures are embedded in actions or percepts, which, as discussed above, are known to interact with imagery. However, this may be overcome by using ambiguous percepts or actions with different imposed temporal structures or interpretations, for instance by imposing different temporal structures on ambiguous stimuli (Fujioka et al., 2010; Schaefer et al., 2011a), or looking at individual differences in beat perception (Grahn and Mcauley, 2009). Using ambiguous stimuli to look at temporal structure, by imposing different temporal structures on identically performed actions, is much harder to achieve for actions. However, experiments looking directly at the common components of music and motor imagery can be easily conceived, for instance by directly comparing the manipulation of tempo in auditory and movement imagery. Thus, there are ways to overcome the complexity of the embedded, implicit nature of temporal patterns in experimental settings.

### **TEMPORAL SKILLS AND EXPERTISE**

Imagery has long been thought of as an integral part of learning and memory, perception and action, information processing and reasoning (Kosslyn et al., 1995). Accurate (or veridical) imagery, be it of movements or music, is typically associated with expertise. Several investigations note increased vividness and/or accuracy for imagery of experts (Ozel et al., 2004; Janata and Paroo, 2006; Pecenka and Keller, 2009), and, in the case of musicians, accurate imagery is considered an integral aspect of music performance (Hargreaves, 2012) and a mark of good musicianship (as evident from historical musical aptitude scales; i.e., Gordon, 1965). Although this is correlational evidence, it may thus be the case that imagery skills are acquired with expertise. While for expert musicians strong temporal skills are clearly needed, the rhythm of movement is also highly important for effective sports performance (MacPherson et al., 2009), and again the implication is that temporal skills can be acquired. Temporal skills are not only important for music and movement, but are reported to be related to intelligence (Madison et al., 2009; Holm et al., 2011), and especially prospective timing may be related to attention and learning (Taatgen et al., 2007). Findings of increased executive functioning after music training (Moreno et al., 2011) could thus be interpreted as being related to increased temporal skills, potentially related to imagery ability. Although it is highly speculative at this point, one could hypothesize that the relation between cognitive control and imagery on one hand, the relation between imagery and temporal skills on the other, and the relation between temporal skills and cognitive ability, all depend on similar or overlapping neural mechanisms. This would have implications for the effects of training temporal skills for imagery ability (which has been identified as an issue in movement rehabilitation settings, cf. Malouin and Richards, 2010), as well as possibilities in cognitive training or rehabilitation.

In sum, I have argued that imagination of music and movement have shared components, which is supported by experimental evidence of their interactions and brain activations, and may be related to generating temporal patterns. Domain-specific expertise appears to be related to both imagery ability and temporal skills, with implications that are based on the known relations between both temporal skills and imagery skills and cognitive functioning. Future work, employing highly specific experimental designs, needs to clarify whether music and movement imagery timing are mediated by similar neural mechanisms, and whether increasing skills in one domain may carry over to increases in the other domain, or to cognitive ability.

### **ACKNOWLEDGMENT**

I thank Corina Logan for comments on an earlier version of this manuscript, and the SAGE Center for the Study of the Mind at the University of California at Santa Barbara for supporting this work.

### **REFERENCES**


tunes. *J. Cogn. Neurosci.* 24, 1382–1397. doi: 10.1162/jocn\_a\_00216


in brain connectivity support perceptual grounding of mental images in primary sensory cortices. *Neuroimage* 98, 216–224. doi: 10.1016/j. neuroimage.2014.05.014


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 01 July 2014; accepted: 23 July 2014; published online: 12 August 2014.*

*Citation: Schaefer RS (2014) Images of time: temporal aspects of auditory and movement imagination. Front. Psychol. 5:877. doi: 10.3389/fpsyg.2014.00877*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Schaefer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Atemporal equilibria: pro- and retroactive coding in the dynamics of cognitive microstructures

## *Mark A. Elliott\**

School of Psychology, National University of Ireland Galway, Galway, Ireland

### *Edited by:*

James M. Broadway, University of California, Santa Barbara, USA

### *Reviewed by:*

J. Scott Jordan, Illinois State University, USA James M. Broadway, University of California, Santa Barbara, USA

### *\*Correspondence:*

Mark A. Elliott, School of Psychology, National University of Ireland Galway, Galway, Ireland e-mail: mark.elliott@nuigalway.ie

Synchronization of spatially distributed neural assemblies at frequencies in the range 30–70 Hz (the "gamma" band) may be instrumental in grouping stimulus features. In agreement with this we have shown that detection reaction times to a grouping target stimulus are expedited when the stimulus is preceded by repeated presentation of a priming stimulus, presented below detection thresholds in a matrix that flickers at particular frequencies in the 27–68 Hz range. This dynamic priming effect can be partly explained as a function of the return phase of the priming stimulus relative to the premask matrix, indicating one of the primary consequences of repeating stimulation is pre-activation of a priming response relative to prime-stimulus presentation. However, this cannot entirely explain the relationship that develops between the timing of stimulus events (in this instance the time of target relative to priming-stimulus presentations) and response. By varying the frequency and phase of priming-stimulus and target presentations we discovered that given a particular relationship between the phase of target presentation relative to the return phase of the prime, target coding is expedited by a prime that achieves its maximum activation at a phase that would precede priming-stimulus presentation by several tens of milliseconds. However, and in addition, the cognition concerned is flexible enough to be able to achieve an identical prime retroactively, that is to say at a phase during or subsequent to priming-stimulus presentation. This occurs because of a different relationship between the phase of target presentation (defined relative to prime frequency) and the frequency of premask-matrix presentation. On this basis, it can be concluded that by virtue of the relationship between its dynamics and the timing of stimulus events, microstructural cognition functions in a temporal context that can shift from past to future states. Consequently and at the lowest level of psychological function, the conventional, one-dimensional model of time flow—from future to past states does not fully explain how cognition can function. In fact depending upon the interaction in phase between different coding frequencies, the same form of cognition can anticipate or retroactively code events. Consequently, and in so far as our cognition at this level provides a content structure for consciousness, our psychological lives may be fundamentally based upon the ability of our cognitive states to travel backwards and forwards across very short intervals of time.

**Keywords: oscillatory synchronization, gamma band, pro- and retroactive microcognition, protention, visual-event coding**

## **INTRODUCTION**

The rhythmic synchronization of neural activity at gamma-band frequencies (30–70 Hz) is believed to be related to the organization of visual events, in particular the binding of individual visual features to form perceptual wholes (Gray, 1999; Singer, 1999). The precise mechanisms concerned in bringing this about are a matter of discussion, one that has from time to time concerned itself with the relationship between the timing of neural and stimulus or other non-proximal events. For instance, recent focus has involved discussion of the relationship between small eye movements (microsaccades) and induced cortical gamma activity, suggesting that recordings of cortical gamma-band activity are essentially related to the frequency of muscular movements in the eye (Yuval-Greenberg et al., 2008, 2009; Bosman et al., 2009; Melloni et al., 2009a,b; see also Hassler et al., 2011 for contradictory evidence). An older discussion has identified a link between stimulus-evoked gamma activity and visual grouping, suggesting that at least the onset of the gamma response is time locked to a stimulus event (Herrmann et al., 1999; Herrmann and Bosch, 2001; Herrmann and Mecklinger, 2001, see Tallon-Baudry, 2009; Martinovic and Busch, 2011 for reviews). This is a controversial theory as other EEG, as well as the physiological literature usually reports oscillatory-gamma activity to be unrelated to the phase of stimulus events (see Tallon-Baudry and Bertrand, 1999 as well as Pantev, 1995; Fries et al., 2007, for reviews).

There are also a number of psychophysical studies of the relationships between stimulus timing and the synchronized appearance of stimulus elements, feature binding and visual grouping. Of these, and taking into account critiques that challenge the necessity and sufficiency of some synchronization paradigms (Farid, 2002; Elliott et al., 2006b), there are a class of paradigms that have employed stimulus synchronizations presented below detection thresholds which prime or bring about the Gestalt organization of the synchronized stimulus elements (Elliott and Müller, 1998; Usher and Donnelly, 1998). These studies acknowledge a now large body of evidence indicating that elements of a visual scene are often bound ahead of attentional deployment and are very unlikely to be coded by the same mechanisms as those mediating direct conscious experience of the Gestalt (e.g., Duncan and Humphreys, 1989; Rensink and Enns, 1995; Driver et al., 2001). Usher and Donnelly showed that synchronization of orientation at a frequency in the gamma band significantly biases subsequent orientation judgments even though observers cannot reliably report the synchronized orientation. In Elliott and Müller's paradigm (**Figure 1**), embedding a figurally-relevant grouping as one phase of a multiphase premask that flickered at 40 Hz, leads to faster detection reaction times (RTs) to a subsequently presented target grouping, without prior attentional deployment to the location of the grouping in the premask. Observers could not detect the presence of the premask grouping and because presentation of this

not presented, an absent response. In **(B)** are shown example sequences of four image frames that comprise the premask matrix. In the intraphase conditions (upper panels) one premask frame consists of four crosses in square arrangement; see the upper far left panel. In the interphase condition (lower panels) these four crosses are distributed across frames. stimulus does not cue target presentation, it is referred to as a prime.

What is surprising about the results reported by Elliott and Müller (1998) is the specificity of the priming effects to 40 Hz. Elliott and Müller (2004) subsequently discussed a set of experiments in which premask matrices were presented at frequencies in single Hertz steps over the range 30–50 Hz. These experiments extended upon the original finding reported by Elliott and Müller (1998) in that they showed priming effects not to be confined to 40 Hz. Instead, priming was found when premask matrices flickered at 33 Hz, 39–40 Hz and at 46–47 Hz. Described in terms of a "Generalized Phase Angle Hypothesis" (GPAH), Elliott and Müller (2004) observed that priming occurs for primes presented within premask rhythms that would all be in phase alignment at regular 148 ms intervals, implicating modulation of the premask-presentation rhythm by a slower (EEG theta) rhythm of approximately 6.75 Hz with which they would share a common phase angle (see **Figure 2**). The GPAH predicts that, for a given priming frequency *f* and corresponding period duration τ = 1/*f* , facilitation reoccurs at every time point.

$$\mathbf{J}(\mathfrak{r}) = (n\mathfrak{r} + \mathbb{W})\bullet\mathfrak{r} - T$$

where *n*τ is a frequency-specific integer multiplier and *T* denotes a constant quantal time delay. The term +½ accounts for the observation that, for *f* = 40 Hz, maximal facilitation occurs at phase angles of 180◦ relative to the rhythm of premask-matrix presentation (Elliott and Müller, 2000).

Prime generation given regularly ordered stimulus frequencies in phase with a slower, presumably endogenous rhythm is only one part of the story. An analysis of the same dataset, published earlier by Kompass and Elliott (2001) found that priming varied in magnitude (or priming was or was-not present) for frequencies according to the time of target presentation expressed in terms of the phase of the premask-matrix presentation frequency (referred to in terms of a "Return Phase Hypothesis" or RPH for oscillatory priming). In fact priming was maximal for targets presented at a time ahead of the premask-matrix presentation phase at which the priming-stimulus would have been presented if premask-matrix presentation had continued. This indicates the prime to be a cognitive response that can develop *in advance* of

**the priming frequencies (33, 39–40, and 46–47 Hz—black functions) align phases at 148 ms and are thus also phase aligned with a stimulus-evoked rhythm of 6.75 Hz (red function).**

the priming stimulus, most likely as a function of the rhythmic

nature of premask-matrix presentation. The picture thus far developed identifies a form of preactivation of neurons coding the priming stimulus as a function of an interaction between one or more phases of the premaskpresentation rhythm and an inferred, but not directly observed rhythm of around 6.75 Hz. This picture is however complicated by the absence of a corresponding EEG response matching the 40-Hz rhythm of premask presentation (Elliott et al., 2000), but the presence nevertheless of a 33–34 Hz response (a rhythm predicted by the GPAH to encourage prime formation, see Elliott et al., 2003). Given that prime formation is preattentive and based upon a stimulus that is not detected (Elliott and Müller, 1998; Shi and Elliott, 2007), while it is not possible to refer to prime generation as "protentive" after Husserl's (1928) definition (which directly concerns an "experienced" future state), it seems that the prime can, under some circumstances, represent the temporal advancement (or pre-activation) of visual cognition. The question explored in this paper is what those circumstances are with the expectation that they are describable in terms of patterns of interaction in the dynamic systems coding the prime.

The work presented here describes the results of 7 experiments in Study 1 followed by 2 subsequent experiments in Study 2. The experiments presented in Study 1 define a very precise temporal relationship between stimulus frequency and phase referenced to a slower endogenous rhythm of 6.69 Hz. It is this relationship that results in maximum prime activation ahead of priming-stimulus presentation. In other words, when coupled with the slow 6.69-Hz rhythm, the frequency of premask-frame presentations preactivates a prime relative to priming-stimulus presentation.

Study 2 extends upon these results and show, by varying premask-matrix presentation time, that primes not only predate but can achieve maxima in synchrony with the presentation phase of the priming stimulus (Experiments 8 and 9). They may also slightly lag the priming-stimulus phase (Experiment 9). These experiments show that prime maxima that predate primingstimulus presentation rely upon an interaction of slow theta with the gamma-band premask-matrix presentation rhythms—as predicted by the GPAH. Conversely phase aligned and retroactive priming is not based upon this interaction. These may also occur for any frequency depending upon their return phase—as predicted by the RPH. Taken together, these experiments show the temporal relations required for prime pre-activation as well as for retroactive priming, allowing us to discuss the role dynamic microcognition plays in defining a cognitive equilibrium (a state in which the immediate outcome of cognition is stable and successful) that extends across a small interval spanning future and past time.

### **STUDY 1 MATERIALS AND METHODS PARTICIPANTS**

Participant data are given in **Table 1**. All participants in all experiments had normal or corrected-to-normal vision. They performed one block of practice trials immediately before the experiment proper and were naive as to the precise experimental conditions presented in the experiment. Participants were paid at a rate of 10.00 DM (deutschemark) per hour.

All participants provided informed consent to participate in the experiments, with protocols approved by departmental research ethics committees convened at the University of Leipzig and the Ludwig-Maximilians University, Munich.

### **APPARATUS AND STIMULI**

Stimulus image frame generation, event timing, and data collection were controlled by an IBM compatible PC, which also controlled oscilloscopic image presentation through an Interactive Electronics Systems point plotter buffer with 8 Mb frame store memory (Finley, 1985). Image frames were presented on a 6 Tektronix 608-oscilloscope monitor equipped with a very fastdecay P15 phosphor. The use of a P15 phosphor ensured that on-screen image persistence reduced to 10% of normal image


#E refers to experiment number; #N to number of participants; Mean age and (SD) standard deviation in years; f to the premask-matrix presentation frequency range (in Hertz); Steps to the resolution (in Hertz) between steps; #Trials to the number of trials per participant; #Blocks to the number of blocks per session; #Sessions to the number of experimental sessions. E#2a–c and also E#4a–c were run using the same participants, within experiments, and with counterbalanced experimental order.

intensity within 2.8μs (microseconds) of image termination (Bell, 1970). The Interactive Electronic Systems point plotter buffer allowed pixels to be plotted at a rate of one pixel every microsecond.

The stimulus display (illustrated in **Figure 1**) consisted of a matrix of premask elements distributed across 4 sequentially presented premask-matrix image frames. In Experiments 1–7, this sequence of image frames was presented rapidly and repeatedly for 600 ms, after which the premask matrix transformed into a static target matrix of corner junctions. Under target conditions a subset of these junctions would group in one display region to form an illusory target square (illustrated in **Figure 1A** right). Participants viewed these displays at a distance of 57 cm, maintained via a chin rest. Experiments were conducted in a dimly lit room (mean screen surround luminance 0.078 cd/m2), with stimulus luminance maintained at 0.3 cd/m²upon a background field of 0.075 cd/m2. The 3 <sup>×</sup> 3 premask matrix (**Figure 1A** left), subtended 7◦51 × 7◦51 of visual angle at the center of the screen. Premask elements were crosses of size 51 and were separated from their nearest horizontal and vertical neighbors by 2◦39 . Premask crosses consisted of 19 tightly spaced points, so that their segments appeared as uninterrupted lines. Premask image frames could consist of 1, 2, 3, or 4 crosses presented simultaneously (**Figure 1B**), so that the number of pixels presented in a given frame were 21, 42, 63, or 84. In order to ensure all frames were equiluminant, an additional 979, 958, 937, or 916 pixels, respectively, were plotted to an invisible corner of the display (with X,Y coordinates 0,0), equalizing the number of pixels plotted in a single frame. Junction elements in the target matrix (**Figure 1A** right) subtended 26 of visual angle and were separated horizontally and vertically by between 2◦39 and 3◦30 . Each target junction consisted of 11 tightly spaced points, and the target matrix overall consisted of 99 pixels. According to an identical procedure to that used for the premask matrices, an additional 901 pixels were plotted to an invisible corner of the plotter screen. The size and separation of the inducer elements produced a Kanizsa-type square, which, according to recent formulations (Shipley and Kelman, 1992), represents a "good square" with a probability of less than 0.1.

### **DESIGN AND PROCEDURE**

Experiments 1–7 were conducted to examine the relationship between priming and the frequency of premask-matrix frame presentations. Accordingly, premask-matrix frequencies were varied and the ranges of frequencies for each experiment are given in **Table 1**. In **Table 1** experiments are listed in running order and followed the logic that a range of frequencies was examined for frequencies of relevance to priming (Experiments 1–3) followed by experiments with higher frequency resolution that provided a more specific estimate of the frequency bandwidths at which primes are generated (Experiments 4–7). Experiments 2a–c examined an additional hypothesis that priming occurs at particular frequencies, but that these may differ across participants. Accordingly these experiments examined the frequency ranges 28–35 Hz, 36–43 Hz, and 44–51 Hz with counterbalanced experimental order. No clear evidence was found to support this idea. Experiments 4a–c also ran with the same participants in counterbalanced order.

Premask-presentation frequency was defined in terms of a constant presentation time for each of the 4 individual premaskimage frames. For example, for a premask presentation frequency of 40 Hz the premask-image frames were presented at a rate of 10 repeats per second, which given a constant exposure duration of 25 ms (ms) and an inter-frame interval of less than 1 ms resulted in a matrix frequency of 40 Hz. As illustrated in **Figure 1B**, premask-matrix presentation was divided into two critical conditions: in the first and on 50% of trials, the premask elements were pseudo-randomly (or "interphasically") distributed across all 4 image frames with 4 elements appearing in frame 1 controlled to avoid the possibility of accidental, figurally relevant spatial organizations arising within this image frame. This is illustrated in the lower panel of **Figure 1B**. On the remaining trials, premask-matrix frame 1 included the synchronous (or "intraphase") presentation of 4 premask elements in square arrangement at the precise matrix locations which could (on 50% of trials) be occupied by the 4 corner junctions defining a Kanizsa-type target square (the target present condition). This is illustrated in the upper panel of **Figure 1B**. "Kanizsa-type" here refers to the illusory geometrical forms that emerge as a function of the collinear organization of appropriately oriented corner junctions. An example of this is illustrated in **Figure 1A** (right panel). In a further 50% of trials the matrix of corner junctions presented after premask-matrix presentation included elements that did not group to form an illusory square (the target-absent condition).

The factors in all experiments were Prime (Intra-phase/Interphase premask presentation), Target (Present/Absent) and premask-matrix presentation Frequency (*f* —see **Table 2**). Following a brief computer-generated tone, participants were presented with the flickering 3 × 3 premask matrix which after 600 ms reduced to a matrix of simple 90◦ corner junctions by removal of superfluous line segments. Participants had then to

### **Table 2 | %Errors.**


#Refers to experiment number; f to the premask-matrix presentation frequency range (in Hertz); #Trials to the number of trials overall; #Errors to the number of errors; %Errors to the number of errors expressed as a percentage of overall number of trials.

discern the presence or absence of a Kanizsa-type square (target) within this matrix and produce a target-present/absent reaction time (RT), using one of two separate response keys, as rapidly and accurately as possible. In each experiment all factors were varied randomly: **Table 1** details the numbers of blocks, trials and sessions over which participants completed the experiments.

### **RESULTS OF STUDY 1**

Those trials with error responses (see **Table 2**) were removed from the data prior to subsequent analyses. Error RTs tended to be slower overall than correct RTs, and analysis of the probability correct by RT revealed no significant correlation between RT and accuracy, which argues against the correct data being contaminated by accuracy-speed trade-offs. Examination of the correct RTs revealed non-normal distribution with pronounced positive skew. A Kolmogorov "D" test showed RT distributions to be approximately lognormal and on this basis subsequent analyses were conducted on the exponents of the means of logtransformed RT distributions (for supporting ideas see Box and Cox, 1964, 1982).

In this study we sought (i) to establish evidence for priming across a wider range of frequencies than those presented by Elliott and Müller (2004), while (ii) better specifying the bandwidths at which priming is discovered. Evidence of priming is typically indicated by significant Target × Prime interaction in the analysis of variance (ANOVA see all experiments reported in Elliott and Müller, 1998, 2000, 2001; Elliott et al., 2000; Conci et al., 2004; Becker et al., 2005; Elliott et al., 2006a; Shi and Elliott, 2007). Resolution of this interaction always refers to a target-specific RT advantage for the intra- vs. interphase premask conditions. Given each experiment examined the factors Target (present/absent and indicative of search), Prime (intra vs. inter and indicative of priming) and *f* (premask-matrix presentation frequency), we sought to resolve either the three way interaction, which would indicate both target- and frequency-specific priming, and/or one or both two-way interactions (Target × Prime; Prime × *f*), which would be expected to indicate target- and frequency-specific priming, in the latter case if the error term was sufficiently large to render the three-way interaction non-significant (perhaps the case given the quite high resolution but narrow frequency bands examined in some experiments). **Table 3** presents relevant results of repeated measures ANOVAs carried out for each experiment.

With respects to previous studies, the only anomalies are an absence of a target effect in Experiment 4c (indicating absent search to be as efficient as target search) and an inability to resolve the Target × Prime interaction in Experiment 4a. This may be due to the proximity of the examined frequency band to the intraphase-premask detection threshold at 21 Hz: leading to effects on target-absent trials. Note that analysis of the arcsine-transformed error data using repeated-measures ANOVA with the same terms as those used for analysis of the RT data showed no systematic effects (Target or Target × Prime) between experiments, indicating that participants properly performed the target detection task and were not differentially influenced by any frequency or frequency band in doing so. Using the method proposed by Grice et al. (1977), we were unable to find the data of any participant to be influenced by speed-accuracy trade-offs.

Simple-effects analyses were carried out to determine at which frequencies priming occurs using the error terms from the threeway interaction, or from the Prime × *f* interaction resolved in analysis of the target trials only (predicted from a significant Target × Prime interaction carried out for Experiment 3). Priming frequencies are given in **Table 4**.

Preliminary analysis quite clearly shows that the modulated pattern of frequency-specific priming identified and reported by Elliott and Müller (2004) not only replicates across experiments, it may replicate and be specified with a finer resolution across a range spanning (at least) 29.5–67.75 Hz. That is to say across a very major part of the EEG gamma bandwidth. Study 1 shows some evidence for priming at 29.5 Hz alongside more robust evidence at bands spanning 32.25–33.5 Hz; 38–40 Hz; 45.75– 46.5 Hz; 53.25–53.75 Hz; with wider bands identified as spanning 59–60.75 and 65.75–67.5 Hz. Although not supported by significant Prime × *f* interactions it is likely that even within these fast-frequency bands there are particular frequencies that prime more efficiently than others. No other frequency tested was associated with priming. Together with the frequency specificity of priming this reinforces the idea that prime formation is a function of the Generalized Phase Angle Hypothesis (GPAH). For illustrative purposes the target RT data for Experiments 1–7, following


T, Target; P, Prime; f, premask-matrix presentation frequency. Huynh-Feldt or Greenhouse Geisser adjustments applied where sphericity assumptions are not met (see Huynh and Feldt, 1976).


Prime × f interactions reported here refer to analysis of target trials only. Where not specified the interaction term derives from the omnibus ANOVA reported in *Table 3*; Reported are prime(f), the magnitude of the priming effects [in milliseconds (ms)] and significance (p). Huynh-Feldt or Greenhouse Geisser adjustments applied where sphericity assumptions are not met. In #6 and #7 a number of frequencies primed resulting in a strong priming main effects [F(1, 10) = 32.9, p < 0.0001 for #6; F(1, 10) = 25.3, p = 0.001 for #7] but with reduced power in the Prime × f interactions. Reported priming frequencies were tested for significance using simple-effects analysis and the error term in the Prime × f interaction. However, in these instances, as the significance values are themselves at best indicative, they are not reported here.

removal of a linear slope that describes a general increase in RT with increasing frequency, are presented in **Figure 3**.

To test the GPAH the Prime × *f* pattern was examined using the Lomb-Scargle or least-squares spectral-analysis method. This method allows for an analysis of cyclical structure with a better resolution than conventional Fourier methods and is designed for short time series or time series with unequal or missing data (Scargle, 1982). Frequency separation between priming bands was originally suggested to be 6.75 Hz. The Lomb-Scargle method was applied to all differences between intra and interphase target RTs over all premask-matrix presentation frequencies in Experiments 1–7, that is to say across both frequencies at which priming was recorded as well as frequencies at which there

was no priming (see **Table 4**). **Figure 4**, presents the resulting Lomb-Scargle periodogram which exhibits a single significant peak located at 6.69 Hz. This is remarkably close to the original estimate of 6.75 Hz and makes a much more substantive case for consideration of the prime as a function of the interaction in phase between premask-matrix and EEG-theta rhythms at (1000/6.69 =) 149 ms intervals.

Having established that approximately the same frequency separation predicted by the GPAH over the 28–51-Hz range, characterizes priming between approximately 27 and 68 Hz, it is logical to suppose that the data support the Return Phase Hypothesis (RPH). Simple priming (e.g., that shown by Elliott and Müller, 1998, 2000, etc.) usually occurs when the prime is presented immediately prior to target presentation. However, and exploiting the logic that the premask frame (phase of premask presentation) will differ if premask-matrix presentation frequencies differ but presentation time overall remains constant, Kompass and Elliott (2001) concluded that the relative position of frame 1 (containing the priming stimulus) at termination of premask-matrix presentation determines whether it is an effective prime or not. If premask-matrix frame 1 is about to be presented or has just been presented, primed target RTs are maximally reduced leading to larger priming effects.

For the present experiments, averaged data over premaskpresentation frequency were examined using a simplified version of the procedure described by Kompass and Elliott (2001). For each frequency, the time of target presentation (600 ms posttrial onset) is transformed into a corresponding phase of the premask-matrix cycle. In **Figure 5** the subtraction of the mean intra- from interphase RTs (for some frequencies the priming effects) are plotted for each frequency over the resulting phase. **Figure 6** shows schematically that targets will be presented at a different phase (frame) in the cycle of premask-frame presentations dependent upon the relation between premask-matrix

**FIGURE 5 | Priming as a function of return phase: on the ordinate the subtraction of the mean intra- from interphase RTs (for some frequencies the priming effects) are plotted for each frequency over the time of target presentation (600 ms post-trial onset) transformed into a corresponding phase of the premask-matrix cycle.** Here, the abscissa is divided into two where Phase = 0 represents the onset of frame 1—the priming frame (asterisked). To the left of this are represented frequencies at which the target appears at a phase corresponding to frames 1 and 2 and to the right are frequencies at which the target appears at phases corresponding to presentation of frames 3 and 4. Frames are illustrated below in sequence from frame 1 (the priming frame) to the left to frame 2 at which point the sequence continues to the far right proceeding leftwards from frame 3. Frame presentation is cyclical until premask-matrix termination at 600 ms (see

**Figure 1**). Above are listed the frequencies that correspond to data points in the chart. Almost all priming frequencies lie in the phase range 0.025 (53 Hz) to –0.3 (38 Hz) with the very large majority at times corresponding to target presentation in phase with frame 4, that is to say, ahead of priming stimulus (frame 1) presentation. The non-linear function describes the pattern of mean intra- from interphase RTs over premask-matrix phase (for some frequencies this is the magnitude of priming) [<sup>y</sup> <sup>=</sup> <sup>36</sup>∗x<sup>3</sup> <sup>−</sup> <sup>14</sup>∗x<sup>2</sup> <sup>−</sup> <sup>27</sup>∗<sup>x</sup> <sup>+</sup> <sup>11</sup>; <sup>F</sup>(3,79) <sup>=</sup> <sup>6</sup>.23, <sup>p</sup> <sup>=</sup> <sup>0</sup>.001] and shows peak priming to occur at phase −0.193, which collapsed across frequencies is equivalent to approximately ¼ (22.8%) the duration of a single frame (the relevant frame being frame 4). Considering the lowest and least frame-1 proximal of the priming frequencies contributing to the frame 4 peak (32 Hz **Table 4**), this indicates a fully developed prime as early as 24 ms [(1000/32.5)∗0.772] ahead of priming-stimulus presentation.

frequency and the premask stopping frame given a fixed overall premask duration of 600 ms. In **Figure 5**, the abscissa is divided into two where Phase = 0 represents the onset of frame 1—the priming frame. To the left of this are represented frequencies at which the target appears at a phase corresponding to frames 1 and 2, and to the right are frequencies at which the target appears at phases corresponding to presentation of frames 3 and 4. In this conceptualization the onsets of frames 3 and 4 are represented by negative phase values because, relative to the cyclical premask-matrix presentation, they occur at times more proximal to the subsequent presentation of priming frame 1. Positive values represent frames more proximal to a previous onset of frame 1. What is clear from inspection of **Figure 5** is that the maximum differences between intra- and interphase RTs occur for targets presented at times that peak midway through the frame 4 phase and so are slightly ahead in phase relative to frame 1. In this respect, the data of Study 1 is overall consistent with the RPH described by Kompass and Elliott (2001) and indicates that a prime develops in advance of priming-stimulus presentation. The most conservative parametric estimates of the RPH derive from fitting a curve to the data (presented in **Figure 5**). This analysis indicates that priming should be expected to be at its maxima during the first quarter of the presentation phase of frame 4 (indicated in **Figure 5**). Consideration of the lowest and least frame-1 proximal of the priming frequencies contributing to the frame 4 peak (32 Hz **Table 4**), suggests the prime can be developed as much as 24 ms ahead of priming-stimulus presentation.

## **DISCUSSION OF STUDY 1**

Experiments 1–7 show that oscillatory priming, originally found by Elliott and Müller (1998) to be particular to 40-Hz and subsequently by Elliott and Müller (2004) to occur selectively but at regularly spaced frequencies in the 28–51-Hz band, both replicates and extends as far as 67 Hz. On this basis it may be concluded that the capacity for oscillatory prime formation is present across the majority, if not all of the EEG gamma band. The GPAH is shown here to refer to a slower, presumably internal rhythm of 6.69 Hz, with which it is believed premask-matrix presentation frequencies must phase align in order to bring about the priming effects (see **Figure 2**).

Related to this, analysis of the RPH reveals priming to occur for target presentation times that correspond in phase with the final frame of the sequence of premask-image frames. In other words, the prime forms at a time preceding presentation of the priming frame. This leads us to ask whether the RPH is a conclusive description of the conditions required for priming. At issue is a problem of confusability between the RPH and GPAH, brought about because premask matrices were all presented for 600 ms irrespective to frequency. Four ∗ 149 ms (the 1/*f* representation of the GPAH value of 6.69 Hz) is almost equivalent to the time of premask-matrix termination (596 vs. 600 ms). Perhaps some or all priming frequencies are favored due to the interaction of premask-matrix termination time with the slower (149 ms/6.69 Hz) rhythm?

There were two experiments in Study 2: experiment 8 sought clarity on the issue discussed above by using frequencies in the range 28–51 Hz in single Hertz steps while setting premaskmatrix presentation time to 700 ms. A presentation time of 700 ms does not harmonically relate to the GPAH rhythm and so allows examination of the hypothesis that it is the matching phase relation that exists between priming frequencies and 6.69 Hz at 149 ms that brings about the priming effects (illustrated in **Figure 2**). While priming effects have been demonstrated at various premask-matrix presentation times for 40 Hz (Elliott and Müller, 1998), in each case premasks were presented at close to integer multiples of 149 ms (300, 600, 1200, 2400 ms) and a regular frequency sequence was not examined. Experiment 9 was based on two findings in Experiment 8. The first refers to the function describing priming over phase (illustrated subsequently in **Figure 7**). This shows maximum priming at phase 0, and so at the time when targets were presented in phase with priming-frame (frame 1) presentation. The second is the finding that priming occurs across frequency bands rightward shifted by approximately 1 Hz relative to the priming frequencies recorded in Study 1 and previous studies (Elliott and Müller, 1998, 2004). Experiment 9 tested the RPH directly by setting premask-presentation times at different premask-matrix return phases at which priming was expected to differ irrespective to premask-matrix presentation frequency. Accordingly, it also questioned the frequency-specificity of oscillatory priming.

**FIGURE 7 | Description of this figure is identical to that of Figure 5 except in Experiment 8 premask matrices oscillated for 700 ms and in Experiment 9, for various frequencies, presentation times corresponded to cycles of 21 (frame phase 0.25), 22.4 (phase −0.4), and 23.8 (phase −0.05) of the stimulating frequency.** Analysis of Study 1 (described in **Figure 5**, symbols and function here in gray) is also presented for the purpose of comparison. The priming effects recorded in Experiment 8 (black symbols and function) are unlike those of Study 1 in that almost all priming frequencies lie in the phase range 0.09 (47 Hz) to −0.02 (34 Hz) and are centered on target presentation in phase with priming stimulus (frame 1) presentation. The solid black line is a non-linear function that describes the pattern of mean intra- from interphase RTs over premask-matrix phase

## **STUDY 2 MATERIALS AND METHODS**

In Experiment 8 there were 12 participants (4 male, mean age 24.1, all with normal or corrected-to-normal vision) and in Experiment 9 there were 15 participants (5 male, mean age 23.5, all with normal or corrected-to-normal vision). In Experiments 8 and 9, participants were paid at a rate of C8.00 per session. All participants provided informed consent to participate in the experiments, with protocols approved by a departmental research ethics committee convened at the Ludwig-Maximilians University, Munich.

The design and procedure used in Experiment 8 matched that of Experiment 2, except the 24 frequencies in the range 28– 51 Hz were presented in a single design with 3840 trials delivered over 5 sessions with 16 blocks per session. Unlike Experiment 2, Experiment 8 employed a premask-matrix presentation time of 700 ms.

In Experiment 9, the design and procedure used matched that of previous experiments with the following exceptions. Six frequencies were presented with single-Hertz resolution in the range 35–40 Hz in a single design with 2592 trials delivered [<sup>y</sup> = −8.5∗z<sup>2</sup> <sup>−</sup> <sup>0</sup>.37∗<sup>z</sup> <sup>+</sup> 12, where <sup>z</sup> <sup>=</sup> (<sup>x</sup> − −0.0083)/0.58; <sup>F</sup>(2, 21) <sup>=</sup> <sup>8</sup>.16, p < 0.005] and confirms peak priming to occur at phase 0. As expected the priming effects recorded in Experiment 9 are similar to those of Experiment 8 (red symbols, rounding errors cause some phases to be slightly out of alignment) with priming in the range 0.25 to −0.05 and so primarily retroactive. The solid red line is the corresponding non-linear function that describes the pattern of mean intra- from interphase RTs over premask-matrix phase [<sup>y</sup> = −2∗z<sup>2</sup> <sup>+</sup> <sup>2</sup>.6∗<sup>z</sup> <sup>+</sup> 12, where <sup>z</sup> <sup>=</sup> (<sup>x</sup> − −0.017)/0.52, function estimated on the limited set of phase means that allow no estimate of significance]. This function suggests peak priming to occur at a phase of 0.09. Importantly and given a lack of interaction with frequency, Experiment 9 indicates return phase and not frequency as key to oscillatory priming.

over 3 sessions with 18 blocks per session. As in Experiment 8, premask-matrix presentation time was varied but in the following manner: for each premask-matrix presentation frequency and independent of frequency, the premask matrix was presented for times equivalent to 21, 22.4, or 23.8 premask-matrix frame presentations. This varied premask-presentation time for each frequency × frame number combination is illustrated by the following examples: for premask matrices presented at *f* = 36 Hz—equivalent to 1/*f* = 27.78 ms, premask-matrix presentation times would be 583, 622, and 661 ms (corresponding to 21, 22.4, and 23.8 premask-matrix frame repeats, respectively). Reference to **Figure 5** confirms that these multiples correspond to premask- presentation frame phases of 0.25 (in phase with the end of frame 1), −0.4 and −0.05. For premask matrices presented at *f* = 39 Hz—equivalent to 1/*f* = 25.64 ms, premaskmatrix presentation times would be 538, 574, and 610 ms [corresponding to 21, 22.4, and 23.8 premask-matrix frame repeats and corresponding to premask- presentation frame phases of 0.25 (in phase with the end of frame 1), −0.4 and −0.05, respectively].

Given only one frequency × frame phase combination corresponds with priming as predicted by the GPAH, in Experiment 9 we expected peak priming to occur for targets presented slightly ahead and immediately after frame 0 (i.e., for 23.8 [frame phase −0.05] and 21[frame phase 0.25]) but not for targets presented following 22.4 (frame phase −0.4) premaskmatrix frame presentations. In this way, Experiment 9 was a direct test of the RPH, but also examined whether particular premask-presentation frequencies are necessary for oscillatory priming.

In all other respects the methods used in Experiments 8 and 9 were identical to those described for the previous experiments.

### **RESULTS OF STUDY 2**

As with the previous experiments, trials with error responses were removed from the data prior to subsequent analyses. In Experiment 8 there were 2116 errors (4.59% of all trials), and in Experiment 9 there were 1007 errors (2.59% of all trials). In both cases the error RTs tended to be slower overall than correct RTs, and analysis for speed-accuracy relations using the method recommended by Grice et al. (1977) revealed no significant correlations, arguing against the correct data being contaminated by accuracy-speed trade-offs. As with the previous experiments, repeated-measures ANOVA revealed no systematic patterns in either set of error data. Examination of the correct RTs revealed non-normal distribution with pronounced positive skew. A Kolmogorov "D" test showed RT distributions to be approximately lognormal and on this basis subsequent analyses were conducted on the exponents of the means of logtransformed RT distributions. Analysis of Experiment 8 was carried out as previously described for Study 1. For Experiment 9, a repeated measures ANOVA was carried out on the factors Target (present/absent and indicative of search), Prime (intra vs. inter and indicative of priming), *f* (premask-matrix presentation frequency) and Frame Phase (frame phases 0.25, −0.4, and −0.05, equivalent to 21, 22.4, and 23.8 premask-matrix frame repetitions).

In Experiment 8 priming overall was found to be weaker than in Experiment 2 leading to the reduced significance of the Target × Prime interaction (**Table 5**). What is clear is that there are still frequencies at which priming occurs selectively and that some of these frequencies differ from those in Experiment 2. In fact, there are similar bands of priming frequencies in Experiment 8 to those in Experiment 2, but these bands appear to be rightward shifted by around 1 Hz in the frequency dimension with priming occurring at slightly faster frequencies than in Study 1.

Of particular importance is the transformation shown in **Figure 7**. Plotted in comparison with data from Study 1 (gray symbols and function, described fully in **Figure 5**), the black function and symbols present data from Experiment 8. These show that priming effects no longer occur for frequencies at which the target appears at phases corresponding to presentation of frames 3 and 4, but are instead exactly centered on phase = 0, which is the time equivalent to the onset of frame 1—the priming frame. The Lomb-Scargle periodogram presented as the red function in **Figure 4** shows that the separation of frequencies can no longer be described in terms of a significant regular pattern at any frequency. Experiment 8 thus shows that for premaskmatrices terminating at a time out of phase with the 6.69 Hz (the rhythm characterizing the separation and therefore phase alignment of priming frequencies in Study 1), prime generation no longer appears to precede, but achieves maximum activation in synchrony with priming-stimulus presentation.

The results of Experiment 9 are consistent with the expectation that if premask-matrix presentation times vary and are not necessarily in phase with the slow theta rhythm of 6.69 Hz, prime maxima will not precede but will be aligned with or follow frame phase 0 at which target presentation aligns with frame 1 or priming-stimulus presentation. In addition and importantly, the pattern of priming effects over frame phase, alongside an absence of major interactions between Prime, Frame Phase and *f* , indicate the Prime × Frame Phase interactions do not vary as a function of premask-matrix presentation frequency. Consistent with expectations, based initially upon the RPH but


T, Target; P, Prime; f, premask-matrix presentation frequency. Huynh-Feldt or Greenhouse Geisser adjustments applied where sphericity assumptions are not met. T(P × f) interactions reported here refer to the Prime × f interaction in the target trials only. T(Pf) refers to target-priming effects. For Experiment 9, T(P × PH) and T(Ph) refer to Prime × Frame Phase interactions in the target trials. There were no significant Prime × Frame Phase × Frequency interactions in Experiment 9.

guided more specifically by the results of Experiment 8, peak priming was found when targets were presented immediately following presentation of frame 1 with priming also evident at a time equivalent to the end of frame 4 (e.g., for 23.8 [frame phase −0.05] and 21[frame phase 0.25]) premask-matrix frame presentations. Priming was not found for targets presented following 22.4 (frame phase −0.4) premask-matrix frame presentations. This pattern of results clearly indicates that it is return phase and not frequency alone that determines which premaskmatrix presentation frequencies prime target detection. Review of the priming frequencies supports this: the mean or greater than mean inter minus intraphase RTs were found for premask matrices presented at 35 and 39 Hz (frame phase 0.25) and 36, 37, and 38 Hz (phase −0.05). No effects were previously found for premask matrices presented at 35, 36, or 37 Hz, while Experiment 9 failed to record priming effects when premask matrices flickered at 40 Hz.

### **GENERAL DISCUSSION**

Study 1 shows that oscillatory primes generated through repeated and regular priming stimulus presentation, achieve maximum priming potential at a presentation phase that predates the phase of priming-stimulus presentation. This is a strong endorsement of the Return Phase Hypothesis (RPH), which also appears to model priming more reliably than either expectation that particular frequencies will prime preferentially over others (e.g., 40 Hz, Elliott and Müller, 1998, 2000), or the Generalized Phase Angle Hypothesis (GPAH) which predicts anticipatory but not synchronized or retroactive prime activation maxima.

In consideration of presentation parameters such as premask-frame presentation frequency and the phase of target presentation, this appears attributable to an interaction between premask-frame presentation frequency and an internal rhythm of 6.69 Hz. Varying premask-matrix presentation time and thereby the phase of subsequent target presentations, Study 2 appears to corroborate this conclusion. Experiments 8 and 9 show that when targets appear out of phase with the 6.69 Hz rhythm, target priming is weaker and no longer achieves its maximum at a premask-matrix presentation phase ahead of the phase of priming-stimulus presentation. On the one hand a finding of pre-activation in cognitive mechanisms coding regular stimulus events should not be surprising: it is well-known that in order to respond to movement successfully trajectories need to be computed in advance of the event so, for instance, one can place ones hand in the correct position in space at the correct time to successfully intercept and catch the incoming ball. On the other, it is unexpected that anticipatory coding occurs in conjunction with a slow oscillation in a band more commonly associated with working memory (and so in this instance retroactive coding—see Klimesch et al., 2005). This may indicate EEG theta rhythms are more closely concerned with temporal coding than previously thought, whether that coding is pre- or retroactive).

Anticipatory coding is not a new phenomenon as evidenced in tapping tasks using stimuli in isochronous visual sequences (e.g., Klemmer, 1957, 1967) referred to as Negative Mean Asynchrony, tapping slightly ahead of metronomic stimulus presentations is reviewed in Aschersleben (2002), and Repp (2005). Consideration of frequency highlights one similarity between the tapping studies and those presented here: notably and given motor and other physical constraints, the upper rate limit for 1:1 synchronization between stimulus and tapping response is observed at rates of 5– 7/s and so at rates which include, and may be in synchrony with the 6.69 Hz reported here. It seems quite possible that an anticipatory response may generate in sensori-motor mechanisms following premask-matrix presentations at particular frequencies, and that this response is based upon the phase alignment of premaskframe frequency with the slow theta oscillation also measured in tapping. However, (and with it in mind that sensori-motor may refer to the eye muscles as equally as it does to those resulting in a finger movement), claiming similarities must, in the present context, also acknowledge major differences between the design of tapping studies and that of the experiments presented here. In the present experiments participants do not respond to the beat and are not required to pay attention to any temporal aspects of the premask matrix to successfully complete the detection task, particularly not the rates of premask-frame presentations which are in any case too fast to be resolved perceptually. This qualifies acceptance that the upper rate limits of tapping and the internal rhythm measured here are the same rhythm, a question resolved by combining paradigms in future research. Task differences may, however, account for why a regular anticipatory priming effect is observed across a fairly broad and very fast frequency band, whereas anticipatory tapping tends to decrease with increasing tempo and occurs at much slower frequencies.

Of major significance, the experiments presented here show that anticipatory priming (prime pre-activation) occurs maximally when priming frequencies are in phase with the 6.69-Hz rhythm, but not when frequencies and rhythm are out-of-phase. Under the latter conditions, priming occurs for targets presented at phases either synchronous with or slightly after the phase of priming-stimulus presentation. Consequently, a summary hypothesis is that anticipatory coding occurs generally given a phase alignment of the oscillation imposed by the stimulus, or the oscillation at which stimulus coding occurs spontaneously, with the cycle of a slower oscillation. It can also be concluded that anticipatory coding does not take place when this interaction does not occur. It cannot be concluded, however, that 6.69 Hz is the only slow frequency permitting this interaction; mathematically several other frequencies may align phases at several other times leading to the possibility of other slow rhythms being instrumental in anticipatory coding. Some evidence that this may not be the case derives from Experiment 8 in which similar frequencies were found to prime as in Study 1, but at a different phase, where a different series of frequencies might be expected to prime if there had been an interaction with a rhythm of 5.7 Hz (700 ms is a multiple of 5.7 Hz, which has periods of 175 ms duration). However, a more systematic investigation is required in which premask-matrix presentation time is varied while holding premask-matrix frequency constant to reach any firm conclusions about the exclusivity of 6.69 Hz.

While it is interesting that prime pre-activation seems to occur only as a function of the phase interaction between different oscillations, it is possible with some caution to relate this with other phenomena: for example, induced and evoked gamma responses in physiological and EEG recordings and Husserl's notion of protention. The prime here is a spatial grouping that expedites detection of a subsequently presented visual grouping (the target) and on this basis the paradigm of Elliott and Müller has been discussed in terms of the visual-binding literature (Gray, 1999; Singer, 1999, for reviews). In this literature oscillations are induced by stimulus activity and are not usually phase related to that activity, while in a second literature (e.g., Herrmann et al., 1999), visual binding is evoked (that is to say phase related) selectively by presentation of a grouping stimulus. In the present experiments, examples of binding (priming) are discussed in terms of oscillatory-stimulus presentations at gamma-band frequencies. Primes are found to form both in phase with and at phases not directly related to the timing of the priming stimulus. It remains to be seen whether this priming paradigm provides a single method by virtue of which it can be shown that induced and evoked gamma responses are functionally equivalent with respects to visual binding.

With respect to protention, the prime is oscillatory and maximally active ahead of the phase in premask-matrix presentation at which the priming-stimulus frame is presented. In a sense the cognition advances in phase to end up preceding the event it codes and in this respect one might claim the prime to advance into future time—in the sense intended by Husserl. This would require a closer investigation of prime evolution over time. In this respect the study of Elliott and Müller (1998) presented premask matrices for 300 ms as well as for longer durations in an unblocked design, meaning participants could not know in advance for how long a given matrix would flicker. They found no differences between frequencies in terms of priming potential, which would suggest that by 600 ms the prime is pre-active (and therefore in future time relative to priming stimulus as well as priming stimulus presentation). However, the intraphase premask that generates the prime is not detected by participants for premaskmatrices presented at frequencies higher than 21 Hz (Elliott and Müller, 1998) and while visible in the overall context of the premask matrix, the contents of this frame are not distinguishable temporally from the contents other frames. This indicates that, if pre-active, the prime cannot be considered protentive in the sense intended by Husserl—that is, fully realized as a unique psychological event in consciousness, ahead of the actual event. Instead we might assume that conscious realization of events is mediated sometimes by anticipatory cognition and sometimes by cognition that occurs subsequent to the coded event. On this assumption we might conclude that if conscious experience itself is not protentive, the cognitive basis for conscious experience shifts between past, present and future times even though this cognition would not result in a relative time-stamped experience of event structure. In so far as our cognition at this level provides a content structure for consciousness, our psychological lives may be fundamentally based upon the ability of our cognitive states to travel backwards and forwards across very short intervals of time.

### **ACKNOWLEDGMENTS**

This research was funded by deutsche Forschungsgemeinschaft project grant EL248/1 to Mark A. Elliott. The author is indebted to Hermann J. Müller, Hans Geissler, Raul Kompass, and Stanislava Antonijevic for their contribution and to Konstantin ´ Meyer, Diana Böttger, Juliane Wendt, Doerthe Seifert, and Bernhard Ostler for their assistance in carrying out the experiments.

## **REFERENCES**


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 22 May 2014; accepted: 20 August 2014; published online: 12 September 2014.*

*Citation: Elliott MA (2014) Atemporal equilibria: pro- and retroactive coding in the dynamics of cognitive microstructures. Front. Psychol. 5:990. doi: 10.3389/fpsyg. 2014.00990*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Elliott. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Psychological time as information: the case of boredom†

## *Dan Zakay1,2 \**

<sup>1</sup> School of Psychology, Interdisciplinary Center, Herzliya, Israel

<sup>2</sup> Department of Psychology, School of Psychological Sciences, Tel-Aviv University, Tel-Aviv, Israel

### *Edited by:*

Simon Grondin, Université Laval, Canada

### *Reviewed by:*

Michael Roy, Elizabethtown College, USA Joseph Glicksohn, Bar-Ilan University, Israel

#### *\*Correspondence:*

Dan Zakay, Department of Psychology, School of Psychological Sciences, Tel-Aviv University, Ramat-Aviv 69978, Tel-Aviv, Israel e-mail: dzakay@post.tau.ac.il

†This paper is dedicated to the memory of Iris Levin, A great time researcher.

The flow of time is experienced by humans although the exact nature of time is not well understood. The importance of time in humans' life is not in dispute and is reflected by several dimensions like duration, which is best representing the naïve meaning of time. Psychological time serves several important functions which are essential for being able to act and survive in a dynamic environment. In the present paper we argue that psychological time in the form of sensing the pace of the flow of time provides important information to the executive system which control and monitor behavior. When information processing load is below an optimal level for a specific Individual a feeling of boredom is raised. Boredom is accompanied by a slowing of the felt pace of the flow of time. Boredom is a unique mental state which is linked with decreasing efficiency in cognitive and perceptual performance and is correlated with low job satisfaction and general well-being. As such, boredom poses a threat to normal functioning.We suggest that the felt slowing in the flow of time is a signal which, similarly to pain, is aimed at alerting the executive system that resources should be recruited in order to cope with the hazardous state.

**Keywords: boredom, prospective-timing, psychological-time, attentional-gate, information-processing-load**

## **PSYCHOLOGICAL TIME**

No understanding of human behavior can be complete without referring to the notion of time. Indeed, humans can sense the flow of time, but the exact nature of the mechanism by which this is done remains unclear. What humans (and maybe animals) experience are temporal experiences, which are subjective feelings that corresponds to physical time. Psychological time is a product of the mind more than a reflection of natural chronometric order (Trautmann, 1995). It refers to temporal dimensions such as duration, pace and the order of perceived and internal events. Psychological time provides our information processing system with important information that enables us to represent the environment in our cognitive system and to act accordingly.

### **PROSPECTIVE AND RETROSPECTIVE TIMING**

The experience of time is termed prospective when it is related to the duration of an ongoing interval and the observer is aware of the need to judge that duration. When an observer is not aware of the need to judge duration until the termination of a target interval, the experience of time is a retrospective one (Block, 1989). It is not clear if the subjective experiences associated with prospective and retrospective duration judgments are similar, but it is clear that the two types of temporal experiences are based on different information processing processes. Robust empirical findings as well as a comprehensive meta-analysis (Block and Zakay, 1997; Zakay and Block, 2004) indicate that different timing processes underlie the two judgment types. Retrospective duration judgments can be accounted for by the contextual change model (Block and Reed, 1978) which suggests that when retrospective timing is needed, people retrieve from memory contextual changes that were encoded during a target interval. Retrospective duration judgment is a function of the amount of retrieved contextual changes. The more contextual changes are retrieved, the longer the duration is judged to be. As a result, when information processing during an interval is complex (i.e., remembering a complex geometrical figure), the interval is judged to be longer in retrospect than a respective interval in which information processing was simple (i.e., remembering a simple geometrical figure like a circle). This is because complex information processing is causing more contextual changes to be encoded than simple information processing.

In contradistinction, prospective duration judgment is a function of the amount of attentional resources allocated for timing. The more resources are allocated for timing the longer prospective duration judgment is (Brown, 1997). The result is a mirror image of retrospective duration judgment. Prospective duration judgments of same time periods are longer when non-temporal information processing during a target interval is simple than when it is complex. The reason is that the more demanding non-temporal information processing is, the more attentional resources are consumed by it, leaving fewer resources for timing (Zakay, 1999).

## **ATTENTION AND PROSPECTIVE TIMING**

At any given moment, attentional resources are divided between all the concurrent tasks that need to be carried out simultaneously, including timing (Kahneman, 1973; Zakay, 1989). Zakay and Block (1995) introduced the attentional gate model (AGM) which is based on Church and Gibbon's (Church, 1984) timing model in animals. An attentional gate was added to the animal model. The gate is controlled by the amount of attentional resources allocated for timing and determines the number of pulses emitted by a pacemaker that can pass through the gate in a time unit. The pacemaker emits the pulses continuously at a constant pace. The pulses are accumulated and counted in a cognitive timer (Wearden, 2004). The more attentional resources are allocated for timing, the more pulses are "allowed" to pass through the gate. Thus, prospective duration judgment is a function of the number of accumulated pulses in a given time period (Zakay and Block, 1997). A similar attention-based model, but with a different gating mechanism was introduced by Lejeune (1998). In this model a dynamic switch is controlled by the attentional resources allocated for timing. The switch is opened and closed at a frequency determined by the amount of attentional resources allocated for timing. With more attentional resources, the higher the frequency and the larger the number of pulses that can pass through, and be accumulated in the cognitive counter (For a comparison between the two models see Zakay, 2000).

The attentional gate as well as the dynamic-switch serves as mechanisms for the regulation of attentional resources between concurrent non-temporal tasks and timing. According to both models when non-temporal tasks are simple and non-demanding prospective timing of same clock time intervals will be longer than when non-temporal tasks are complex and demanding. Because of this state of affairs, prospective duration judgment can serve as a measure of non-temporal information processing load (Zakay and Schub, 1998). The reason for this is that at any given moment attentional resources have to be divided between all concurrent tasks that have to be performed, temporal or non-temporal, and because prospective duration judgment is a function of the amount of attentional resources allocated for timing, prospective duration judgment can be used as a sensitive measure of concurrent non-temporal information processing load. When the load is low, more attentional resources can be allocated for prospective timing and duration estimations become longer as compared with conditions with high non-temporal information processing load. This was validated in several studies (see Brown, 2008) as well as in a meta analytic review (Block et al., 2010).

### **FUNCTIONS OF PSYCHOLOGICAL TIME**

As was already stated, temporal experiences are essential for enabling humans to represent the temporal aspects of both the external and internal environments, thus enabling adaptation and survival. For example, being able to judge the duration of an event is essential for knowing how to deal with similar events in future encounters. However, we argue that temporal experiences provide the cognitive and meta-cognitive systems with important information which enables optimal monitoring of behavior. Monitoring of spoken communication is just one example (Zakay et al., 2014). In the course of a conversation when party A asks partyBaquestion, a temporal expectation regarding the response latency is evoked. This temporal expectation reflects some kind of a norm. When actual response latency is significantly longer or shorter than the temporal expectation, party a suspects that the response is not based on real knowledge and therefore it can't be trusted. This indicates that the duration of the response latency is being monitored and timed prospectively. The ongoing prospective duration judgment is compared with the temporal expectation, a process which is well illustrated by the AGM.

In this paper we focus on one specific function of psychological time, namely, providing information about the concurrent level of non-temporal information processing load. This function was not yet elaborated in the literature.

### **THE NEED FOR INFORMATION AND FOR INFORMATION PROCESSING LOAD**

Humans need a certain amount of information in order to maintain a satisfactory level of adaptive behavior (Kuhltham, 1991). Information is a product of variability in stimulation (Garner, 2014). Woodburn (1957) reports experiments on human behavioral effects following prolonged exposure to a monotonous environment. It was found that under such conditions thinking was impaired, childish emotional responses appeared, visual perception was disturbed, hallucinations developed and brain wave patterns were altered. Similar findings were found in sensory and perceptual deprivation experiments (Zubek and Welch, 1963; Zakay and Lobel, 1983; Grassion, 1986). Similar effects are found in real life situation which resemble perceptual deprivation like in the case of snow-blindness which create a ganzfeld (Avant, 1965). It can be concluded that the need for meaningful information is a genuine need of the cognitive system which strives to gain the optimal amount of information (Merhabian, 1977).

### **TEMPORAL EXPERIENCES AS INFORMATION**

Michon (1972) introduced the idea of considering time as information. He meant that temporal experiences provide information about the succession of events. Like other perceptual dimensions, psychological time provides our information processing system with important information that enables the representation of the environment such that adaptive behavior becomes possible.

We elaborate on this notion and propose that temporal information informs the executive system which control and monitor behavior about the ongoing state of the system's performance and functioning. It should be noted that information is more than mere stimulation. This is demonstrated by studies of perceptual deprivation (e.g., Grassion, 1986), in which the amount of information is normal but it lacks variability. More research is needed in order to understand the exact nature of temporal information. For example: How one feels the pace of time? Are retrospective and prospective experiences providing the same type of information or different types? Regardless of the need for more research, the importance of temporal information is clear.

Since any type of behavior takes a certain amount of time, by monitoring the actual time a certain behavior endures and by comparing it to temporal norms or expectations, it is possible to monitor the regularity of behavior. We already gave the example about monitoring the adequacy of spoken communication (e.g., Boltz, 2005). Another example is waiting behavior (Zakay et al., 2009). When one is waiting for an event, and the event is delayed in comparison to the expected waiting duration, a temporal experience of slowing of the pace of time accompanied by a general

feeling of tension emerges (Osuna, 1985; Loftus et al., 1987). This signals the system that something is wrong.

Here we focus on the fulfillment of the need for information and for information processing. When this need is not satisfied the system is in danger of not being able to perform optimally, as will be illustrated in the next paragraphs. This state is manifested as an emotion of boredom accompanied by a temporal experience felt as the slowing of the pace of the flow of time or boredom, which signals the system that it is currently engaged with a suboptimal level of non-temporal information processing.

## **BOREDOM**

Boredom is defined as a unique psychophysiological state possessing interrelated and inseparable emotional, motivational, perceptual, and cognitive concomitants (O'Hanlon, 1981).

Boredom is a common emotion, which can appear as a result of a specific situation or as a typical characteristic of an individual. In the last case we speak about boredom proneness (BP), which is a predisposition to experience boredom (Farmer and Sundberg, 1986).

Boredom is an important issue in psychology, education and work-life, and yet, it is not receiving the adequate attention from researchers, as it deserves. The importance of boredom stems from its links with well-being, psychopathologies, jobsatisfaction and other important aspects of human behavior (Smith, 1981).

Situational boredom is experienced when one finds him/herself in a situation in which most of one's attentional resources are free and are not allocated to a specific task which demands information processing. This might be the result of a monotonous environment which lacks stimulation and variance, or from having to perform a routine, non-challenging task or having to listen to a redundant lecture which does not provide any new information and is read in a monotonous voice. The difference between boredom and situations like leisure or play is that whereas in the latter one is absorbed in the activity and a sense of time disappears, in boredom one wishes to quit the situation and the sense of time is augmented (Csikszentmihalyi, 1990). People characterized as having high BP tend to experience boredom even in situations in which the level of stimulation and required information processing load are normal (Csikszentmihalyi, 2000).

Boredom is maintained by an environment that is perceived as static, with the actor remaining largely disconnected from the processes that comprise the environment (Farmer and Sundberg, 1986). Boredom can be induced experimentally by exposing participants to sensory or perceptual deprivation conditions for long periods (Zakay and Lobel, 1983; Grassion, 1986). In reality, boredom and monotony at work were found to be associated (Drory, 1982).

Boredom and BP are negatively correlated with need for cognition, which indicates a lower level of cognitive motivation than that of other people (Cacioppo et al., 1996). Watt and Blanchard (1994) found that individuals, who were less likely to engage in an enjoyable effortful cognitive activity, were more prone to experience negative affects of boredom when compared to high need-for-cognition persons.

## **BOREDOM AND MALADAPTIVE BEHAVIOR**

Boredom and BP were found to be linked with maladaptive behavior in several domains.

Whereas boredom was not found to be significantly related to levels of intelligence and education (Hill, 1975), it is recognized as a widespread and significant problem. Boredom and lack of curiosity were reported to be the most common cause of drug use (Samuels and Samuels, 1974), and has been associated with eating disorders for both obese and non-obese persons (Abramson and Stinson, 1977).

Bored students were rated more often as maladjusted by teachers in comparison to other students (Fogelman, 1976). In worklife, job dissatisfaction and diminished performance efficiency, tend to be highly correlated with boredom and BP (O'Hanlon, 1981). BP was found to be a predictor of aggressive and risky driving (Dahlen et al., 2005).

Evidences of an inverse relationship between the ability to cope adaptively with boredom and psychopathology were reported by Hamilton et al. (1984). High boredom- copers reported better well-being and greater compliance with organizational safety rules, compared with low boredom- copers (Annilee, 2007).

Positive correlations between boredom and BP and between level of hopelessness in a hopelessness scale and negative correlations with personal life satisfaction across many dimensions were reported (Neugarten et al., 1961).

### **BOREDOM, BP, AND PSYCHOLOGICAL TIME**

Based on the former review of prospective timing and attentional model like the AGM, the state of boredom can be defined as a mental state characterized by low level of non-temporal information processing load. The negative emotion which accompanies boredom leads one to wish for the ending of the situation, and therefore, like in waiting, most of a person's available attentional resources are allocated for prospective timing. The result is a feeling of duration lengthening or slowing of the pace of the flow of time (Zakay, 2012). Note, that even if the feeling of the flow of time might be considered a perceptual illusion (Gruber and Block, 2013; Block and Gruber, 2014), it is still a source of information. This is similar to perceptual illusions like apparent motion or the phi phenomenon which provide significant information, albeit an illusionary one. Not much research has tested the relation between perceived duration and boredom. In one of the BP scales (Farmer and Sundberg, 1986) three items out of 28 relate to time. Watt (1991) found that highly boredomprone individuals perceived time as passing more slowly during a boring task than low boredom-prone persons, but the two groups did not differ in their objective or chronometric timepassage estimates. Similarly, Wittman and Paulus (2008) report that high boredom-prone individuals perceive a slowing of the pace of time and overestimated durations in time-estimation tasks, when a reproduction method was used. (Note that if a production method is used durations will be underestimated, respectively).

It is of interest to note that the other pole of boredom, termed "flow," is a state of peak enjoyment, energetic focus and creative concentration experienced by people engaged in adult play which has become the basis of a highly creative approach to living (Csikszentmihalyi, 1990, 2000). From Psychological time perspective, whereas boredom produce a significant increase in duration judgment, flow is a state in which attentional resources are almost fully allocated for non-temporal information processing and as a result duration judgment is minimized (Time flies by when you are having fun; see Zakay, 2012. For more interpretations of this phenomenon see Gable and Poole, 2012, or Sackett et al., 2010).

### **CONCLUSION**

Psychological time fulfills several vital functions like in the planning and performing of psychomotor activities and movements (Flanagan and Wing, 1997) and in meta-cognition, like in human monitoring of human communication (Zakay et al., 2014). In the present paper we elaborate the notion that psychological time should be considered as information. We further dwell into this consideration and suggest that temporal information is essential in alerting the executive system which control and monitor behavior that the overall level of information processing load is lower than the optimal level required for an adequate functioning of the system. This is manifested by a unique state and emotion called boredom. We reviewed studies showing that boredom and BP are linked with lowering cognitive and perceptual performance, with the use of drugs, with lowering job satisfaction and educational achievements, in reducing the amount of effort one can invest in performing tasks and in low level of need for cognition, and with lowering the level of general well-being. The practical consequences of boredom include diminished performance efficiency and health (O'Hanlon, 1981; Annilee, 2007). This is obviously a hazardous state that is not desired from an evolutionary perspective. People usually wish to find themselves in a mental state of "Flow," which is the opposite pole of boredom, defined as a state of peak enjoyment, energetic focus, and creative concentration (Csikszentmihalyi, 2000). Indeed, Wilson et al. (2014) found that people preferred to administer electric shock to themselves instead of doing nothing.

In a state of boredom the felt pace of the flow of time is slowed down. This can be explained by attentional models of prospective timing like the AGM (Zakay and Block, 1995). Prospective duration judgment is sensitive to the division of attentional resources between concurrent temporal and non-temporal tasks because it is a function of amount of attentional resources left for timing after the required amount was allocated for concurrent non-temporal tasks. From this perspective, we suggest that the alerting function of psychological time is similar to that of pain. Pain is an unpleasant sensory and emotional experience associated with actual or potential tissue damage. Ecclest and Crombey (1999), suggest that pain is salient in naturally complex environments because the selection of pain interrupts attention, rupture behavior and imposes priority on escape actions. The interruptive function of pain is that pain is selected for action from within complex affective and motivational environments to urge escape. In both cases of boredom and pain, the system is required to change the division of attentional resources in order to cope with the situation.

The alerting function of psychological time is based on a comparison between ongoing prospective temporal judgments and temporal norms and expectations. We suggest that each individual is characterized by an idiosyncratic level of information processing (Merhabian, 1977), required for optimal behavior. When this level is achieved, it is accompanied by a certain felt pace of time. The ongoing felt pace of time is continuously compared with the norm and a state of a too slow pace gives rise to a feeling of boredom.

Further research is needed in order to validate the function of psychological time which was introduced here. A better understanding of the link between boredom, temporal experiences, and information processing might lead to a more complete comprehension of the functions of psychological time on the one hand, and to contribute to the development of effective methods for dealing with the state of boredom and with boredom-proneness.

### **REFERENCES**


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 29 June 2014; accepted: 01 August 2014; published online: 21 August 2014. Citation: Zakay D (2014) Psychological time as information: the case of boredom. Front. Psychol. 5:917. doi: 10.3389/fpsyg.2014.00917*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Zakay. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Children's mental time travel during mind wandering

## *Qun Ye, Xiaolan Song\*, Yi Zhang and Qinqin Wang*

Department of Psychology, College of Education, Zhejiang Normal University, Jinhua, China

### *Edited by:*

Claire Zedelius, University of California at Santa Barbara, USA

#### *Reviewed by:*

Gregory West, University of Montreal, Canada Brianna Morseth, University of California at Santa Barbara, USA

### *\*Correspondence:*

Xiaolan Song, Department of Psychology, College of Education, Zhejiang Normal University, 688 Yingbin Avenue, Jinhua, Zhejiang 321004, China e-mail: xlsong@zjnu.cn

The prospective bias is a salient feature of mind wandering in healthy adults, yet little is known about the temporal focus of children's mind wandering. In the present study, (I) we developed the temporal focus of mind wandering questionnaire for school-age children (TFMWQ-C), a 12-item scale with good test–retest reliability and construct validity. (II) The criterion validity was tested by thought sampling in both choice reaction time task and working memory task. A positive correlation was found between the temporal focus measured by the questionnaire and the one adopted during task-unrelated thoughts (TUTs) by thought sampling probes, especially in the trait level of future-oriented mind wandering. At the same time, children who experienced more TUTs tended to show worse behavioral performance during tasks. (III) The children in both tasks experienced more futureoriented TUTs than past-oriented ones, which was congruent with the results observed in adults; however, in contrast with previous research on adults, the prospective bias was not influenced by task demands. Together these results indicate that the prospective bias of mind wandering has emerged since the school-age (9∼13 years old), and that the relationship between mental time travel (MTT) during mind wandering and the use of cognitive resources differs between children and adults. Our study provides new insights into how this interesting feature of mind wandering may adaptively contribute to the development of children's MTT.

**Keywords: mental time travel, mind wandering, task-unrelated thought, spontaneous thought, prospective cognition**

### **INTRODUCTION**

Mental time travel (MTT) refers to"the faculty that allows humans to mentally project themselves backward in time to re-live, or forward to pre-live events" and plays a very important role in autonoetic consciousness that helps people to maintain a continuous feeling of "self" extending from the personal past through the present to the personal future (Tulving, 1985). Recent research has demonstrated that MTT can arise spontaneously or involuntarily (Berntsen and Jacobsen, 2008), that is, being initiated without any conscious act of the will. This phenomenon is labeled as spontaneous (involuntary) MTT, and has been found very common in everyday life (Berntsen, 1996; Rubin and Berntsen, 2009). Although a lot of research on the spontaneous MTT has been conducted with adults, little is known about children's spontaneous MTT. Conducting such research on children is especially important given that the presence of MTT has been established in early childhood (Atance and O'Neill, 2005; Busby and Suddendorf, 2005).

Mind wandering refers to engagement in cognitions unrelated to the current demands of the external environments (Schooler et al., 2011), and constitutes as much as 50% of our waking thoughts (Killingsworth and Gilbert,2010). Many researchers have suggested that the content of mind wandering shows remarkable temporal focus (Smallwood et al., 2009b; Song and Wang, 2012), suggesting that spontaneous MTT may be the major component in mind wandering. Moreover, it has been suggested that the variation in mind wandering represents an important individual difference (Kane et al., 2007; Mrazek et al., 2013; Unsworth and McMillan, 2014), and that this difference can also be reflected in its temporal focus. Yet so far to date, there was not a valid tool to measure the individual differences in spontaneous MTT.

Furthermore, the prospective bias (i.e., people are more inclined to experience future-oriented mind wandering rather than past-oriented mind wandering) of mind wandering is a salient feature in healthy adults, and has been observed in different populations (Smallwood et al., 2009b, 2011; Smallwood and O'Connor, 2011; Stawarczyk et al., 2011; Song and Wang, 2012). There is also some evidence to suggest that executive resources could support future-oriented mind wandering. Specifically, subjects with higher working memory capacity (WMC) tend to experience more future-oriented mind wandering than the ones with lower WMC (Baird et al., 2011), and the prospective bias of mind wandering was curtailed by the requirement to continuously monitor the task (Smallwood et al., 2009b). Such results could be explained by the argument that simulating the future recruits more cognitive resources than recalling the past (Addis et al., 2007; Szpunar et al., 2007). In the current research, the question is whether there are any prospective biases of mind wandering in children? If so, what is the relationship between different temporal focuses of children's mind wandering and executive resources?

Given the above considerations, in study 1, we aimed to develop and validate a measuring tool – the temporal focus of mind wandering questionnaire for children (TFMWQ-C) – to explore the characteristics of children's spontaneous MTT during mind wandering. And in study 2, we tested the criterion validity of the TFMWQ-C in two laboratory tasks with different working memory load, and explored how executive resources would influence the temporal focus of spontaneous MTT during mind wandering.

It has been suggested that the key developments of MTT occur between 3 and 5 years of age (Atance and O'Neill, 2005; Busby and Suddendorf, 2005; Atance, 2008b), and its neurophysiological foundations also need to develop throughout childhood (Atance, 2008a). According to our preliminary research, in which we interviewed 100 children about their understanding of their inner experiences, children could not describe their mind wandering experiences accurately by introspection until 8 years old (Chen, 2013). Consequently, we only included children ages 8 or older to participate in the present research.

## **EXPERIMENT OVERVIEW**

Two studies were conducted to investigate the characteristics of children's spontaneous MTT during mind wandering. Study 1 developed the TFMWQ-C and determined its reliability and validity in a large sample of school-age children. Study 2 used a separate sample to explore the criterion validity of the TFMWQ-C by experience sampling method (ESM) in two tasks with different cognitive load, and so the association between the executive resources and the temporal focus of their spontaneous MTT for children could be explored.

## **STUDY 1**

In the current study, we aimed to develop and validate a questionnaire to measure the individual differences in the temporal focus during spontaneous MTT in daily life for primary school students. At the same time, the gender and grade differences in the temporal focus of spontaneous MTT were investigated.

### **METHODS AND RESULTS**

### *Participants*

The first sample [for item development and exploratory factor analysis (EFA)] included 490 school-aged children from a primary school in East China (52.3% female, mean age = 10.94, range 8–14 years; Sample B+C, **Table 1**). The second sample [for confirmatory factor analysis (CFA)] included 250 schoolaged children from the same school (54.4% female, mean age = 10.53, range 8–13 years; Sample D, **Table 1**). An independent sample of 66 students was recruited for test–retest reliability (54.5% female, mean age = 10.64, range 9–14 years; Sample E, **Table 1**).

### **Table 1 | Sample characteristics.**


The research procedure was in accordance with the ethical principle of the 1964 Declaration of Helsinki (World Medical Organization). The institute review board of Zhejiang Normal University approved the research procedure.

### *Interview*

To facilitate the development of the item pool to the target population, we recruited 22 school-aged children from two primary schools for the pilot interview (see Sample A, **Table 1**). The main contents of the interview involved the interviewee's understanding of mind wandering, and the content, temporal focus, frequency and emotional valence of their mind wandering experiences. We encouraged the interviewees to give several examples of their own mind wandering experiences. Each interview was conducted one on one and lasted for about 20 min. Because the main goal of the current study was to assess children's MTT during mind wandering, only the results about the temporal focus were described here.

The participants' responses in the interview showed that the children age 8 and older were very familiar with their mind wandering experiences, and had no difficulty in giving examples and reflecting on mind wandering episodes in their daily lives. As soon as children showed the capacity to generate these examples, tendencies to recall the past and to envision the future were both very common in their reports of mind wandering episodes.

### *Item development*

The item pool was constructed based on the interview and the existing questionnaires about the daydreaming experiences for adults and children (Singer and Antrobus, 1972; Rosenfeld, 1979; Vooijs et al., 1992). After piloted item development with 222 participants (see Sample B, **Table 1** and expert reviewed system, 18 items were retained. A five-point response scale (1-strongly agree, 2-agree, 3-uncertain, 4-disagree, 5-strongly disagree) was used to promote adequate variance and scale reliability.

### *Exploratory factor analysis*

The sample included 333 Chinese school-aged children, and 268 complete surveys were obtained with an efficient rate of 80.48% (see Sample C; **Table 1**). A principal component analysis (PCA) with oblique rotation reduced the 18 items to two factors, which accounted for 55.19% of the variance in the data. Six items were removed as they did not load strongly on one factor or had limited conceptual relevance to any particular factor. The two factors were labeled as "Future Orientation" (item 1, 3, 5, 7, 9, 11) and "Past Orientation" (item 2, 4, 6, 8, 10, 12) respectively. Inventory items and factor loadings are presented in **Table 2**. The Kaiser measure of sampling adequacy was 0.91, sharing common factors among these items. Bartlett's test of Sphericity also indicated that the variable data was suitable for factor analysis.

### *Confirmatory factor analysis*

We performed a CFA of the two-factor model in a new sample of 250 school-age children (see Sample D; **Table 1**), using maximumlikelihood estimation and the AMOS 4.0 (Arbuckle and Wothke, 1999). The correspondence between the fitted covariance matrix of the two-factor model and the sample covariance matrix of the

### **Table 2 | Pattern matrix factor loadings of the TFMWQ-C (oblique rotation method).**


Factor loadings exceeding 0.5 are highlighted.

actual data was tested by a number of fit indices. χ<sup>2</sup> (a nonsignificant value corresponds to an acceptable fit), is known to be susceptible to estimate parameters and sample size and it has been emphasized that it is unusual to obtain a non-significant χ<sup>2</sup> when performing CFA on self-report questionnaires (Byrne, 1994). In order to reduce the effect of sample size, it is generally considered that the ratio between chi-square and degrees of freedom <2 can be used to indicate the fitness of the model (Carmines and McIver, 1981). In addition, four indices of model fit were computed: the root mean square error of approximation (RMSEA), the standard root mean square residual (SRMR), the comparative fit index (CFI), and the goodness-of-fit index (GFI). These values all reached the ideal of the priori standard [χ<sup>2</sup> (53)=85.755, *<sup>p</sup>*<0.05, <sup>χ</sup>2/df <sup>=</sup> 1.618, RMSEA <sup>=</sup> 0.050, SRMR <sup>=</sup> 0.046, CFI <sup>=</sup> 0.965, GFI = 0.944]. And a moderate degree of correlation between the two factors was revealed (*r* = 0.58, *p* < 0.001).

At the same time, all 12 items of the scale were significantly related to the latent factor (all *p* < .001) and the average value of these item's factor loadings was 0.63. And the Cronbach's Alpha value for TFMWQ-C was 0.86, indicating the satisfactory internal reliability of the scale. The combination of these indices indicated an acceptable fit.

### *Test–retest reliability*

The temporal stability of the TFMWQ-C was examined in an independent sample (see Sample E, **Table 1**) over a 2 week period. Paired sample correlation analysis showed an excellent test–retest reliability of TFMWQ-C in school-age children (The Pearson *r* was 0.70 for Future Orientation, 0.78 for Past Orientation, and 0.80 for total score of TFMWQ-C respectively, *p* < 0.001).

### *Gender and grade differences*

In this part, we looked at gender and grade differences in the TFMWQ-C based on the sample that the CFA was conducted on. A multivariate analysis of variances (ANOVA; dependent variable: average score in the Past Orientation and Future Orientation respectively) revealed no main effect of gender or grade, but a significant Gender × Grade interaction effect (Pillai's Trace value was 0.05, *F* = 3.28, *p* < 0.05). To better understand the interaction effect, we performed univariate analysis on the variances of the two factors respectively (Past Orientation: interaction effect, *F*(2,244) = 5.17, *p* < 0.01; Future Orientation: interaction effect, *F*(2,244) = 4.14, *p* < 0.05). The simple main effects were further analyzed, and the results showed that for Future Orientation, the scores of girls varied across grades [*F*(2,244) = 3.45, *p* < 0.05], while those of boys did not [*F*(2,244) = 1.38, *p* = 0.26]. *Post hoc* LSD tests indicated that for girls, the scores on Future Orientation of the fourth and fifth grade were higher than that of the third grade (both *p* < 0.05), and no difference was observed between the fourth and fifth grade (*p* = 0.83). The results for the scores on Past Orientation were substantially the same with Future Orientation [Girl: *F*(2,244) = 6.63, *p* < 0.01; Boy: *F*(2,244) = 1.42, *p* = 0.24]. This interaction effect suggested that there was a significant increase between 3rd and 4th grade in past-oriented and future-oriented mind wandering, but only for girls (**Table 3**, **Figures 1** and **2**).

### **DISCUSSION**

In study 1, we developed a questionnaire to assess MTT during mind wandering for school-age children. The resulting TFMWQ-C was shown to have a reliable two-factor structure, derived by EFA and confirmed by CFA, and was found to have satisfactory internal reliability and test–retest reliability. Our findings thus confirmed the suggestion that there are empirically determinable individual differences in MTT during mind wandering for schoolage children.

It also showed that at least for school-aged children, the tendency to experience past-oriented MTT during mind wandering was positively correlated with the future-oriented counterpart,



which suggests that the ability to simulate future events relies on many of the same cognitive and neural processes as remembering past events (Schacter and Addis, 2007; Schacter et al., 2007; Buckner, 2010; Richmond and Pan, 2013).

Another finding was the gender difference in the development of temporal oriented mind wandering, which suggests that there is a significant increase in the experiences of spontaneous MTT between 3rd and 4th grade for girls but not boys. This finding parallels other gender differences in cognitive abilities, such as differences in verbal ability, that emerge during the primary school years (Hyde and Linn, 1988).

## **STUDY 2**

Having established the construct validity and internal consistency reliability of TFMWQ-C in study 1, we next evaluated the criterion validity of TFMWQ-C in two laboratory tasks that were commonly used in mind wandering research [choice reaction time task (CRT) for study 2A and 1-back working memory task (WMT) for study 2B] by examining the correlation between the participants' scores on TFMWQ-C and their frequencies of past and future-oriented MTT during mind wandering. In these two tasks a thought sampling method was used to catch participants' spontaneous episodes of mind wandering. This method was the most widely used one in assessing mind wandering, involving periodically interrupting participants by thought probes during a task at unpredicted occasions and asking them to report whether their real-time inner experience was on the task or task-unrelated thoughts (TUTs; Giambra, 1995; Teasdale et al., 1995; Schooler et al., 2004). There was a broad literature validating the self-reported mind wandering obtained through thought sampling method in a variety of task situations among adolescents and adults (Smallwood and Schooler, 2006; Smallwood et al., 2008; Christoff et al., 2009; Killingsworth and Gilbert, 2010; Mrazek et al., 2013).

Another goal of Study 2 was to explore the temporal focus of children's spontaneous MTT during laboratory tasks. Although the prospective bias of mind wandering in different situations has been observed across different populations among adults (Smallwood et al., 2009b, 2011; Stawarczyk et al., 2011; Song and Wang, 2012), little is known about whether it is the same for children.

For adult participants, previous research demonstrated that future-oriented thinking was more prevalent in CRT tasks than in WMT (Smallwood et al., 2009b). In addition, some researchers have argued that working memory load should disproportionately reduce the amount of prospective thought relative to retrospective thought in adults (Smallwood et al., 2009b, 2011). Therefore the third goal of this study was to explore whether the availability of executive resources influences the prospective and retrospective focus during mind wandering for children in the same way as adults. CRT andWMT are two tasks commonly usedfor investigating the relation between executive resources and prospective bias of mind wandering (Smallwood et al., 2009b, 2011), and they differ on the need to recruit executive resources. CRT merely requires the subjects to wait for a color number to occur and judge whether the number is odd or even, while WMT makes the subjects keep the recent numbers in mind and judge whether previous number was odd or even. Here study 2A adopted CRT and study 2B adopted WMT.

## **STUDY 2A**

### **MATERIALS AND METHODS**

### *Participants*

Seventy-one school-age children completed this experiment (see Sample F; **Table 1**). All participants had normal or corrected to normal vision. Two participants were excluded from the analysis as their accuracy rates in CRT exceeded 3 SD from average.

### *Procedure*

Participants first completed an adapted version of CRT (Smallwood et al., 2009b) and then completed the TFMWQ-C at a computer. At the end, the participants received a gift for their participation.

### *Choice reaction time task*

Stimuli for CRT were numeric digits, 1–9, which were constituted by 190 frequent non-target numbers (colored black) and 24 infrequent target numbers (colored green) in a white background with a quasi-random order of presentation. Stimulus presentation rate was 1 item every 2000 ms (followed by 1000 ms fixation cross). Participants were required to make a decision about whether an infrequent number was odd or even using the computer keyboard (F for the odd, J for the even). The stimuli were presented using E-Prime presentation software on a computer (Schneider et al., 2002). The testing session for this task lasted approximately 15 min.

### *Thought probes*

During the experiment, participants wore headphones. At six different pseudo-random occasions a "ding" sound suddenly appeared via the headphone with a prompt screen showing the thought sampling question with three options: "Just in the moments prior to the probe, what were you thinking about? 1-Thinking about something from the past,or 2-Just being on the task or 3-Thinking about something from the future." Participants made responses by keyboard. Before the task, the participants received instructions and examples explaining the different options to ensure all of them understood the question.

### **RESULTS AND DISCUSSION**

Participants maintained reasonable accuracy throughout the task (*M* = 0.83, SE = 0.01). Next we considered the correspondence between the trait level and the state level of the temporal focus in spontaneous MTT by computing the correlation between the scores on TFMWQ-C and the frequencies of future/past-oriented mind wandering during laboratory tasks. The results showed a moderate positive correlation between the average total score in TFMWQ-C and the frequencies of TUT in CRT (*r* = 0.26, *p* < 0.01), indicating that the participants who reported more spontaneous MTT in their daily lives also reported more TUTs during CRT (**Table 4**). At the same time, the score on Future Orientation in TFMWQ-C was positively correlated with the frequency of future-oriented TUTs during CRT task, whereas there was no such correlation in Past Orientation. Another interesting finding was that participants experienced more future-oriented TUTs than past-oriented TUTs [*M*future = 0.25, *M*past = 0.14, *t*(68) = 3.93, *p* < 0.001]. These results provide evidence that the prospective bias of mind wandering during laboratory tasks that has been observed in adults is also present in school-age children.

### **STUDY 2B**

### **MATERIALS AND METHODS**

The children who participated in study 2A completed the 1-back WMT 1 week later (see Sample F, **Table 1**). The procedure of the WMT was the same as study 2A except that here the participants were required to decide if the stimulus preceding the infrequent target (a green "?") was odd or even (targets *N* = 24, non-targets *N* = 202, probe *N* = 6). The experiment lasted ∼13 min. After


N = 69. Q.F, average score of Future Orientation in the questionnaire; Q.P, average score of Past Orientation in the questionnaire; Q.FP, average total score in the questionnaire; CRT.F, frequencies of future-oriented TUT in the CRT; CRT.P, frequencies of past-oriented TUT in the CRT; CRT.FP, frequencies of TUT in the CRT. \*p < 0.05, \*\*p < 0.01.

excluding two participants with very low accuracy rates and one participant with a software error from the analysis, there remained a total 66 valid data in both tasks.

### **RESULTS AND DISCUSSION**

Participants maintained similar reasonable accuracy rates throughout the task (*M* = 0.83, SE = 0.01), whereas it was not significantly different from that of CRT [*t*(65) = 0.11, *p* = 0.91]. However, reaction time in the WMT (*M* = 1279 ms, SE = 31.56) was significantly slower than the one in the CRT (*M* = 1002 ms, SE = 17.13), *t*(65) = 8.25, *p* < 0.001. From this perspective, the WMT indeed was more demanding than the CRT. As expected, TFMWQ-C scores also correlated with probe-caught TUTs during the WMT (*r* = 0.32, *p* < 0.01), and a positive correlation was found again between the temporal feature revealed by the questionnaire and TUTs during WMT, but this was also limited in the Future Orientation (**Table 5**). At the same time, the frequency of TUT during WMT was significantly associated with lower accuracy rate (*r* = −0.38, *p* < 0.01), which was consistent with the claim that mind wandering was always associated with poor performance during a highly demanding task.

Combined with the results of study 2A, a highly positive correlation between the frequencies of TUT during the CRT and the WMT indicated a reasonable level of consistency in mind wandering across two different demanding contexts (*r* = 0.55, *p* < 0.001). Next, we considered how the task demands influenced the temporal focus of TUTs. A 2 (temporal focus: past, future) × 2 (task environment: CRT, WMT) repeated ANOVA yielded a main effect of temporalfocus [*F*(1,65)=29.93, *p*<0.001, η2 <sup>p</sup> =0.315], indicating that future-oriented TUTs were more common than past-oriented TUTs. However, the main effect of task environment [*F*(1,65) <sup>=</sup> 0.22, *<sup>p</sup>* <sup>=</sup> 0.64, <sup>η</sup><sup>2</sup> <sup>p</sup> = 0.035] and the temporal focus × task environment interaction effect [*F*(1,65) = 1.89, *<sup>p</sup>* <sup>=</sup> 0.17, <sup>η</sup><sup>2</sup> <sup>p</sup> = 0.028] were not observed (**Figure 3**).

Children in both tasks tended to experience more futureoriented TUTs rather than past-oriented ones, as had been seen in adults. However, unlike adults, the prospective bias in children was not influenced by task demands (Smallwood et al., 2009b). There

**Table 5 | Means (***M***), standard deviations (***SD***), and correlations between TFMWQ-C andWMT.**


N = 68. Q.F, average score of Future Orientation in the questionnaire; Q.P, average score of Past Orientation in the questionnaire; Q.FP, average score in the questionnaire; WMT.F, frequencies of task-unrelated thought of Future Orientation in the WMT; WMT.P, frequencies of task-unrelated thought of Past Orientation in the WMT; WMT.FP, frequencies of task-unrelated thought in the WMT. \*p < 0.05, \*\*p < 0.01.

are two possible reasons for the result. One reason may lie in the fact that the participants did the CRT 1 week before the WMT, which made them more practiced in theWMT and reduced the difference of the executive load between the two tasks. Although the accuracy rate and the frequency of TUT indeed did not decrease significantly in the WMT, the longer RT suggested the higher executive load in WMT. Therefore the more likely explanation is that the relation between the temporal focus of MTT during mind wandering and the executive resources in children is different than in adults. Nevertheless, the prospective bias of mind wandering has emerged in school-age and the relation between different temporal focuses of mind wandering and executive resources deserves further investigation.

### **GENERAL DISCUSSION**

The studies presented in this article were designed to examine the temporal focus of children's spontaneous MTT during mind wandering. In study 1, we used EFA and CFA to develop the TFMWQ-C, which contained two factors: past and future orientation. The TFMWQ-C was demonstrated to be a reliable and valid instrument for measuring the individual differences in spontaneous MTT for children from 3rd grade to 6th grade in

primary school. Further validation of the TFMWQ-C is necessary across different cultures or special populations with extreme scores in the two temporal dimensions. For instance, specific deficits in prospective thought have been shown to increase suicide risk and undermine many adaptive cognitive functions (O'Connor et al., 2007), so we should pay more attention to the people extremely low in spontaneous MTT. Similarly, retrospective bias in mind wandering was often associated with negative mood (Smallwood and O'Connor, 2011). Therefore the relation between the propensities to engage in spontaneous MTT with different temporal focuses and mental health deserves further investigation, and early interventions for abnormal MTT during mind wandering would be valuable. In addition, it is important to recognize the gender and grade differences in the propensity to experience spontaneous MTT (i.e., for girls but not boys, there was a significant increase between 3rd and 4th grade in the past-oriented and future-oriented mind wandering.). Researchers have already provided a comprehensive review of necessary cognitive components for MTT, including working memory, self-awareness, theory of mind, and executive function (Suddendorf and Corballis, 2007). Therefore, the differences described above may reflect different trends in some of these components between gender and age. The key factors in the development of MTT for children will be an important issue for future research.

Study 2 provided evidence for the criterion validity of TFMWQ-C, which showed a significant positive correlation between the trait level and the state level of the temporal focus in spontaneousMTT.At the same time, the significant negative correlation between the frequency of mind wandering during theWMT and task performance suggests that thought sampling can also be appropriate for school-age children (see Mrazek et al., 2013 for related findings). Another intriguing finding was the lack of effect of executive resources on the temporal focus of spontaneous MTT for school-age children. For adults, the prospective bias in mind wandering was curtailed by the requirement to continuously monitor the task (Smallwood et al., 2009b). However, in the current study, the prospective bias of TUTs for children was not influenced by the task demands; that is, the children inclined to experiencing more future-oriented TUTs rather than past-oriented TUTs in both tasks. In order to address this discrepancy further, a better way may be setting a group of tasks with systematically increasing demands (e.g., 1-back, 2-back, and 3-back tasks) and observing the influence of task demands on the temporal focus of TUTs.

To our knowledge, this is the first study to investigate children's spontaneous MTT during mind wandering. Given that the study of different temporal focuses on children's mind wandering is in its infancy, there are a number of interesting directions for future research. First, mentally projecting the self forward in time enables a coherent and stable personal identity extending from the past to the future (Tulving, 1985). A growing number of researchers have acknowledged the close relationship between the ability to re-experience the past and simulate the future, and existing developmental and neuroimaging data suggested that thinking about one's past and future may be similar, but not fully overlapping in cognitive processes (Busby and Suddendorf, 2005; Szpunar et al., 2007). Consistent with this hypothesis, we also found the correlation between the tendencies to prospective and retrospective during mind wandering for children when a big sample was adopted in Study 1. It will be important to further examine the relationship between the mental processes involved in looking into the future and back to the past from a developmental perspective.

The second import direction for future research concerns the relation of mind wandering to the development of theory of mind, based on the assumption that both of them deal with the function of mental simulation. On the one hand, episodic representation is the main way to characterize mind wandering (Song and Wang, 2012). On the other hand, according to the simulation theory of theory of mind (Harris, 1992; Flavell, 2004), children become able to speculate the mental states of other people through a kind of role-taking or simulation process. Our view is that the situational characteristics of mind wandering are likely to provide more opportunities for mental simulation, thus contributing to the development of theory of mind. Our previous research showed that the 4 year old children who reported more experiences of mind wandering during a 3 min resting state had more advanced theory of mind ability (Chen, 2013). Similarly, studies have showed that fantasy assessments are significantly related to the theory of mind performance among preschool children (Taylor and Carlson, 1997). More research on this topic is therefore recommended.

Third, future research could also focus on the functionality of prospective bias during mind wandering in consideration of prospective memory (PM). PM refers to memory for activities to be performed in the future. According to Klinger's Current Concerns Theory (Klinger, 1999, 2009, 2013), mind wandering is often goal-directed and preparing for the future. If mind wandering possesses the value of anticipation and planning of personally upcoming events, then such properties may be best served by prospective thought during mind wandering (Baird et al., 2011). In the classic study of PM paradigm (Einstein and McDaniel, 1990), subjects often start with a brief distraction task, which aims to avoid the goal of PM tasks being stored in working memory and to generate a certain degree of forgetting, and then they perform the ongoing task embedded with prospective targets. An adapted design could explore the relation between the frequency of prospective thought in distraction/ongoing task and the performance on the PM task. If mind wandering helps one to better maintain PM targets while sacrificing speed or accuracy of distraction/ongoing task, then these lapses of attention could be viewed as instrumental (Cohen, 2013). In our view, situational characterization of mind wandering may promote the intentional encoding of PM as well as strengthen the links between the target and future goals, so as to improve performance on PM. Similarly, some studies have found that age and episodic future thinking abilities were significant predictors of PM performance (Nigro et al., 2013; Neroni et al., 2014). Therefore, future research could shed light on the relationship between the prospective thought during mind wandering and PM from an empirical perspective.

Last but not least, in contrast with the assumption that mind wandering always occurs at a significant cost to task performance, several studies have examined its potential virtues. Studies of fantasy in children have suggested the link between imaginative predisposition and creativity (Taylor, 1997; Singer and Singer, 2008). Interestingly, in the course of our interview, most of the interviewees admitted that the content of their mind wandering sometimes was beneficial to their creative activities (e.g., writing and inventions). The discrepancy in the perspectives to mind wandering may be due to the fact that there is more than one type of cognitive component in mind wandering, and that only some of them play positive roles. Individuals with similar frequencies of mind wandering can have enormous variations in the content and other characteristics of mind wandering. We argue that the study of mind wandering needs to take "deconstructive" strategies, not just "integrative" ones. The good news is that more and more researchers have begun to take this kind of approach (Smallwood, 2013). For instance, from the different temporal focuses (Smallwood et al., 2009b, 2011; Andrews-Hanna et al., 2010; Smallwood and O'Connor, 2011; Stawarczyk et al., 2011; Song and Wang, 2012), from the different emotional tones (Smallwood et al., 2009a; Killingsworth and Gilbert, 2010; Ruby et al., 2013), from the individual differences (McVay and Kane, 2012a,b; McVay et al., 2013), from the relation between mindfulness and mind wandering (Mrazek et al., 2012; Schooler et al., 2014), and so on. One might expect the deconstruction of mind wandering will be the focus of future research.

In conclusion, we extended the study of mind wandering to a child population, developed and validated a questionnaire to measure the individual differences in the temporalfocus of spontaneous MTT in daily lives for primary school students, investigated the temporal characteristics of their mind wandering in daily lives and laboratory situations, and suggested some future research directions in this area.

### **ACKNOWLEDGMENTS**

We thank all the participants for their contribution to this study, and we thank the staff of the Xiaoshan Yinhe Experimental Primary School for their help with the data collection. We thank Xueqing Zhu, Yi Zhang, Qinqin Wang, Qianqian Shen, Ruoyu Huang, Tong Jiang, and Xiaoxiao Gu for their assistance in conducting the study. We are particularly grateful to Gail Heyman for the help with writing. We also thank the Associate Editor Claire Zedelius and two anonymous reviewers for their helpful comments on the manuscript. This research was supported by grants from Zhejiang Provincial Social Sciences Foundation (11ZJQN076YB), Humanities and Social Sciences Fund of the Ministry of Education of P.R. China (10YJCXLX040), National Natural Science Foundation of China (31300832), and the Program for Innovative Research Team of Zhejiang Normal University (PIRT0809) to Xiaolan Song.

### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 20 May 2014; accepted: 05 August 2014; published online: 21 August 2014. Citation: Ye Q, Song X, Zhang Y and Wang Q (2014) Children's mental time travel during mind wandering. Front. Psychol. 5:927. doi: 10.3389/fpsyg.2014.00927*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Ye, Song, Zhang and Wang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

**OPINION ARTICLE** published: 19 June 2014 doi: 10.3389/fpsyg.2014.00624

## *Michael M. Roy1,2\**

<sup>1</sup> Department of Psychology, Elizabethtown College, Elizabethtown, PA, USA


*Edited by:* Simon Grondin, Université Laval, Canada *Reviewed by:*

Anne-Claire Rattat, University of Toulouse-CUFR JF Champollion, France

### **Keywords: time, optimism, bias, prototype, memory, future**

People appear to be biased in their predictions of the future; predicting that projects will take less time than they actually will (see Roy et al., 2005a; Buehler et al., 2010; Halkjelsvik and Jørgensen, 2012, for review) and that they will be unlikely to experience future misfortunes (see Dunning et al., 2004, for review). These biases are frequently attributed to people being overly optimistic (Armor and Taylor, 1998, 2002; Dunning et al., 2004). However, research indicates that many of these seemingly motivated biases could actually be due to task characteristics or other non-motivational causes (Juslin et al., 2000; Chambers and Windschitl, 2004; Roy et al., 2005a). Here I review research, focusing on my own work on time estimation and self-assessment, which indicates that these seemingly optimistic biases might often have other nonmotivational causes. Further, I discuss why a belief that people are overly optimistic might be problematic.

## **OPTIMISM IN TIME ESTIMATION?**

The tendency for people to think that they will finish tasks earlier than they actually will is frequently thought of as fitting into a larger category of optimistic biases (Armor and Taylor, 1998, 2002). It has been proposed that underestimation is due to people forming an optimistic scenario of how a task will be completed and ignoring memory for how long similar tasks have taken in the past (Kahneman and Tversky, 1982; Buehler et al., 2002).

However, underestimation of future task duration might not be due to optimism about the future, but instead be due to biased memories of the past (Roy et al., 2005a; see also Griffin and Buehler, 2005; Roy et al., 2005b). Similarly, it has been proposed that it is the quality of people's episodic (Szpunar et al., 2013) and semantic (Irish and Piguet, 2013) memories that predict their ability to envision the future (Schacter et al., 2008; Szpunar and McDermott, 2008). In support of the view that biased predictions are caused by biased memories, research indicates that factors that influence memories of past task duration have parallel influences on predictions of future task duration (Roy and Christenfeld, 2007, 2008) with biased memories frequently associated with biased predictions (Thomas et al., 2004, 2007). Furthermore, correcting memory for past task duration is one of the only interventions that has been found to successfully reduce bias and improve accuracy for predictions of future task duration (Roy et al., 2008). In relation, people seem to be accurate in estimates for tasks where frequent feedback on timing is given (Tobin and Grondin, 2012), but not for tasks where feedback is absent (Tobin and Grondin, 2009; Tobin et al., 2010; Bisson et al., 2012). In the field of software development, predictions of for how long a new project will take are more accurate when data on task duration for similar projects is utilized during predictions than when it is not (Jørgensen, 2004a; Furulund and Moløkken-Østvold, 2007).

While bias in predicted duration appears to often be due to bias in memory, memory itself could be biased by people's motivations. However, bias in remembered duration often appears to be due to task characteristics such as novelty (Boltz et al., 1998; Hinds, 1999; Roy and Christenfeld, 2007; Tobin et al., 2010), relative duration (Yarmey, 2000; Roy and Christenfeld, 2008; Tobin and Grondin, 2009), size of potential estimation anchors (Thomas and Handley, 2008), and duration since completion (Roy et al., 2008). These tasks characteristics may alter attention that is paid to the task (Thomas and Weaver, 1975) or memory storage size associated with the task (Ornstein, 1969; Block and Reed, 1978) and bias estimation. Task characteristics cause bias in remembered duration that in turn causes bias in predicted duration. For example, actors and observers appeared to be more influenced in their estimates, for both past and future tasks, by task characteristics, such as overall task duration and number of remembered task components, than they were by being the more motivationally involved person performing the task (Roy et al., 2013a; see also Byram, 1997; Hinds, 1999; Jørgensen, 2004b).

The reason that memory and prediction might be similarly biased is that both might rely on a similar constructive process with estimation based upon a general prototypical representation for task duration that is adjusted up or down depending on the specifics of the situation (Burt and Kemp, 1994; Roy et al., 2005a). Because memory for past task duration is often biased (see Wallace and Rabin, 1960; Fraisse, 1963; Block and Zakay, 1997 for reviews), the prototypical representations that people have for many tasks, which is the average of previous experience, is similarly biased. A result of prototypical memories that underestimate duration is an expectation that tasks in the future, such as travel plans, will take less time than they will in actuality (van de Ven et al., 2011). While people frequently underestimate how long tasks will take in the future, much of this bias appears to be due to task characteristics and not due to optimism or motivation. This is not to say that motivation never plays a role in bias; at times people's motivations influence their memories for past task duration (Meade, 1963; Schwab et al., 2013) and their predictions of future duration (Buehler et al., 1997; Byram, 1997). However, research indicates that bias in estimated duration frequently exists without motivational causes.

### **OPTIMISM IN OTHER JUDGMENTS?**

A person's estimate of task duration is often tied to their perceived competency at the task. For many tasks, quicker completion is linked to high ability. Indeed, more experience with a task is often linked to a greater tendency to underestimate task duration (Boltz et al., 1998; Hinds, 1999; Roy and Christenfeld, 2007). Further, people often appear to be overly optimistic in their self-assessments, rating themselves as above average on a number of skills and personality traits (see Taylor and Brown, 1988; Chambers and Windschitl, 2004; Dunning et al., 2004; Sedikides and Gregg, 2008, for reviews). It would seem logically impossible for the majority of people to be above average (Taylor and Brown, 1988). However, similar to time estimation, this apparent bias may be due to people relying on prototypical representations of ability when assessing their own abilities (Krueger, 1998; Gigerenzer, 2002; Moore, 2007; Galesic et al., 2012; Roy et al., 2013b). People understand that various skills have skewed ability distributions and their self-assessments are often related to distribution shape: high when most are believed to be good and low when most are believed to be bad (Galesic et al., 2012; Roy et al., 2013b). When the skill being rated has a skewed distribution, a tendency to rely on prototypical representations can lead to self-assessments that, on face value, appear to indicate a belief in the self as being falsely unique (above or below average), but actually may indicate a belief in the self as being prototypical (near the mode of the distribution; Roy et al., 2013b). Further, a reliance on prototypes can also help explain why people underestimate the likelihood that they will experience rare events, such as getting cancer, and overestimate the likelihood that they will experience common events, such as owning a car (Chambers et al., 2003; Kruger and Burrus, 2004).

It is also possible that other task characteristics, such as how clearly defined and specific the task is (Dunning et al., 2004; Roy and Liersch, 2013) or how much information people have about their own ability and the ability of others (Chambers and Windschitl, 2004), can influence selfassessments. On a related note, people's overconfidence in their responses on quizzes and tests might have more to do with the type of questions being asked than with actual self-confidence (see Juslin et al., 2000 for review). While people appear to overestimate their abilities and performance, these biases appear to often be due to specific aspects of the skill, performance or event and not due to optimism or motivation.

### **PROBLEMS WITH PERCEIVED OPTIMISM**

While people might not be overly optimistic about how their future will unfold, a *belief* in optimism as the cause of bias may be problematic both for researchers and for people in general. Inasmuch as biased predictions are viewed as being caused by optimism and that being optimistic is viewed positively, people might not seek out the true causes of bias and ways to eliminate bias.

Even though people acknowledge they are often biased in their predictions (Buehler et al., 2002; Armor et al., 2008), it is not clear if they know why they are biased. In general, people are not always able to accurately describe their decision-making processes (e.g., Wilson and Hodges, 1992). It appears that, like researchers (Taylor and Brown, 1988; Armor and Taylor, 1998, 2002; Dunning et al., 2004), people often attribute their errors to being overly optimistic (Armor et al., 2008). The belief in optimism as a cause of bias can be problematic for two reasons: first, people might be missing the real causes of their error. To make accurate and unbiased predictions for how long it will take to complete a task, people would need to realize that factors such as task length (Roy and Christenfeld, 2008), familiarity (Roy and Christenfeld, 2007), and complexity (Roy et al., 2013a) biased their memories for these tasks and correct these memories before estimating future task duration. Simply ascribing their error to optimism would make them miss the actual causes.

Second, while optimism appears to be a popular (Christensen-Szalanski and Beach, 1984), convenient, and more importantly, common excuse for error, there are few negatives to being seen as overly optimistic. While people believe that they are at times overly optimistic, they actually believe that they should try to be more optimistic (Armor et al., 2008). They appear to be willing to deal with what they believe are the negative consequences of their optimism. To a certain degree, they would be correct to do so because a large number of positive outcomes have been associated with having an optimistic outlook (e.g., Rasmussen et al., 2009). Because they do not mind their self-diagnosed cause of their bias, people will not be motivated to seek out the real cause. Interventions aimed at decreasing bias by decreasing optimism are likely to be unpopular as well as misguided (Roy et al., 2005a).

### **SUMMARY**

People often appear to be overly optimistic in their predictions of the future. However, much of this apparent optimism could be due to other processes such as an over reliance on prototypical representations. In the case of time estimation, people's representation for task duration is often too short, causing them to underestimate future completion times. To improve predictions, factors that bias memory need to be taken into account and corrected. If, instead, bias in prediction is attributed to an optimistic outlook, then people will be unlikely to identify and correct the true causes of their bias. Interventions might be better aimed at decreasing the *belief* in over-optimism, not optimism itself.

## **REFERENCES**


a meta-analytic review. *Ann. Behav. Med.* 37, 239–256. doi: 10.1007/s12160-009-9111-x


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 April 2014; paper pending published: 28 May 2014; accepted: 02 June 2014; published online: 19 June 2014.*

*Citation: Roy MM (2014) Belief in optimism might be more problematic than actual optimism. Front. Psychol. 5:624. doi: 10.3389/fpsyg.2014.00624*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Roy. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## A spoon full of studies helps the comparison go down: comparative analysis of Tulving's spoon test a

## *Damian Scarf\*, Christopher Smith and Michael Stuart*

Department of Psychology, University of Otago, Dunedin, New Zealand

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

### *Reviewed by:*

Mathias Osvath, Lund University, Sweden Michael Beran, Georgia State University, USA Corina J. Logan, University of California Santa Barbara, USA

### *\*Correspondence:*

Damian Scarf, Department of Psychology, University of Otago, P. O. Box 56, Dunedin 9054, New Zealand e-mail: damian@psy.otago.ac.nz

Mental time travel refers to the ability to cast one's mind back in time to re-experience a past event and forward in time to pre-experience events that may occur in the future. Tulving (2005), an authority on mental time travel, holds that this ability is unique to humans. Anticipating that comparative psychologists would challenge this claim, Tulving (2005) proposed his spoon test, a test specifically designed to assess whether non-human animals are capable of mental time travel. A number of studies have now employed the spoon test to assess mental time travel in non-human animals. Here, we review the evidence for mental time travel in primates. To provide a benchmark, we also review studies that have employed the spoon test with preschool children. The review demonstrates that if we compare the performance of great apes to that of preschool children, and hold them to the same criteria, the data suggest mental travel is present but not ubiquitous in great apes.

**Keywords: mental time travel, episodic memory, episodic foresight,Tulving's spoon test, comparative cognition**

At its heart, comparative psychology is founded on the principle of continuity. Darwin (1871) articulated this principle perfectly when he stated that any differences between human and non-human animal minds are differences of degree (i.e., quantitative) but not kind (i.e., qualitative). Undoubtedly, the vast and growing comparative literature is consistent with Darwin's (1871) view, however, several recent reviews have suggested that discontinuities may also exist (Tulving, 2005; Premack, 2007; Suddendorf and Corballis, 2007; Penn et al., 2008). On this point, views range from Penn et al.'s (2008, pp. 110) contention that Darwin (1871) was mistaken and overlooked the "profound functional discontinuity between the human and non-human mind" to a more nuanced view suggesting that, although the principle of mental continuity largely holds true, there are examples of discontinuity in mind between human and non-human animals (Tulving, 2005). In the current review we tackle one of the more widely asserted discontinuities – mental time travel (Suddendorf and Corballis, 1997, 2007; Roberts, 2002; Suddendorf and Busby, 2003; Tulving, 2005). The term mental time travel was coined by Suddendorf and Corballis (1997) and denotes the ability to cast one's mind not only back in time to reexperience a past event (i.e., episodic memory) but also forward in time (i.e., episodic foresight) to pre-experience events that may occur in the future.

Tulving (1983) initially conceptualized mental time travel purely in terms of the recall of past events (i.e., what) and their spatial (i.e., where) and temporal (i.e., when) context (Tulving, 1983). However, the fact that an individual could remember these aspects of an event without having personally experiencing it, led to a reconceptualization that included autonoetic consciousness or the knowledge that one's memory of an event is a product of them having personally experienced it (Tulving, 1985). Finally, Tulving (2002) added proscopic chronesthesia (i.e., foresight), suggesting the processes used to re-experience past events could also be utilized to pre-experience future events. Fittingly, recent functional

magnetic imaging research supports Tulving's view, demonstrating that the same brain areas are active when a person is asked to reflect on a past event and simulate a future event (Addis et al., 2007; Buckner and Carroll, 2007; Schacter et al., 2012).

While Tulving's (1983) conceptualization of mental time travel has changed over time, his view that it is a uniquely human ability has remained unchanged. For example, Tulving opened his seminal book, *Elements of Episodic Memory*, by stating "Remembering past events is a universally familiar experience. It is also a uniquely human one" (Tulving, 1983, pp. 1) and more than 20 years later he restated this thesis, "Human beings possess a form of memory (episodic memory) and a form of consciousness (autonoetic consciousness, or "autonoesis") that no other animals do. Thus, the thesis is that these two aspects of the mind are unique in humans, in the sense that the mental capacities that define them do not exist in quite the same full-fledged form in other species. They do not exist in insects, in birds, in mice or rats, in cats or dogs, and not even in gorillas and chimps" (Tulving, 2005, pp. 6). In addition to restating his thesis, Tulving (2005) also proposed the spoon test, a paradigm that would allow one to test it his claim that mental time travel is uniquely human. At present, we believe the spoon test is the best test of mental time travel that can be used with both young children and non-human animals and, consequently, it forms the backbone of this comparative review.

The spoon test builds on earlier proposals by Köhler (1922) and Suddendorf (1994), both of which highlighted the potential significance of demonstrating that a non-human animal could prepare in the present for a temporally distant event. To describe the spoon test, Tulving refers to an Estonian children's story in which *"*... *a young girl dreams about going to a friend's birthday party where the guests are served delicious chocolate pudding, her favorite. Alas, all she can do is to watch other children eat it, because everybody has to have her own spoon, and she did not bring one. So the next evening, determined not to have the same disappointing experience*

*again, she goes to bed clutching a spoon in her hand"* (Tulving, 2005, pp. 44). According to Tulving (2005) the young girl's behavior demonstrates that she is able to reflect on her experience of being at the party without a spoon (i.e., episodic memory) and, by placing the spoon underneath her pillow, shows that she can entertain the possibility she may again attend the party in her dreams (i.e., episodic foresight). Tulving (2005, pp. 44) suggests that if a nonhuman animal were to pass an analogous version of the spoon test it would "force the rejection" of his hypothesis that mental time travel is uniquely human.

## **DEVELOPMENTAL STUDIES**

Developmental studies of mental time travel provide an important comparison when assessing mental time travel in non-human animals. Those working with non-human animals have been set an ever growing number of criteria that their spoon tests must adhere to (Tulving, 2005; Suddendorf and Corballis, 2007, 2010; Suddendorf et al., 2009), however, one could argue that these criteria are only relevant in so far as they have been upheld in developmental studies that have concluded young children are capable of mental time travel (Suddendorf and Busby, 2005; Russell et al., 2010; Suddendorf et al., 2011; Scarf et al., 2013; Atance and Sommerville, 2014; Payne et al., 2014). To this end, we will first review the small number of developmental studies that have employed the spoon test to assess mental time travel in young children.

Suddendorf et al. (2011) presented 3- and 4-year-old children with a novel problem, in which a specially shaped key could be used to open a locked box. In the first room, children were shown a box with a triangle or cross shaped keyhole and the experimenter demonstrated how a key that matched the shape of the keyhole could be used to open it, revealing several stickers. Children were then given the opportunity to perform the task twice themselves, obtaining a sticker each time. Children were then distracted and the key they had previously used to open the box was replaced by a broken key. After demonstrating the broken key could not be used to open the box, the experimenter ushered the child into another room where they played games for 15 min. After 15 min children were presented with four differently shaped keys, one of which matched the key they used to open the box, and were told they could pick one key to take with them back to the first room. While the majority of 4-year-olds (65%) chose the correct key, the performance of the 3-year-olds (29%) was not significantly above chance.

Using a problem similar to that used by Suddendorf et al. (2011) and Scarf et al. (2013) had 3- and 4-year-old children dig up a locked treasure chest in a large outdoor sandbox. After establishing that they did not have a key to open the treasure chest, the experimenter asked children to go back to the lab with them. Children then left the lab and returned after a 24 h delay. When they returned to the lab, children were told they would be going back out to the sandbox and were asked to pick one of three items (a key, windup toy, or bouncy ball) to take with them. While a significant number of 4-year-old children selected the key, the performance of 3-year-old children was no different from chance. To further investigate the impact of the delay on the performance of the 3-year-old children, separate groups of 3-year-old children were tested after a 0, 15, or 30 min delay. The performance of the 3-year-old children

decreased in a linear fashion over the 0, 15, and 30 min delays. The impact of the delay on the performance of 3-year-old children (Scarf et al., 2013) suggests that, while 3-year-old children are capable of mental time travel, they are constrained by their ability to retain the original episode. Indeed, Atance and Sommerville (2014) have demonstrated that if memory of the original episode is controlled for, there is no difference between the performance of 3- and 4-year-old children on the spoon test.

One potential limitation of the developmental studies reviewed above is that they did not include a delay between the selection phase and children being given the opportunity to use the item they selected. Thus, the studies only tested children's foresight for the very next event (Redshaw and Suddendorf, 2013). However, building on Suddendorf et al. (2011) and Redshaw and Suddendorf (2013) recently demonstrated that the performance of 4-year-old children on the spoon test is not impacted by inserting a 5-min delay after the selection phase, suggesting that the mechanism used to plan for the very next event may be the same mechanism used to plan for more distant events. It will be important for future studies to investigate the potential impact of longer delays. In addition, it is an open question as to whether 3-year-old children are also unaffected by the imposition of a delay after the selection phase.

### **COMPARATIVE STUDIES**

Comparative studies of the spoon test have employed a different procedure to that used in the majority of the developmental studies reviewed above (cf. Redshaw and Suddendorf, 2013)1. Specifically, rather than insert the delay between the original episode and the selection phase, comparative studies have inserted the delay after the selection phase (cf. Beran et al., 2012). This difference somewhat complicates the comparison between studies, because in all the comparative studies there are examples of a subject selecting the correct tool, but failing to transport it to the testing room following the delay. However, given that a subject may simply select the correct tool due to the fact it has previously used it to obtain food, for the comparative studies we will define a successful trial as a subject selecting the correct tool, taking it with them to the delay room, and transporting it to the testing room2.

The first comparative study was conducted by Mulcahy and Call (2006), who taught bonobos and orangutans to use a tool to retrieve a reward from an apparatus. After learning this, the apes were presented with several tools (two suitable and six unsuitable) in the testing room but with access to the baited apparatus

<sup>1</sup>The present review focuses on comparing the performance of children and nonhuman animals on comparable spoon test paradigms. Due to this we have not included the innovative series of studies conducted by Correia et al. (2007) and Raby et al. (2007) with scrub-jays. Future developmental studies may look to adopt a version of Clayton and colleagues' paradigm to see if children's performance mirrors that of the scrub-jays.

<sup>2</sup>This approach makes the calculation of chance somewhat difficult due to the fact there is no clear chance value for transporting the tool. However, given that all of the studies we discuss coded tool transport as a binary outcome (i.e., either successful or unsuccessful), we set the chance value for tool transport at 0.5. To calculate an overall level of chance we then multiplied this value by the probability of selecting the correct tool by chance, which is simply a function of the number of target and non-target tools subjects had to choose from. The performance of each subject was assessed using a binomial probability test (see **Table 2**).

blocked. After 5 min, in which they were allowed to choose freely between the items, the apes were ushered out of the testing room and any tools that remained in the test room were removed. After a 1 h delay, the apes were allowed back into the testing room with the baited apparatus now accessible. Across 16 trials, subject's performance ranged from 13–94% (**Table 1**). However, while the performance of subjects demonstrates they are capable of returning to the test room with the tool after a significant delay, any conclusions drawn from this study are somewhat tempered by the fact the animals made their selections with the apparatus in view, raising the possibility that tool selection was cued.

Independently of Mulcahy and Call (2006) and Osvath and Osvath (2008) trained two chimpanzees and one orangutan to use a plastic hose to suck fruit soup from an apparatus. Once subjects had learned this, they were called individually into a selection room and given the opportunity to select one of four items, one of which was the plastic hose. After selecting an item, subjects were ushered out of the selection room and back into their enclosure. Critically, from the selection room, subjects were not able to see the apparatus or the room that it was housed in. After a 1-h delay, subjects were allowed access to the testing room. Across 14 trials, subject's performance ranged from 79–86% (**Table 1**). In a second experiment, Osvath and Osvath (2008) went one step further by presenting a grape as one of the items subjects could choose. Impressively, the performance of all three subjects was comparable to that of the first experiment, suggesting that they were able to forgo the immediate reward in order to acquire a better reward in the future.

In contrast to Mulcahy and Call (2006) and Osvath and Osvath (2008), Dufour and Sterck (2008) found little evidence chimpanzees were capable of passing the spoon test. Dufour and Sterck (2008) trained 10 chimpanzees to use a hook to obtain a bottle of juice placed outside of their individual feeding compartment. During testing, four categories of objects (hooks, straws, branches, and sticks) were made available in a compartment that all subjects could access. After 10 min, the items that remained in the compartment were removed. Testing occurred 1 h later, with subjects invited to enter their feeding compartment. Surprisingly, across


#### **Table 1 |The trial-by-trial performance of several primate species.**

<sup>1</sup>Dekleva et al. (2012), Experiment 1a; <sup>2</sup>Dufour and Sterck (2008), Experiment 5; <sup>3</sup>Osvath and Osvath (2008), Experiment 1; <sup>4</sup>Mulcahy and Call (2006), Experiment 1. 0 indicates an incorrect trial, 1 indicates a correct trial, and – indicates no trial.

17 trials, only three subjects performed at least one trial correctly (**Table 1**).

It is important to note that Dufour and Sterck's (2008) study, although including a standard version of the spoon test, focused largely on an exchange version of the spoon test in which subjects exchanged a token with a human, rather than used a tool, to acquire food. Consistent with the performance of their chimpanzees on the standard spoon test, the chimpanzees also performed poorly on the exchange version of the task (Dufour and Sterck, 2008). Two recent studies, however, have demonstrated that chimpanzees (Osvath and Persson, 2013) and orangutans and bonobos (Bourjade et al., 2014) can also successfully pass this task.

As a whole, the comparative studies clearly show that there is marked individual variation in the performance of several great ape species on the spoon test. The high performance of some individuals, however, suggests that, although not ubiquitous, some great apes appear capable of mental time travel (**Table 2**).

### **ONE TRIAL TO RULE THEM ALL**

Our conclusion that some great apes appear capable of mental time travel is based on using the same criteria we (Scarf et al., 2013) and others (Suddendorf et al., 2011) have used with preschool children. One issue that must be addressed, however, is the number of test trials the developmental and comparative studies employ. While developmental studies have universally tested children on only 1 trial, the comparative studies have tested great apes on between 14 and 17 trials of the same problem. Suddendorf and Corballis (2010) have argued that mental time travel can only be inferred if single trials are used due to multiple trials (a) potentially resulting in associative learning and, (b) raising the possibility subjects' performance is based


<sup>1</sup>Dekleva et al. (2012), Experiment 1a; <sup>2</sup>Dufour and Sterck (2008), Experiment 5; <sup>3</sup>Osvath and Osvath (2008), Experiment 1; <sup>4</sup>Mulcahy and Call (2006), Experiment 1. Chance (Overall) was calculated by multiplying the chance of the subject selecting the correct tool (Selection), based on the number of target and non-target tools made available, by the chance of the subject returning with the correct tool following the delay (Transport). The chance of a subject transporting the tool, given all studies coded it as a binary outcome, was set at 0.5. The performance of each subject was assessed using a binomial probability test. Given the large number of tests conducted, a modified Bonferroni correction was employed (Holm, 1979). Holm's (1979) adjustment uses the rank of each p-value (Rank) to calculate the adjusted alpha (Alpha).

on generalization rather than a memory of a specific one-time event.

Pragmatically, the single trial criterion is unrealistic. Indeed, the apes cannot be verbally informed about the events that are about to unfold and it is difficult to see how they could possibly anticipate the fact they are about to be presented with a novel test. With respect to associative learning, one could argue that, at present, the data do not accord with this account. Indeed, the marked individual variation displayed by great apes is not consistent with an associative account, which one would expect to result in much more uniform performance. Further, an associative account would predict a decrease in performance across trials rather than the increase Suddendorf and Corballis (2010) suggest. As Osvath (2010) has pointed out, if associative learning was at play, the value of selecting the tool would quickly diminish due to the fact it cannot immediately be used to attain the reward it has been associated with. Finally, in one of the few studies to test monkeys, Dekleva et al. (2012) found that not a single monkey on a single trial, of the 17 test trials each were given, was able to pass the spoon test (**Table 1**). Again, given monkeys are quite adept at associative learning, an associative account would predict that monkeys would perform at a comparable level to great apes. Of course, additional studies will need to be conducted to ensure the failure of monkeys is due to the absence of mental time travel rather than a contextual variable (Bitterman, 1964). The failure of monkeys on the spoon test, however, is consistent with research looking at the ability of monkeys to plan over shorter time scales (Beran et al., 2004; Scarf and Colombo, 2009; Scarf et al., 2011a,b, 2014).

### **CONCLUSION**

In summary, if we apply the same criterion that has been used in developmental studies of mental time travel to studies conducted with great apes, it seems we must reject Tulving's (2005) hypothesis that mental time travel is uniquely human.

### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 01 July 2014; accepted: 28 July 2014; published online: 12 August 2014. Citation: Scarf D, Smith C and Stuart M (2014) A spoon full of studies helps the comparison go down: a comparative analysis of Tulving's spoon test. Front. Psychol. 5:893. doi: 10.3389/fpsyg.2014.00893*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Scarf, Smith and Stuart. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Making progress in non-human mental time travel

## *Corina J. Logan\**

Department of Psychological and Brain Sciences, SAGE Center for the Study of the Mind, University of California Santa Barbara, Santa Barbara, CA, USA \*Correspondence: corina.logan@psych.ucsb.edu

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

#### *Reviewed by:*

Tom V. Smulders, Newcastle University, UK

**Keywords: mental time travel, episodic memory, future planning, non-human, imagination**

Humans can remember unique past events and plan for the future and they can imagine themselves at these events when they are not currently occurring, an ability often called mental time travel and thought to be distinctly human (Suddendorf and Corballis, 2007). The behavior of many non-human species indicates that they can also remember unique past events to plan for the future, however it is not known whether they actually imagine themselves outside of the present (Clayton et al., 2003).

Corballis (2013a,b) recently commented on new research on the hippocampus (a brain region involved in spatial navigation) showing that rats may be able to imagine themselves in situations other than their current one because when resting they replay particular neurological sequences indicated by the firing of "place cells" associated with familiar and novel spatial trajectories (Gupta et al., 2010). Corballis considers this possible evidence that non-humans imagine themselves in the past and future and, thus, mentally travel in time. I agree that nonhumans likely do imagine themselves in the past and future, however, it is important to note that while both humans and non-humans activate these neurological sequences to engage in goal-directed behavior (Ekstrom et al., 2003), there is no direct evidence yet that these sequences indicate that imagination or planning is involved (Gupta et al., 2012). These studies lack evidence from brain imaging studies that link imagination and planning to the neurological sequences associated with spatial trajectories.

Brain imaging studies would also help evaluate Corballis (2013a,b) and Suddendorf's (2013) claim that human mental time travel is more complex than that in non-humans because we can imagine not only locations, but also other aspects of particular scenarios including ". . . people, things, places, [and] actions" (Corballis, 2013a, p. 5). It is too early to arrive at this conclusion due to the lack of research investigating imagination in non-humans as well as the absence of nonhuman mental time travel experiments that examine behavior and neurological activity at the same time. To understand what mental processes are involved in mental time travel, we must look at what mental processes are occurring in the brain when performing behavioral experiments. Investigating the question of what nonhumans can imagine requires studying neurological activity across the whole brain, not just the hippocampus, since brain areas outside of the hippocampus are active when humans imagine other individuals, objects, and actions (e.g., Decety, 1996; Hassabis et al., 2013; Schlegel et al., 2013; see Polyn and Sederberg, 2014 for a review). Investigations of whole brain activity combined with creative experimental designs could determine whether non-humans use imagination to mentally travel in time.

### **EXAMINING WHOLE BRAIN ACTIVITY**

While the advancement of technology will open avenues for studying neurobiology at greater spatial and temporal resolutions, progress is also being made using existing technology in new ways (e.g., examining bird brain activity from real-time behavior using micro-PET scans: Marzluff et al., 2012; Cross et al., 2013). These advances have broadened our ability to test hypotheses about the complexity of non-human cognition.

Investigations of whether imagination and planning are involved in the replay of neurological sequences associated with space will benefit from a research approach that combines technologies. One such approach could use the hippocampal tetrodes that record place cell activity in conjunction with electrocorticography (ECoG) and positron emission tomography (PET) to detect which other brain regions are active during replay events. ECoG has a high temporal resolution (on the order of milliseconds) and the implanted electrodes allow the animal to behave normally (i.e., not anesthetized in a scanner; Buzsáki et al., 2012). PET has a low temporal resolution, but allows a higher resolution of active brain areas thus complimenting the lower spatial resolution of ECoG. PET also allows brain activity to be examined in the context of normal behavior because the scans detect positively charged particles that result from the metabolism of a radioactive glucose tracer. It takes several minutes for the tracer to be metabolized sufficiently to represent all of the neural activity during the time of interest, thus giving experimenters time to conduct the behavioral trials and then anesthetize and scan the subject. Indeed, a study using ECoG and PET scans found that results from the two methods were significantly in agreement and together they provided a higher resolution than each method could when used in isolation (Chandra et al., in press). Used together during mental time travel experiments, ECoG, PET, and hippocampal tetrodes could begin to illuminate whether brain areas involved in imagination and planning are active during replay events and, thus, whether non-humans imagine themselves in the past and future.

Identifying which areas in the brains of non-humans are involved in imagination and planning remains a challenge since brain structures vary across species, thus inferences cannot necessarily be based on human brain activity and anatomy (e.g., Krubitzer et al., 2011). Yet this lack of knowledge gives even more strength to the argument to examine activity from the whole brain to facilitate a solution to this problem.

### **CREATIVE EXPERIMENTAL DESIGNS**

Experiments that require the subject to replay neurological sequences associated with spatial trajectories while at rest will be useful because they should activate the brain areas involved in imagination and planning. For example, using the Gupta et al. (2010, 2012) spatial maze, each of the four feeders could have a unique color and contain food that is more or less preferred, thus establishing an order of preference for the feeders. Keeping the subject at the starting position in the maze, a photo or video of the feeder with the preferred food shown to the resting animal should evoke the neurological sequences associated with the spatial trajectory for traveling to that feeder (see Miller et al., 2013 for a similar experimental design used for humans). Using this same paradigm, experimenters could incorporate a time component by having the preferred feeders only dispense food after a delay, while the least preferred food is always available. If the preferred foods are not available when the image is shown because an insufficient amount of time has passed for them to become accessible again, then the subject should choose the least preferred and always available feeder as represented by the neurological sequence it replays. These experimental conditions should be contrasted with control conditions in which the only difference is that no imagination is required. For instance, showing the animal a novel picture (e.g., a white background). This would test whether the animal remembers *what* food is *where* and *when* it should run to a particular feeder. What, where, and when are the three components of mental time travel that are detectable by observing behavior (Clayton et al., 2003). This paradigm uses an experimental design similar to that used for western scrub-jays (Clayton et al., 2003) and adds the neurobiological component necessary to determine whether non-humans also engage imagination.

## **CONCLUSION**

To answer the question of whether nonhumans imagine themselves in the past and future it is necessary to go beyond behavioral studies and investigate behavior in conjunction with brain activity. By examining activity across the whole brain in the context of creative experimental designs that test conditions requiring imagination against controls that do not, it should be possible to determine whether they imagine themselves not only in particular places, but also in the context of other individuals, objects, and actions. It is wise to reserve judgment on the distinctness of humans until comparable data, especially on the brain activity behind the behavior, exist in non-human species.

## **ACKNOWLEDGMENTS**

I thank Margaret Tarampi and James Knierim for useful discussions; Anthony Dickinson and Tom Smulders for manuscript feedback; Dieter Lukas for discussions and manuscript feedback; and I am grateful for support from a Junior Research Fellowship at the SAGE Center for the Study of the Mind at the University of California Santa Barbara.

## **REFERENCES**


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 15 February 2014; accepted: 24 March 2014; published online: 09 April 2014.*

*Citation: Logan CJ (2014) Making progress in nonhuman mental time travel. Front. Psychol. 5:305. doi: 10.3389/fpsyg.2014.00305*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Logan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## A method for generating an illusion of backwards time travel using immersive virtual reality—an exploratory study

## *Doron Friedman1, Rodrigo Pizarro2, Keren Or-Berkers 1, Solène Neyret 2, Xueni Pan3 and Mel Slater 2,3,4\**

<sup>1</sup> Sammy Offer School of Communication, The Interdisciplinary Center IDC Herzliya, Herzliya, Israel

<sup>2</sup> Event Lab for Neuroscience and Technology, Departament de Personalitat, Facultat de Psicologia, Avaluació i Tractaments Psicològics, Universitat de Barcelona, Barcelona, Spain

<sup>4</sup> Institució Catalana de Recerca i Estudis Avançats, Barcelona, Spain

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

*Reviewed by:* Beatrice De Gelder, Louvain University, Belgium Doug Bowman, Virginia Tech, USA

### *\*Correspondence:*

Mel Slater, Event Lab for Neuroscience and Technology, Departament de Personalitat, Facultat de Psicologia, Avaluació i Tractaments Psicològics, Universitat de Barcelona, Passeig de la Vall d'Hebron 171, 08035 Barcelona, Spain

e-mail: melslater@ub.edu

We introduce a new method, based on immersive virtual reality (IVR), to give people the illusion of having traveled backwards through time to relive a sequence of events in which they can intervene and change history. The participant had played an important part in events with a tragic outcome—deaths of strangers—by having to choose between saving 5 people or 1. We consider whether the ability to go back through time, and intervene, to possibly avoid all deaths, has an impact on how the participant views such moral dilemmas, and also whether this experience leads to a re-evaluation of past unfortunate events in their own lives. We carried out an exploratory study where in the "Time Travel" condition 16 participants relived these events three times, seeing incarnations of their past selves carrying out the actions that they had previously carried out. In a "Repetition" condition another 16 participants replayed the same situation three times, without any notion of time travel. Our results suggest that those in the Time Travel condition did achieve an illusion of "time travel" provided that they also experienced an illusion of presence in the virtual environment, body ownership, and agency over the virtual body that substituted their own. Time travel produced an increase in guilt feelings about the events that had occurred, and an increase in support of utilitarian behavior as the solution to the moral dilemma. Time travel also produced an increase in implicit morality as judged by an implicit association test. The time travel illusion was associated with a reduction of regret associated with bad decisions in their own lives. The results show that when participants have a third action that they can take to solve the moral dilemma (that does not immediately involve choosing between the 1 and the 5) then they tend to take this option, even though it is useless in solving the dilemma, and actually results in the deaths of a greater number.

**Keywords: time travel, virtual reality, body ownership, trolley problem**

## **INTRODUCTION**

We introduce methodology based on immersive virtual reality (IVR) that aims to provide an experience of travel backwards in time to relive a past event, and potentially change history. We further consider how such an experience might change attitudes and views of the virtual time traveler, given an illusory experience of going to the past and undoing actions that originally led to an unfortunate outcome.

In one view of today's physics time travel at the macro level to the past is not feasible, and even if it were apparently achieved it would be travel to a parallel universe (see extensive discussions in Deutsch and Lockwood, 2009; Deutsch, 2011). Nevertheless, the idea of time travel has long been one that has captured the imagination—both travel to the future in the story of *Urashima Tarô* (8C) and H.G. Wells' *The Time Machine*, and travel to the past (e.g., *Lest Darkness Fall* by L. Sprague de Camp). Time travel is a recurring theme in cinema; a recent example that included time travel is *Looper*<sup>1</sup> , involving the curious practice of criminals in the future sending their enemies to the past in order to be executed by paid assassins there.

Travel to the past raises scientific and philosophical issues that future travel avoids: it includes the possibility of changing of history with the paradoxes that this can imply. For example, in the Grandfather paradox, a person travels to the past and kills one of their own ancestors ensuring that they would have never been born to travel back to the past and accomplish this act. The movie *Looper* illustrates one of the potential paradoxes involved in time travel, where time travelers from the future can change history by amending the past, so that their own existence vanishes—leaving the viewer to wonder how any of the events of the movie could have taken place at all. Moreover, there have been recent attempts to discover whether time travelers are walking amongst us by

<sup>3</sup> Department of Computer Science, University College London, London, UK

<sup>1</sup>http://www.sonypictures.com/movies/looper/

examining whether the Internet contains any trace of their messages (Nemiroff and Wilson, 2013). Some of the paradoxes and philosophical arguments around the concept of time travel to the past are discussed, for example, in Grey (1999); Dowe (2000).

Irrespective of whether time travel to the past is in any way possible, here we consider the question—what if it were possible? What if someone could travel back through time and experience a sequence of events, and be able to intervene in order to change history? Specifically, we simulate a sequence of events that has a tragic outcome (deaths of strangers) in which the participant unavoidably plays an important role. In our setup the participant is caught in a classic moral dilemma: if he or she does nothing then five people would die for certain; if he or she acts then five people might be saved but another would die. This is derived from the moral dilemma known as the "trolley" or "boxcar" problem where an out of control trolley or boxcar on a railway track would kill five people unless diverted, in which case it would kill one. Such problems are usually addressed empirically by questionnaires2, where respondents are required to indicate whether they would do nothing allowing the five to die or divert the trolley thus killing one. Typically 80–85% of people choose to divert the trolley sacrificing one to save five (Hauser et al., 2007). Such classic moral dilemmas as the trolley problem have been studied in moral philosophy and more recently by neuroscientists in the context of the processes involved in action in moral decision making—see Greene et al. (2001); Cushman et al. (2006). Here we have simulated a moral dilemma similar to the trolley, except that the agent of harm is a (virtual) human rather than a run-away trolley, and where it is possible to find a solution to the problem without the deaths of anyone. Our work is exploratory considering whether the ability to go back through time and intervene, to possibly avoid all deaths through finding the solution, might have an impact on how the participant responds to such moral dilemmas, and also have an impact on how past actions with unfortunate outcomes in their own lives are seen.

We use IVR to give people the illusion of having traveled back through time. IVR can create three specific types of illusion that we exploited for this purpose: presence, body ownership and agency. The first, "presence," is the illusion of being in the place depicted by the virtual environment (place illusion) and of the reality of the events taking place there (plausibility) (Sanchez-Vives and Slater, 2005; Slater, 2009). The second is the illusion of "body ownership": in IVR it is possible to endow participants with an alternate life-sized virtual body that is spatially coincident with their own real body. This substitute virtual body is seen through a wide field-of-view stereo and head-tracked head-mounted display (HMD) from first person perspective when looking directly down toward the own body and also when looking in a mirror. This can give rise to the illusion that the substitute body is the participant's own body (Petkova and Ehrsson, 2008; Slater et al., 2010). Through real-time motion capture the virtual body moves synchronously with the real body, thus giving the third illusion ("agency") (for example, as in Banakou et al., 2013) where the participant has the sensation of being the cause of the movements of the body. We aimed to create strong illusions of presence, body ownership, and agency, and using these to project a participant back through virtual time to relive and be able to intervene in a sequence of events over and over again.

If you are involved in a sequence of events and then step into a machine that takes you back to the start of those events, there are various possibilities regarding what you would perceive and how you might be able to act. Of course you should see the events that had occurred, which includes perceiving that earlier version of yourself doing whatever you originally did, together with all the consequences of those actions. However, immediately there is a problem: can you intervene in and change those events or not? If you have physical presence then you can cause events just as any other agent in the situation. This physical presence also implies that your former self should be fully aware of you. However, were this to be the case then the first time that you had experienced those events your future self should have been present, and you should have a memory of that having occurred. Alternately you may have no physical presence but just be an observer of those past events. This type of situation is discussed in Deutsch (2011) where this example is used to show that the possibility of physical intervention requires the existence of parallel universes (the Multiverse). Here, since it is virtual reality, and we are not bound by the laws of physics governing time travel, we can adopt an intermediate position. You as the time traveler can effect changes to the events, therefore changing history, but you are not visible to your past self.

We use two different conditions in our exploratory study. In both participants are embodied in a virtual body and experience a sequence of events in which they are faced with the choice of allowing five people to die or saving the 5 at the cost of 1. In the *Time Travel* condition at the end of this sequence they are transported back to the start of the events, but see and hear their previous self-representation carrying out the actions that they had carried out before. In this condition they can cause events, but the previous incarnation of the self is not aware of the current self. Subsequently once the sequence of events is played out again, possibly changed from the first time around due to new events caused by the participant, the participant is once again transported back to the start, now seeing and hearing the two previous self-incarnations. Thus, there are three trials. In the *Repetition* condition the first trial is the same as the Time Travel condition, but in the second trial the participant is simply faced with exactly the same events again, as in a video game where a "life" is lost and the game starts again. Here there is no notion of "time travel"—the representations of the previous incarnations are not shown, and the participant is free to act, in the second and third trials of course utilizing the knowledge gained from the previous trials. It should be noted that the first trials in both conditions are therefore identical and provide an IVR experience representing a version of the classic trolley-type of moral dilemma.

The goal of this exploratory study was to discover (i) whether the Time Travel condition would be more likely to result in the illusion of time travel than the Repetition Condition, (ii) the extent to which the experience of illusory time travel might influence attitudes toward morality, moral dilemmas and "bad decisions" in personal history.

<sup>2</sup>http://moral.wjh.harvard.edu

The thinking behind our approach is that at some level the brain does not distinguish between reality and virtual reality. Therefore, there would be implicit learning that the past is mutable. The illusion that the past can be changed might have important consequences for present day attitudes and beliefs including implications for psychotherapy.

### **METHODS**

### **THE SCENARIO**

The specific scenario we created was an art gallery on two levels (ground and upper) (**Figure 1**). This is based on earlier work on action in response to a moral dilemma (Pan and Slater, 2011). In this situation the participant learns to operate a virtual elevator that takes (virtual human) visitors to the upper level or down from upper level to the ground level at their request, and also learns to operate an alarm that freezes the elevator in place and makes an alarm sound. After six visitors have entered the gallery there are five people browsing the paintings upstairs and one person downstairs. A seventh person enters the gallery and asks to be taken to the upper level. Upon arrival at the upper level, and while still on the elevator platform, he immediately takes a gun out of his pocket and starts shooting at the five people there. The participant (elevator operator) then has a choice to make: either leave the gunman to possibly kill all five people or send the elevator down, where the one person might be killed instead. The participant has also previously learned that pressing an alarm button will immediately freeze the elevator in place (but this is no use at this moment since the gunman is already shooting). After a few seconds of this mayhem the scene dissolves and the participant is back at the start of the whole sequence of events. Note that one potential solution to the dilemma is to trap the gunman on the elevator half way between the two floors by pressing the

**FIGURE 1 | The gallery room scenario. (A)** A participant wearing the head-mounted display and the motion capture suit. **(B)** The gallery with three visitors at the upper level, 1 on the ground level, and 2 waiting to be taken up. The workbench is shown with the up and down elevator control buttons and the red alarm button. The mirror reflects the virtual body of the participant (here female), which can also be seen from behind. The participants saw the environment from the first person perspective of the body, and the virtual body was coincident in space with their real body. **(C)** The gunman shoots at the five people on the upper

level. **(D)** The time travel (2nd time around)—where the participant is embodied in the rightmost body by the workbench, and sees his previous self-carrying out the actions from the 1st time around. **(E)** The time travel 3rd time around, where the gunman is shooting. The participant is in the leftmost body behind the workbench, and the two earlier clones are to his right. **(F)** A close up of the embodiment illustrating visuomotor synchrony. Here the participant sees through the eyes of the virtual body and as she raises her arms the arms of the virtual body raise synchronously, and this is also seen in the mirror reflection.

alarm button. However, the first time that participants experience this sequence of events this possibility is not useful, since they do not know that the seventh person is a gunman until he is already shooting. Therefore, if the goal were to save the five visitors on the upper level then freezing the elevator in place would be exactly the wrong thing to do since it would leave the gunman in the position to shoot all of them.

In the Repetition condition exactly the same sequence of events will then reoccur. In this condition the participant can choose how to act now with knowledge about what is likely to happen, and can of course change his or her actions compared to the first "life." In the Time Travel condition, however, each participant sees and hears the earlier incarnation of him or herself performing what they had actually done in the first round. We refer to this virtual human character, that has the identical virtual body as the participant, and that re-enacts the first round actions of the participant as P1, and the actual human participant as P. P1 does the same actions as the participant had actually done and is unaware of P. If P does not take any action (i.e., pressing buttons) the sequence of events will unfold the same way as in the first round, otherwise any action can result in a different sequence of events and outcome.

Eventually in this second round there may be five people on the upper level and one on the ground level (depending on the actions of the participant) as before, and the gunman enters again. The participant might infer that the same is likely to happen as in the previous round and therefore try to take steps to stop this. In particular, P might realize the solution, and try to trap the gunman between levels. In any event, whatever happens, once again after the shooting takes place (or if the gunman is trapped) the scene dissolves and the participant is back at the start. For those in the Repetition group this is the start of the third identical trial. However, those in the Time Travel condition now see their two previous selves (P1 and P2) doing what they had done before. Each participant finally experiences the three trials, and then the virtual reality part of the experiment is terminated.

### **THE ABSTRACT REPRESENTATION OF TIME TRAVEL**

Our approach to time travel is designed to provide participants with the illusion of time travel. We will show how our approach is different from typical simulations or video games, in which the player can repeat a game multiple times. In this section we explain our approach and illustrate it in relation to the gallery scenario described above.

There are two levels of abstraction: the logic layer and the VR layer. The logic layer is an abstraction handled by an automated reasoning engine, which deals with the unfolding of the narrative, and the VR layer has a much richer description of the virtual world, which supports the immersive experience of virtual time travel.

In the art gallery scenario the virtual environment includes the following objects: visitors in the gallery, the participant, the elevator, a workbench with a mirror, up and down buttons to control the elevator and an alarm button to freeze the elevator. The participant is instructed to operate the elevator and follow the visitors' requests as described earlier, and at the end of the scenario, after the shooting, is transported back in time (or to replay the scenario again in the Repetition condition). Here we concentrate on the Time Travel condition since the repetition condition only involves running the same program three times. We refer to the original scenario as the 1st time around, and after the time travel to the past, the 2nd time around, and then the 3rd time around. The destination time was always fixed—time travel back to the beginning of the scenario. However, the system does support time travel to arbitrary times in the past.

Following time travel to the past the participant is embodied in a new virtual body (looking identical to the first one). We will refer to the virtual bodies representing the participant in the past as clones. The participant sees and hears him- or herself from behind and to the right side of the previous clone (**Figure 1D**, Supplementary Movie S1). Everything takes place exactly as it did in the 1st time around, until the participant takes actions that modify the story line. Our system allows multiple time travels and thus supports a large number of co-existing clones, and in this case two past selves during the 3rd time around.

The crux of the time travel approach is the reasoning engine, which is able to track and maintain causality. Causality raises many scientific and philosophical questions. In this paper we adopt the common sense notion of causality, which has also been the subject matter of empirical studies (e.g., Michotte, 1963). In our case the only way for the participant to "change the past" is by controlling the elevator (directly by pushing its buttons or indirectly by activating and deactivating the alarm). The reasoning engine can deduce, for example, that if the gunman is not on the upper floor then he cannot shoot the visitors in the upper floor, and that if the visitors are not killed then they remain alive.

Our system tracks causality by maintaining an abstract representation called *history*, which consists of a sequence of states and a sequence of actions. At each moment the virtual world is in some well-defined state, and objects perform actions that transfer the world from one state to another. The reasoning engine attempts to keep the history of the 2nd time around as close as possible to the 1st time around, but it is also takes into account the new actions introduced by the participant in the 2nd time around, together with their causal effects.

All objects are defined by their state. For example, a visitor is defined by its location and whether it is alive or dead, and the elevator's state describes whether it is in the ground floor, upper floor, or traveling in between floors. An important challenge is to include in the reasoning engine only those state variables and actions that are necessary for the reasoning process. There are many details that need to be modeled in the VR but for tractability we can assume they do not affect the narrative, and thus are not included in the reasoning engine. For example, in the abstract specification it is not important exactly where the visitors are, so in terms of the reasoning engine their location is abstracted to being either outside the gallery, on the ground level, or on the upper level. In the VR layer exact trajectories in space and time are maintained.

We distinguish between three types of entities, in terms of causality: (i) the participant, (ii) physical objects, and (iii) agents. The latter category includes, in our case, both the visitors and the previous clones of the participant. The human participant is always assumed to have "free will"; the system cannot dictate his actions, and cannot override his previous actions. Second, there are simulated objects; in our case, the elevator and the alarm. We assume that physical objects follow well-defined deterministic rules, and therefore our engine applies standard simulation techniques to these objects.

The main difference between our approach and other simulations is the way we model human behavior; this is the third class of entity—the agents. For these entities the algorithm tries to "replay" their behavior as much as possible, i.e., simulated humans will act during the 2nd time around exactly as they did the 1st time around, unless this involves a logical contradiction. This part of our method is again consistent with our common sense view; even if we have approximate models of human behavior, sometimes we regard people's actions as idiosyncratic and arbitrary. It is exactly this seemingly unpredictable behavior that makes narratives compelling, and this is what we want the time travel approach to maintain. In our case such time travel modeling is restricted to humans, but in general there can be other low probability events that we want to be repeated the 2nd time around in the same way and at the same moment that they happened in the 1st time around. In our case we have opted to model the previous clones of the participant as a type of agent, just like the visitors.

Technically, all actions have *preconditions* and *postconditions*. Preconditions define the states that specific objects should be in, so that the action can be performed. Post-conditions define the specific states that specific objects should be in after the action is performed. A major element of the time travel approach is that the 2nd time around the system tries to repeat the history exactly as it took place the 1st time around, but if some preconditions do not hold for specific actions then these actions do not take place. In some cases the reasoning engine replaces these actions by similar actions, and in other cases these actions are avoided altogether.

During the 1st time around the system plays events according to a predetermined script and records the participant's actions. The reasoning engine is only activated the 2nd time around, or, in general, after the first occurrence of time travel to the past. From that point onwards the simulation engine operates differently for three types of actions: (i) new actions taken by the participant, (ii) actions performed by simulated physical objects, and (iii) actions that were taken by agents and need to be repeated following what happened the 1st time around.

If the participant takes an action it is always executed—if the participant was able to take this action in the VR then it cannot have been logically invalid. The action is also recorded in the history of the 2nd time around; this is necessary for further time travels (in our case there is also 3rd time around). A second category consists of those actions taken by simulated physical objects. In our case this applies to the elevator and the alarm they are simulated in the 2nd time around regardless of the 1st time around. If the participant does not intervene then the alarm and the elevator would behave exactly as they did in the 1st time around. Also, if we would have attempted to replay the elevator's actions in the same way that we model agents then any slight divergence of the 2nd time around from the 1st time around might result in incoherent behavior. This discrepancy in the way we model physical objects and human agents is in accordance with common sense and everyday psychology: as people we pay much attention to human behavior and its timing, but we do not pay as much attention to automated objects, as long as they behave consistently.

The third category of actions consists of those that were taken by agents the 1st time around and now need to be repeated. Following time travel the virtual time is reset to the beginning of the 1st time around. As time passes (both virtual and physical time), whenever the virtual time coincides with the virtual start time of a recorded action, the reasoning engine checks the preconditions of this action. If these are satisfied then the action takes place, exactly as it did the 1st time around. However, if any of the preconditions are violated, the action does not take place. Instead, the reasoning engine tries to automatically replace it by a similar action, as explained below. If no replacement is possible, the action is ignored.

In our scenario there are only two types of actions that can be replaced by other actions. The main automated replacement is for the shooting action by the gunman. The 1st time around the gunman shoots the five visitors in the upper floor, let us label them V1, V2, V3, V4, and V5. The 2nd time around the gunman tries to shoot the same number of visitors at the same (virtual) time. If the 2nd time around the gunman is again at the upper floor then the shooting takes place as it did in the 1st time around. However, if the gunman is now on the ground floor the reasoning engine tries to match new targets for the shooting actions. The result is that the gunman shoots the single visitor that is on the ground floor, and even the participant. Since there is only one visitor in the ground floor, V6, the reasoning engine replaces V1 by V6 and V2 by the participant. The other visitors, V3–V5, cannot be replaced, so these actions are omitted from the history.

A similar action correction takes place for the action "enterfloor," which takes place anytime that any of the visitors (or the gunman) exits the elevator. If the engine reaches a time that the visitor has to exit the elevator, that visitor is in the elevator, and the elevator is stationary at one of the floors, then the visitor exits the elevator, regardless of whether it is the same floor as the 1st time around.

There are two "solutions" that the participant can reach in order to prevent any shooting. The first solution is to let the gunman enter the elevator but press the alarm button before the elevator reaches the upper floor. The second solution is to avoid sending the visitors upstairs, but to send the gunman upstairs. Quite a few participants were able to arrive at the first solution (see Results), but none realized the second solution.

An additional solution would be to keep the gunman on the ground floor. However, in this case the gunman would always shoot at least one visitor—the one that was preprogrammed to remain in the ground floor—and possibly the participant.

We provide this narrative as a concrete example illustrating the reasoning engine in action. Consider the following abstract script fragment, which takes place after the gunman enters the gallery, 1st time around (G is the gunman, P is the participant, V1,...,V5 are the upper floor visitors):


```
P pushes elevator up
Elevator starts moving up
Elevator arrives upper floor
G shoots V1, V2, V3, V4, V5
P pushes alarm
alarm on
P travels back in time
```
Second-time around the participant tries to avoid the shooting by turning on the alarm, just after his previous clone (now P1) sent the elevator up. The new action and state are shown in bold.

```
G enters gallery
G enters elevator
P1 pushes elevator up
P pushes alarm
alarm on
Elevator starts moving up
Elevator arrives upper floor
G shoots V6
P1 pushes alarm
alarm off
```
As a result the elevator is blocked and does not go to the upper floor; this is marked in the two actions shown as crossed out: they happened 1st time around but not 2nd time around. The reasoning engine then replaces the shooting of the visitors in the upper floor by another action—shooting the visitor on the ground floor—this replaced action is shown as underlined. Finally, note that the alarm object is simulated, so now after the shooting it is actually turned off by the clone P1 rather than being turned on as the 1st time around (shown in italic). This is because the alarm is a toggle (pressing it will change its state to the opposite state) and so since the participant had turned it on, when the clone presses the button again it will be turned off. Here actions are preserved rather than outcomes.

In this explanation we have concentrated on the 1st time and 2nd times around but our system allows multiple time travels. Whenever history has changed by time travel it now becomes the relevant history for the next time travel. That is, whenever we write 1st time around and 2nd time around these could be replaced by (*n-*1)th time around and *n*th time around, respectively. Or in other words: whenever history "changes" the original history is discarded and the new history becomes the frame of reference.

An important part of our method is that the reasoning engine is integrated with the IVR system. Where the reasoning engine decides, for example, "the gunman shoots at visitor 1" this has to be transformed into actual animations of the virtual human characters, who are in a particular place in a relatively complex scenario, with a certain body size, a certain distance between them, and so on. Describing how the VR layer is combined with the reasoning engine is beyond the scope of this paper, and will be discussed in later work.

### **THE EXPERIMENT**

### **PARTICIPANT RECRUITMENT**

Thirty-eight participants were recruited for the experiment. Each was assigned on arrival and alternately to one of the two groups (Time Travel or Repetition) by order of their appearance at our laboratory. Each participant was asked to attend the laboratory on two occasions separated by approximately a week. The sample had an equal number of males and females in each of the two groups. However, the results of six participants were discarded (four due to technical problems and two because they failed to attend the second session of the experiment). Therefore, the final sample size was 32, 16 in each group, with an even gender balance. There were no differences between the groups with respect to mean age, prior use of virtual reality, gaming experience, etc. (Supplementary Material 1).

The study was approved by the Bioethics Committee of the University of Barcelona. All participants were given basic information about the experiment (the real purpose of the study was not revealed prior to completion) and signed an informed consent form when they agreed to take part of each phase of the study. They were paid 5C for participating in the first session of the experiment, and 5C more when they came back for the second part.

### **EQUIPMENT**

The virtual environment was implemented in Unity3D and delivered visually through a wide field-of-view stereo HMD the NVIS nVisor SX1113 (**Figure 1A**). This has dual SXGA displays with 76◦H × 64◦V degrees field of view (FOV) per eye, totaling a wide field-of-view of 102◦ horizontal and 64◦ vertical, with a resolution of 1280 × 1024 per eye displayed at 60 Hz. Head tracking was performed by a 6-DOF Intersense IS-900 device. Participants wore Asus HS-1000W earphones4over the HMD.

Tactile feedback was provided based on an Arduino board connected to the computer via USB controlling two small vibrator devices. The vibrator devices were placed in the palmar areas of the middle fingers of each hand to give vibrotactile feedback when the participant touched the buttons that control the elevator in the virtual environment.

Participants in the art gallery were endowed with a gendermatched human virtual body. This moved in real time with the movements of the participant. In order to achieve this we used an Xsens body tracking suit for motion capture and MVN Studio software. Hence movements of the virtual body were mapped in real-time from the motion capture of the participants' real movements.

Based on results from previous papers (Banakou et al., 2013; Peck et al., 2013; Kokkinara and Slater, 2014) we expected high scores on a questionnaire that assessed the illusions of body ownership (the virtual body perceptually experienced as the own body) and agency (the sensation of causing the movements of the virtual body). In this experimental setup participants would have both visual-motor synchrony (through the motion capture) and some visual-tactile synchrony (through the vibrotactile stimulation on the palms of the hands whenever they touched the buttons to control the elevator).

<sup>3</sup>http://www.nvisinc.com/product/products.html?reqview=spec&fid=a0QG00 00009NGzCMAW&fname=nVisor%20SX111

<sup>4</sup>http://www.asus.com/Multimedia/HS1000W/

### **PROCEDURES**

On their first visit to the laboratory participants signed an informed consent form, and completed a demographic questionnaire (giving information about their age, work, and so on) and two implicit association tests (see Section "Implicit Association Test for Morality" and Supplementary Material 4). They then entered the virtual environment using the headtracked HMD and earphones. They heard pre-recorded instructions through the earphones to look around and describe the environment. They first saw a simple training environment, and then the art gallery, with some virtual visitors arriving and going to the upper level or staying on the ground floor looking at the paintings. In this experience they were embodied in a virtual body as described earlier, but did not wear the motion capture equipment, so they were asked to sit in a specific posture, corresponding to that of their virtual body. After exiting the virtual environment they were asked to secretly rate three past decisions in their lives that they regretted and the information was placed in a sealed envelope unseen by the experimenters. They were then asked to read a short passage about the meaning of time travel. The aim of this was to instill the idea that when past history is changed it means that the original history actually never happened (Supplementary Material 5). Finally the participants were paid for their attendance at the first session. There were two experimenters present throughout. Procedures are illustrated in the Supplementary Movie (this is in three parts only for reasons of space).

Approximately 1 week later they returned to the laboratory, and during this time they experienced the full scenario described in Section "The Scenario," according to the condition to which they had been assigned. Prior to the start of the scenario they first learned how to control the elevator to take visitors to the upper floor, and to bring the elevator down again. They controlled the elevator by pressing up and down buttons on a virtual workbench in front of them. They also learned about the red alarm button on the workbench, and that it would freeze the elevator and emit an alarm sound. They learned that the button was a toggle that would switch on the alarm, or switch it off if the alarm was on.

In this second week's exposure they were standing with full body tracking, saw their virtual body by looking at it directly from first person perspective, and also in a mirror that was part of the workbench. Since they were wearing the motion capture suit there was visual-motor synchrony between their movements and that of their virtual body. When they touched the up, down or alarm buttons they would see their virtual body move accordingly and feel a corresponding vibrotactile sensation on the palm of the active hand.

It is important to realize that in subsequent replays the participants not only saw their previous incarnations carrying out their former actions but also heard themselves speak as they spoke before. For example, the 1st time around one of the visitors always asked the participant for the time. Thus, the subsequent times around the participants could hear their own voice replayed.

A particular trial in both the Time Travel and Repetition conditions was terminated either 7 s after the last shooting and when the gunman did not have a possibility of shooting again (i.e., the elevator was not moving and there were no visitors alive on that level), or when the gunman had been trapped inside the elevator for 7 s. Then the operator would trigger the next trial, or at the end of the third trial terminate the experiment.

At the end of their three experiences the participants took off the HMD and they completed the same two IATs as in their first visit. They then completed a questionnaire concerned with the illusions of body ownership, agency, presence, the illusion of time travel, and other aspects of their experience. They were then asked to think about their three bad decision ratings that they had made the previous week, asked to rate these decisions again, with the sealed envelope available to them. The experimenter was able to record the score from the previous week and the new score, though at no time knew what the decisions were about. After this the participants were interviewed and debriefed, removed all the equipment, and were paid.

The full procedures are described in detail in Supplementary Material 2.

## **RESPONSE VARIABLES**

### *Body ownership, agency, and presence*

Since this is an exploratory study intended to introduce how IVR can be used to generate an illusion of time travel, here we only report the variables that ultimately proved useful in our exploratory statistical model of the results. Details of the variables measured are given in Supplementary Material 3.

The perceptual illusion of ownership with respect to the virtual body seen from first person perspective that substituted the real body was assessed with four questions in the questionnaire administered after the three scenario trials. Each question was in the form of a statement that was rated on a 1–7 Likert scale where 1 represented "strongly disagree," and 7 "strongly agree." The questions are shown in the Body ownership section of **Table 1**. These questions were based on previously published work (e.g., Banakou et al., 2013).

In order to obtain one overall score a factor analysis was carried out on the four variables (*mirror*, *down*, *other*, *mybody*) using principle components factors (Stata 135 ). This resulted in one factor with the best fit to the original data with the smallest uniqueness values per variable, and explaining 76% of the total variance. **Table 2** shows the factor loadings and uniqueness values. From the factor analysis a combined score (*Ownership*) was derived using the regression scoring method. The factor loadings in **Table 2** are equivalent to the Pearson correlations between *Ownership* and the original four variables.

Agency was assessed with the *agency* question. The main point of this was to test the adequacy of the real-time motion capture and display of the virtual body. Whereas due to individual differences there could be variations amongst participants in relation to body ownership, we did not expect much variation with respect to agency since it was a factual statement that the body did (or did not) move according to the movements of the participants. In fact 28 out of the 32 participants rated the agency question with a score of 6 or 7, and the remaining 4 with a rating of 5. We will therefore not

<sup>5</sup>http://www.stata.com



All responses on a 1–7 scale with 1 meaning most disagreement with the statement, and 7 most agreement.

**Table 2 | Factor analysis for body ownership questions.**


refer again to this variable, it is simply an indication of system performance.

The illusion of presence ("place illusion" and "plausibility") was assessed with the questions shown in **Table 1**. Again these were taken from previous papers—see Box 2 of Sanchez-Vives and Slater (2005). The "copresence" question was included to assess how participants assessed the virtual visitors. The *copresence* scores are uncorrelated with both *placeillusion* and *plausibility*, but Spearman's rho = 0.39 (*P* = 0.03) between *placeillusion* and *plausibility*.

**Table 1** also includes questions about "guilt" and whether participants felt that they "tried their best," that proved useful in the analysis. These were not taken from any existing source.

### *Implicit association test for morality*

There were two IAT results from each of the two visits to the laboratory. Both are described in detail in Supplementary Material 4. One was concerned with feelings of guilt based on Xu et al. (2012). However, the Guilt IAT taken after the second visit VR experience resulted in 5 missing values due to procedural or participant errors and so could not be used. The second was based on Perugini and Leone (2009) concerned with the moral behavior of the participant. This IAT has been shown to correlate well with actual moral behavior. There were no missing values in these data. We refer to the two moral IAT scores as *PreIAT* and *PostIAT* for the scores in the first and second week respectively (the second of course taken after the VR experience).

### *Discomfort about 3 past decisions*

The three bad decisions rated by the participants in both the first week and after the end of their experience in the second week were each rated on a 1–100 scale, representing their degree of regret about those decisions (100 the greatest regret). We calculated the mean of the three decision scores each week. The corresponding variables are *PreRegret* for the mean score in the first week, and *PostRegret* for the mean score at the end of the VR experience in the second week.

### *Moral choice scenarios*

At the end of the questionnaire the participants read five moral dilemma scenarios—three were based on the boxcar problem (equivalent to the trolley problem), and two were based on the actual dilemma in the gallery (Supplementary Material 3). For each one they were asked whether or not they would "push the switch"—in each case resulting in the deaths of 5 people or 1. We restrict attention to the boxcar problem of which there were three variants each with a yes/no answer.


a footbridge where he and the observer are standing then the 5 will be saved but the man will be killed. Question: would you push the man off the footbridge onto the track?

From these three we construct a new variable representing the number of scenarios out of the three in which 1 will be saved rather than the 5 [pushing the switch in (a), not pushing it in (b) and not pushing the man in (c)]. We refer to this variable as *save1*, which ranges from 0 to 3. Also we single out the footbridge question since this has a different element involving actively killing 1 by pushing him to save 5. Hence we will also use *footbridge*, which is a binary variable, in place of *save1*.

### *The illusion of time travel*

This was assessed with the question: "The overall experience was more like..." where the response was on a 1 to 7 scale, 1 meaning "Replaying a video game" and 7 meaning "Experiencing time travel." We refer to the corresponding ordinal variable as *timetravel*.

### *Hypotheses and statistical methods*

This exploratory study was motivated by the idea that a strong sense of presence and body ownership in a virtual environment, together with a scenario where participants witnessed their own past actions carried out by a virtual human in which they had previously been embodied, would lead to an illusion of having traveled in time. A second level hypothesis was that the illusion of traveling in time might influence present day attitudes—in particular possibly lessening negative feelings associated with past decisions, giving a different perspective on past actions—including those associated with the experienced scenario.

This being a new area of study we collected a large amount of data (Supplementary Material 3). We focus here only on the core ideas.

For analysis we used statistical models appropriate to the type of data. *Condition* is a binary factor (Repetition = 0, Time Travel = 1), similarly with Gender (Male = 0, Female = 1). We treat *Ownership* as a continuous latent variable representing the subjective illusion of ownership of the virtual body, and positively associated with *mirror*, *down*, *mybody*, and negatively associated with *other* (**Tables 1**, **2**).

*PostIAT* and *PreIAT* are treated as continuous variables, where greater values are associated with more moral behavior (Supplementary Material 4). *PreRegret* and *PostRegret* are likewise treated as continuous variables with greater values indicating greater discomfort about the three past decisions. Our model fitting strategy allows the "Post" variables to be influenced by the "Pre" variables, in other words the "Pre" variables appear on the right hand sides of the equations defining the model.

*Save1* is a variable representing a count out of a maximum of 3. We treat this as a binomial random variable, where greater values indicate a greater propensity to save the 1 (non-utilitarian) rather than the 5 (utilitarian). *Footbridge* is treated as a binary (Bernoulli) random variable, 1 indicating a "yes" answer to pushing the man to stop the boxcar (utilitarian), and 0 "no" (non-utilitarian).

The remaining variables are all questionnaire responses measured on an ordinal scale from 1 to 7.

Stata 13 was used for all statistical analysis. Since we have a multilevel hypothesis (condition, presence and body ownership influence time travel, and time travel in turn influences various other responses) we use path analysis to bring all relationships of interest into one statistical model. Stata 13 has the facility for path models that include factor variables and handles distributions other than normal (the function "gsem"). Path analysis was used since it supports the simultaneous evaluation of multiple stochastic equations—in other words it is not restricted to a single response variable as in the case of the general linear model (regression, ANOVA). Given the specification of any model (i.e., set of stochastic equations) the total covariance matrix is estimated, typically through maximum likelihood estimation. Path analysis was first described in the 1920's (Wright, 1921) with an up to date explanation in, for example, (Kaplan, 2009). We have used this several times before to unravel complex relationships in the context of body ownership studies (e.g., Kilteni et al., 2012; Llobera et al., 2013; Pomes and Slater, 2013; Steptoe et al., 2013). Given the nature of our study, path analysis has been used as an exploratory rather than a confirmatory tool.

The path model specifies each variable according to its type as discussed above. In particular the equations specifying continuous variables are assumed to be normal linear models, the binary, and count variables are treated as binomial-logistic models, and the ordinal questionnaire data as ordered logistic models. Throughout we have used robust standard errors of the coefficient estimates since these allow departure from the strict distributional assumptions underlying the statistical inference models. Moreover, we also relax the assumption of independence between observations, allowing for the fact that there may be less variation in responses within each gender group than between them—in other words we use robust standard errors allowing for clustering on gender.

## **RESULTS**

### **PRESENCE AND BODY OWNERSHIP**

**Figure 2** shows box plots for the presence, body ownership and agency questions (**Table 1**). It can be seen that the subjective levels of presence were high (median of 6 and interquartile range 5–7 for both *placeillusion* and *plausibility*) and slightly lower (median 5) and greater variability (IQR 3 to 6) for *copresence*. The body ownership illusion scores were relatively high (median 5 for each of the positive questions, and 2 for the control question *other*) well in line with previous studies (e.g., Banakou et al., 2013). The *agency* scores were high as discussed earlier.

Overall the goal of producing a system that could lead most participants to a high level of presence, body ownership and agency, was achieved, though of course with variation due to individual differences. There were no overall differences on any of these variables due to condition (Repetition, Time Travel) or gender.

### **HOW PARTICIPANTS ADDRESSED THE MORAL DILEMMA**

As noted earlier in the first round all participants faced the moral dilemma of doing nothing, in which case 5 would die, or sending

the elevator down potentially to endanger 1. There were possible solutions that would have avoided all deaths, but this could not have been known to any of the participants the first time around since they did not know at this stage that the seventh visitor was a gunman until he started shooting.

The one difference with the classical moral dilemma (such as the trolley or boxcar) was that there was a third, albeit useless, action that participants could take once the shooting started the 1st time around—press the alarm. In fact 28 out of 30 subjects selected this as their first action, and 2 selected the Down button (to save the 5) (data on 2 participants was not available). For their second action 12/30 (40%) of participants chose to press the Down button. For their third action 6/30 pressed the Down button. By the end of the sequence in only 1 case was the elevator down though, due to attempts by participants to trap the gunman and thus moving the elevator up and down, and the remaining times it was up (64%) or between floors (32%).

The 1st round resulted in 24/30 (80%) of cases where 5 visitors were shot. The mean and standard deviation of the number shot is 4.8 ± 0.81.

During the second round participants carried out almost double the number of actions in the Time Travel condition compared to Repetition (**Table 3**), and the difference is significant (Wilcoxon rank-sum test, *P* = 0.012). This difference is due to the Time Travel participants having to cope with the 1st round actions taken by their earlier self. In this condition the numbers shot were almost the same between the two conditions (**Table 4**).

The third time around the number of actions in the Time Travel condition reduced to be not much more than the Repetition. This is because participants who had found a solution the second time round could just let this play out again. The numbers shot also decreased again (**Table 4**).

We consider these results in the Discussion.

### **Table 3 | Mean and standard errors of numbers of actions by condition.**


**Table 4 | Mean and standard errors of numbers shot by condition.**


### **PATH ANALYSIS**

**Figure 3** shows the path diagram with statistics in **Table 5**. The path diagram was derived from the hypotheses (Sections "Hypotheses" and "Statistical Methods"). Paths that were not significant (for example, *placeillusion* to *timetravel*) have not been included. Where there is a significant interaction term (e.g., *condition*∗*timetravel*) then following convention the main effects are included even if not significant. **Table 5** gives the complete details about the path analysis. We consider each of the response variables in turn.

The variation in responses to the *timetravel* question is not explained by *Condition* alone. However, the path analysis shows that overall it is influenced by *plausibility*, *copresence* and the interaction between *Ownership* and *Condition*. The variable *timetravel* is positively associated with *plausibility*, *copresence* and in the Time Travel condition is positively associated with *Ownership*. However, there is no association with *Ownership* in the Repetition condition.

*Timetravel* is an endogenous variable in this model, and therefore can be entirely predicted within the model. The correlation between the fitted values of the linear predictor for *timetravel* and the observed values shows a good fit of the model to the data: Spearman's rho = 0.53 (*P* = 0.002).

Taking into account *PreRegret*, the *PostRegret* variable is positively influenced by the Time Travel condition, but the greater the *plausibility* the lower the value of *PostRegret* only in the Time Travel condition. In other words the greater illusion of reality the lower the rating of regret, provided that this was in the Time Travel Condition. The Spearman correlation of the fitted values of the linear predictor and the observed values is 0.60, *P* = 0.0003.

Taking into account the *PreIAT* score the *PostIAT* is negatively associated with the *timetravel* illusion in the Repetition condition, and positively in the Time Travel condition (i.e., there is a significant interaction effect between Condition and *timetravel*). In other words in the Time Travel condition greater levels of the illusion lead to an increase in implicit categorization of the self as moral. (Spearman's rho = 0.52, *P* = 0.002, for the correlation between fitted and observed values, as above).

The feeling of *guilt* is positively associated with the time travel illusion, but only in the Time Travel condition (**Figures 3**, **4**). *Guilt* is an endogenous variable that can be predicted entirely within the model. The fitted values of the linear predictor for *guilt* are well correlated with the observed values (Spearman's rho = 0.62, *P* = 0.002).

The results for *triedmybest* suggested that overall participants did try to save the visitors: this has a median score of 5 (IQR = 2). Moreover it is related to guilt. **Figure 5** suggests that in the Time Travel condition the more that participants felt that they had tried their best the lower their guilt. An ordered logistic regression of *guilt* on Condition, *timetravel* and *triedmybest* allowing for interactions between these two variables and Condition supports this (**Table 6**). This shows that in the Repetition condition *triedmybest* is positively associated with *guilt*, but through the interaction between Condition and *triedmybest*, in the Time Travel condition the association is negative (coefficient = −1.1, *P* < 0.0005). Also in the Time Travel condition *timetravel* is positively associated with *guilt* (coefficient = 1.0, *P* < 0.0005). We show the coefficients since they are almost the same magnitude but opposite in sign. This means for example, that when in the Time Travel condition, participants have a strong subjective illusion of time travel, and a strong belief that they had tried their best then these two effects cancel out.

The variable *save1* represents the propensity to save the 1 instead of the 5. This was negatively associated with *timetravel*. This variable is also endogenous and can be predicted from the model. The fitted values from the model correlate well with the observed values (Spearman's rho = 0.41, *P* = 0.02). Overall the stronger the illusion the more the tendency toward a utilitarian solution but also the greater the guilt.

## **DISCUSSION**

The fundamental contribution of this paper has been to introduce a new method for the induction and exploration of the consequences of a time travel illusion using IVR. The application of the method has tackled a number of broad questions and the results point to some possible answers. However, our attempt to measure the "time travel" illusion was limited to a single question in the questionnaire, the reason being that since this is a new illusion it was not obvious which other questions to ask. A task of future work will be to improve the subjective measure, partially based on analysis of the interviews of participants after their experiences. However, notwithstanding this limitation the findings suggest a set of hypotheses for future work, each of which would ideally require a specific focused experimental study.

First, we have considered whether it is possible to induce such an illusion, and if so some of the consequences. Although the manipulation (Repetition or Time Travel) did not by itself influence the subjective illusion of time travel, it did so in conjunction with the illusion of body ownership, and plausibility, including the extent to which the virtual visitors were experienced as if they were real. Since this is an exploratory study we summarize this finding as a hypothesis: If in IVR with embodiment in a virtual body participants experience a sequence of events, and then are involved in those same events over again where they can also witness their past actions in the same virtual body that they previously had embodied, then they will experience this as time travel provided that they also have a strong sense of body ownership over that virtual body. Place illusion and plausibility will add to that illusion. This is premised on a strong sense of agency over the virtual body. Hence a future experiment would be designed to maximize the probability of high subjective body ownership,


In the first column the dependent variables are shown in bold. Condition has Repetition = 0, Time Travel = 1. Constant refers to the intercept term of the linear predictor of each model equation. \*Refers to an interaction term. Standard Errors are robust and allow for non-independence (clustered on gender). P = 0.000 means P < 0.0005.

**Table 6 | Ordered logistic regression for** *guilt***,** *n* **= 32.**


Condition has Repetition = 0, Time Travel = 1. Constant refers to the intercept term of the linear predictor. \*Refers to an interaction term. Standard Errors are robust and allow for non-independence (clustered on gender). P = 0.000 means P < 0.0005.

using multiple multisensory techniques—exploiting first person perspective, visuomotor synchrony, and visuotactile synchrony, discussed in Kokkinara and Slater (2014). Moreover there would also need to be an explicit "non-embodiment" condition as a control group with an attempt to minimize body ownership.

Our second hypothesis, and suggested future experimental study, is that the experience of such time travel may lead to participants implicitly accepting the notion that the past is mutable. This may lead to a re-evaluation of some of their own past actions that had unfortunate consequences and lessen the negative affect associated with these.

The moral IAT test had as moral categories: honesty, humility, altruism, modesty, sincerity, ethical; and for the immoral ones: deceptive, arrogant, cheater, egoism, vanity, corrupt. Our third hypothesis and suggested study is that the experience of the Time Travel condition together with a strong subjective sense of time travel would lead to a greater propensity to implicit selfclassification as moral, with the meaning of this given by these categories. This result is compatible with Segovia et al. (2009) who carried out an experiment where people watched their selfrepresentation avatars carry out immoral or moral actions. The results suggested that they became more immoral if they had seen their avatar carry out immoral actions compared to if they had seen their avatar carry out moral actions. From the results on *triedmybest* and its relationship to *guilt*, it is likely that most of the participants felt that they were behaving morally therefore resulting in this exploratory finding.

The fourth hypothesis and suggested study is that guilt feelings with respect to harm caused to others may be positively associated with the illusion of time travel in the Time Travel condition. We speculate that this may be because there is a greater likelihood of association of the time travel with actual history. We have seen that plausibility is positively associated with the illusion of time travel, but the causality may go both ways. If participants observe their own actions being replayed in the scenario, then since they remember having carried out those actions for sure, this may add to the illusion that these events actually happened. Therefore, the illusion that harm was done to people would be strengthened, and since that harm, whatever happened, was partly the responsibility of the actions (or non-actions) of the participant, there is greater room for guilt feelings. An experimental study would be required to test these ideas, and also to untangle the direction of causality.

The fifth hypothesis and study is that the illusion of time travel is associated with tending to agree with a utilitarian solution of the classic moral dilemmas. To explore this further, based on the current data, instead of using the variable *save1*, we replace that by the binary variable *footbridge*—which would be an extreme version of utilitarianism not normally chosen by the majority of respondents—i.e., deliberately and directly causing the death of 1 by pushing him into the path of the boxcar in order to save the 5. When this variable is used instead of *save1* then again there is a positive association with the time travel response (coefficient = 0.36, *P* = 0.001). Why people should become more utilitarian is not clear. It could be because the illusion of time travel leads to thoughts about future consequences in the sense that the deaths of five people may have a far greater impact on the future than the death of 1 (even though there may not be any absolute calculus that says that the death of x people is preferable to the death of y if x < y).

One very interesting finding is the behavior in the 1st time around. Recall that almost all the participants pressed the alarm

button and that almost all the visitors were killed. This is in contrast to questionnaire studies, and also the two virtual reality studies that have reproduced moral dilemmas in IVR (Pan and Slater, 2011; Navarrete et al., 2012). In both, participants overwhelmingly saved the 5 (89 and 90% respectively in the relevant condition). It seems that giving participants any alternative action that seems to be related to solving the crisis, even though the action is useless (here pressing the alarm), is not helpful at all. In the classical and previous VR experiments typically one bystander would die out of the six, whereas here we have found it to be nearly all of them.

It is important to note that there were many other variables that were not statistically related to either the experimental conditions or to the illusion of time travel. For example, other than the differences in the number of actions, explainable by the situation of the different conditions, there seems to be no difference in actual behavior between the two conditions. It is also possible that the two IAT tests may have influenced one another. There were several variables with missing data that were not used in the current analysis, so we emphasize that our experiment was only exploratory, and its findings should be regarded as hypotheses for future work.

There is growing interest in the concept of "mental time travel," that is the ability to project oneself to the past to relive past events in imagination (episodic memory), and to project also the future. There is evidence that episodic memory and the capacity to simulate future events share the same neural substrates, suggesting a common neurocognitive system (Botzung et al., 2008) and similar conclusions were drawn from a comprehensive linguistic analysis (Stocker, 2012). This has led to suggestions that episodic memory should be considered a part of a more general faculty of mental time travel, which includes key capabilities such as planning (Suddendorf and Corballis, 2007), or even suggestions for re-conceptualizing memory (Schacter et al., 2007). There is also a debate on whether this capability is unique to humans (Suddendorf and Busby, 2003; Suddendorf and Corballis, 2007).

However, if mental time travel is beneficial, in particular in the domain of self-improvement, then we suggest that our virtual time travel could also be beneficial, even though many of the research questions considered in the field of mental time travel are unrelated to our virtual time travel. At least since psychoanalysis, uncovering and to some extent reliving in imagination episodes from our personal history, have been thought to be beneficial. This has found its way into modern cognitive behavioral therapy, for example, in a method for the treatment of post-traumatic stress disorder (Ehlers et al., 2005). In this case an aspect of the treatment is to identify salient moments in the memory of the traumatic event, and then inventing an "alternative appraisal that the patient finds compelling" and actively incorporating the new appraisal into the trauma memory. This incorporation can be verbal, through imagery, through writing, or though acting out the memory.

Giving people the experience of time travel, and thereby an implicit learning that the past is mutable may be useful in releasing the grip of such past traumatic memories. Moreover, our approach opens the door to laboratory controlled experimental studies of the consequences of virtual time travel in this and other related domains.

### **ACKNOWLEDGMENTS**

This research was funded by the European Research Council (ERC) Advanced Grant TRAVERSE (#227985), the FP7 Integrated Project BEAMING (#248620), and the Leverhulme Trust project "The Exploitation of Immersive Virtual Reality for the Study of Moral Judgments" (F/07 134/CT). We thank Itxaso Barberia for help with the IAT, and Jordi Santiago, Sameer Kishore and Ausiàs Pomés for help with the video. We further thank Sameer Kishore, David Swapp, and Sylvia Terbeck for commenting on an earlier draft.

### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fpsyg. 2014.00943/abstract

### **REFERENCES**

Banakou, D., Groten, R., and Slater, M. (2013). Illusory ownership of a virtual child body causes overestimation of object sizes and implicit attitude changes. *Proc. Natl. Acad. Sci. U.S.A.* 110, 12846–12851. doi: 10.1073/pnas.1306779110


Wright, S. (1921). Correlation and causation. *J. Agric. Res.* 20, 557–585.

Xu, H., Bègue, L., and Bushman, B. J. (2012). Too fatigued to care: ego depletion, guilt, and prosocial behavior. *J. Exp. Soc. Psychol.* 48, 1183–1186. doi: 10.1016/j.jesp.2012.03.007

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 13 June 2014; accepted: 06 August 2014; published online: 02 September 2014.*

*Citation: Friedman D, Pizarro R, Or-Berkers K, Neyret S, Pan X and Slater M (2014) A method for generating an illusion of backwards time travel using immersive virtual reality—an exploratory study. Front. Psychol. 5:943. doi: 10.3389/fpsyg.2014.00943 This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Friedman, Pizarro, Or-Berkers, Neyret, Pan and Slater. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

**OPINION ARTICLE** published: 22 August 2014 doi: 10.3389/fpsyg.2014.00907

## Future directions in precognition research: more research can bridge the gap between skeptics and proponents

## *Michael S. Franklin1,2\*, Stephen L. Baumgart <sup>2</sup> and Jonathan W. Schooler 1,2*

<sup>1</sup> Department of Psychological and Brain Sciences, University of California Santa Barbara, Santa Barbara, CA, USA

<sup>2</sup> Theoretical and Applied Neurocausality Laboratory, Santa Barbara, CA, USA

\*Correspondence: franklin@psych.ucsb.edu

### *Edited by:*

James M. Broadway, University of California Santa Barbara, USA

#### *Reviewed by:*

Imants Baruss, King's University College at The University of Western Ontario, Canada Daryl J. Bem, Cornell University, USA

**Keywords: precognition, retrocausality, psi, skepticism**

## **INTRODUCTION**

Although claims of precognition have been prevalent across human history, it is no surprise that these assertions have been met with strong skepticism. Precognition, *the ability to obtain information about a future event, unknowable through inference alone, before the event actually occurs*, conflicts with the fundamental subjective experience of time asymmetrically flowing from past to future, brings into question the notion of free will, and contends with steadfast notions of cause and effect. Despite these reasons for skepticism, researchers have pursued this topic, and a large database of studies conducted under controlled laboratory conditions now exist. This work roughly spans from the 1930's (e.g., Rhine, 1938) up to this day (Bem, 2011; Mossbridge et al., 2014; Rabeyron, 2014). The accumulated evidence includes significant meta-analyses of forced-choice guessing experiments (Honorton and Ferrari, 1989), presentiment experiments (Mossbridge et al., 2012), and recent replications from Bem (2011, discussed below; Bem et al., 2014).

Perhaps most central to the recent debate regarding the existence of precognition is work by Bem (2011). Bem (2011) time-reversed several classic psychology effects (e.g., studying after instead of before a test; being primed after, instead of before responding) and found evidence across nine experiments supporting precognition. Given the sound methodology and publication at a high-impact mainstream psychology journal, *Journal of Personality and Social Psychology*, this work has prompted the attention of psychologists; and, not surprisingly, the response has been skeptical (Rouder and Morey, 2011; Wagenmakers et al., 2011). While we acknowledge skepticism and close scrutiny is vital in reaching consensus on this topic, given the equivocation surrounding the results, we propose that more research is needed. In particular, we suggest that applied research designs that allow for the prediction of meaningful events ahead of time can move this debate forward. Since it is not obvious how experiments that do not require explicit "guessing" of future events could be used for this goal, we give a general overview of two methodologies designed toward this aim.

## **PHYSICAL IMPLAUSIBILITY**

It is not unexpected that psychologists are most skeptical of precognition (Wagner and Monnet, 1979). This is likely due to their knowledge of the many illusions and biases that influence perception and memory. However, putting these cognitive biases aside, this work is often dismissed out of hand under the assumption that precognition would require overturning basic and essential physical and psychological tenets. Schwarzkopf (2014) illustrates this position:

". . . the seismic nature of these claims cannot be overstated: future events influencing the past breaks the second law of thermodynamics. . . It also completely undermines over a century of experimental research based on the assumption that causes precede effects"

Some clarification is needed here. From a physics perspective, except for several processes studied in high-energy physics (such as B meson decay), non-thermal physics is time-symmetric, perhaps allowing the possibility of precognitive effects. The formalism of time symmetric physics has been used, for example, in the Wheeler-Feynman absorber theory of radiation (Wheeler and Feynman, 1945) as well as in the transactional interpretation of quantum mechanics (Cramer, 1986), in which quantum wavefunction collapse is described as being due to an interaction between advanced waves (traveling backwards-in-time) and retarded waves (traveling forwards-in-time). With regards to precognition, Bierman (2008) has proposed that coherent conditions present in the human brain allow the fundamental time symmetry of physics to manifest itself.

Some quantum mechanical experiments can be interpreted as showing retrocausal influence where a decision at a future time seems to affect a past time. One example is Wheeler's delayedchoice experiment in which the way a photon travels through an interferometer (wave-like or particle-like) appears to be affected by a measurement decision made at a later time (Wheeler, 1984; Jacques et al., 2007). However, information transfer into the past (retrocausal signaling), as opposed to influence without information transfer, remains controversial since it has not yet been demonstrated experimentally. That said, there is no physical law which precludes retrocausal information transfer. There has been some effort put into experimental realization of retrocausal signaling. Cramer proposed that standard quantum mechanics allows the construction of a retrocausal signaling machine using quantum optical interferometry (Cramer, 2007). Though Cramer's work has reached an impasse (Cramer, 2014), an approach of using entangled systems for retrocausal communication may reveal a physical explanation for precognition. Lastly, it is worth noting, that ultimately whether any given theory can accommodate precognition or not is irrelevant; what is relevant are the data.

### **RELIABILITY CONCERNS**

Although it appears premature to rule out precognition from a physics standpoint, there have been concerns regarding the reliability of precognitive effects. In essence, the question boils down to whether there are in fact small, yet real, precognitive effects that are hard to pin down and require further study to isolate, or, whether the evidence for precognition is based on false-positives emerging due to biases in the research process. For a recent overview of these issues in psychology see the November, 2012 issue of *Perspectives on Psychological Science*. Interestingly, a recent commentary (Jolij, 2014) notes the similarity between precognitive effects and those in social priming research. Indeed, both research areas report small effect sizes, replication difficulty, and specific "boundary" conditions (covariates) that moderate the effect (Wilson, 2013). Although researchers point toward metaanalyses to bolster their position, metaanalyses are also susceptible to bias and rarely lead to headway in controversial areas (Ferguson, 2014). The resemblance between precognitive effects and those seen in the mainstream psychological literature has been used to leverage support for precognition (e.g., Cardeña, 2014); however, the difficulties of replicating other paradigms in psychology seems a dubious source of solace for the challenge of replicating precognition findings. Moreover, even if precognition results were robustly replicated as some meta-analyses have suggested, there is always the concern that there is some artifact driving the effect. As such, we suggest new directions for future research in precognition; one that can simultaneously address concerns about the robustness of the effects and the possibility that they are driven by unrecognized artifacts.

## **FUTURE DIRECTIONS IN PRECOGNITION RESEARCH**

What would provide the most compelling evidence for skeptics? Ultimately, we realize that the most convincing demonstration would be to show tangible effects applied in real-world settings. If a paradigm can make accurate predictions about events that people consider important and are incapable of predicting using standard means, then the significance of the paradigm becomes selfevident. Perhaps most compelling would be if an experiment could be devised to predict games of chance and/or the whether it will be a good or bad day on the stock market. Although a few reports exist in the literature of precognitive applications, in particular those that utilize associative remote viewing (predicting silver future: Puthoff, 1984; stock market; Smith et al., 2014), there has not been a single replicable methdology that has translated into consistent winnings in games of chance. Below we give a brief overview of two experiments designed to predict the outcome of random<sup>1</sup> binary events in realtime (specifically, the outcome of a roulette spin, black vs. red, excluding green; see **Figure 1**).

The left side of **Figure 1** presents a general overview of one approach. This experiment is based on work designed to examine whether extended future practice in some domain can extend backwards in time to influence prior performance. The original experiment designed toward this aim used a novel 2-phase Go-NoGo experiment (Franklin, 2007). In phase 1 of the experiment, all participants complete an identical Go-NoGo task in which individual shapes are presented for a second, one at a time, on a computer screen. Each stimulus either requires a response ("Go") or not ("NoGo"). Participants are told to respond (using the spacebar) to shapes A and B and withhold responses to shapes C and D. In phase 2, participants are randomly divided into 2 groups with each group responding exclusively to a single shape (A or B). The rationale is akin to the subtraction method/additive factors methodology (Sternberg, 1969). If phase 1 performance is influenced by only past experience, then there should be no difference in reaction times or accuracy based on future condition assignment. If, however, phase 1 performance is influenced not only by past experience, but future experience as well, systematic differences in performance based on phase 2 condition assignment should emerge. As seen in **Figure 1B**, by mapping shapes A and B to outcomes of the roulette spin (RED and BLACK), it should be possible (assuming a genuine precognitive effect) to use phase 1 performance to predict the roulette spin outcome before the wheel is spun.

Next we describe an experiment using EEG to detect predictive anticipatory activity (PAA; Mossbridge et al., 2014); also known as presentiment, the finding that various physiological measures of arousal are higher preceding the onset of emotionally charged vs. neutral pictures that are randomly presented (Bierman and Radin, 1997; Radin, 1997; Bierman and Scholte, 2002; Spottiswoode and May, 2003; Mossbridge et al., 2012). The specific methodology below extends work reported in Radin (2011), in which the pre-stimulus EEG activity of experienced meditators was found to differ significantly in response to light flashes and auditory tones. As seen in **Figure 1**, by mapping the light flash and auditory tone to a binary target (RED vs. BLACK roulette spin) and by evaluating baseline and pre-stimulus EEG potentials in realtime, it should be possible to predict the state of a future random target, allowing above-chance retrocausal communication. Similar to the first experiment design, the results of the prediction can be compared against chance (50%) with an exact binomial test. Currently, pilot testing with this basic design is underway, along with additional testing to assess whether a stimulus (flash vs. tone) triggered by the appropriate symmetric pre-stimulus response (a "neurofeedback" condition; e.g., flash delivered when occipital EEG increases) can condition response patterns in anticipation to random stimuli determined by

<sup>1</sup>Although there is an important distinction between truly random vs. pseudorandom selection, since any genuine precognitive effect of future stimuli on past behavior/physiology should be independent of selection method, we do not distinguish between these for the purposes of this overview.

roulette spin; allowing for a retrocausal Brain Computer Interface (BCI).

detection of the shape practiced in the future) would allow for a real-time

The design presented in **Figure 1** has the benefit of more protection against anticipation/learning strategies (there is only one future event). Also, extended exposure to the future stimulus may strengthen the effect and allow for more time between the prediction, bet and outcome. Although the EEG experiment relies on fewer data points for each prediction, this method could lead to BCI applications and be more powerful due to the large number of trials collected within and across participants. Altogether, there appears to be no inherent confound in either design given sufficient sample size i.e., we know of no conventional confound that could lead to consistent above chance prediction in real time of a roulette spin. As such, both designs are worth exploring in future research.

AUDITORY (bet BLACK).

## **FINAL THOUGHTS**

Despite the accumulated data, and recent positive findings in the literature, significant controversy remains regarding the interpretation of the evidence for the existence of precognition. Proponents find the combined results as compelling evidence in support of precognition, with similar (small) effect sizes to those reported throughout the psychological literature. Skeptics, however, question potential methodological and/or analytical confounds in those studies, as well as the physical plausibility of precognition. Both, however, agree regarding the profound implications if these bold claims are true. We suggest that although the current state of evidence does not quite merit proponents' strong claim of having

<sup>2</sup> If the ball lands on green, re-spinning would occur until it lands on either black or red.

demonstrated replicable precognition in the laboratory, the accumulated experimental evidence, combined with advances in theoretical physics, warrant further research. We believe the most effective way forward is through the development of paradigms that use software in real-time to predict meaningful future outcomes before they occur. As others have noted (Mossbridge et al., 2014) a new technology that uses behavior and/or physiology to consistently predict random future events above chance would certainly be a "game-changer."

### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 26 June 2014; accepted: 29 July 2014; published online: 22 August 2014.*

*Citation: Franklin MS, Baumgart SL and Schooler JW (2014) Future directions in precognition research: more research can bridge the gap between skeptics and proponents. Front. Psychol. 5:907. doi: 10.3389/fpsyg. 2014.00907*

*This article was submitted to Perception Science, a section of the journal Frontiers in Psychology.*

*Copyright © 2014 Franklin, Baumgart and Schooler. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## ADVANTAGES OF PUBLISHING IN FRONTIERS

FAST PUBLICATION Average 90 days from submission to publication

COLLABORATIVE PEER-REVIEW

Designed to be rigorous – yet also collaborative, fair and constructive

RESEARCH NETWORK Our network increases readership for your article

## OPEN ACCESS

Articles are free to read, for greatest visibility

### TRANSPARENT

Editors and reviewers acknowledged by name on published articles

GLOBAL SPREAD Six million monthly page views worldwide

### COPYRIGHT TO AUTHORS

No limit to article distribution and re-use

IMPACT METRICS Advanced metrics track your article's impact

SUPPORT By our Swiss-based editorial team

EPFL Innovation Park · Building I · 1015 Lausanne · Switzerland T +41 21 510 17 00 · info@frontiersin.org · frontiersin.org