# NEAR-INFRARED SPECTROSCOPY (NIRS) IN FUNCTIONAL RESEARCH OF PREFRONTAL CORTEX

EDITED BY: Nobuo Masataka, Leonid Perlovsky and Kazuo Hiraki PUBLISHED IN: Frontiers in Human Neuroscience

#### *Frontiers Copyright Statement*

*© Copyright 2007-2016 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

> *All copyright, and all rights therein, are protected by national and international copyright laws.*

*The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88919-944-0 DOI 10.3389/978-2-88919-944-0

### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

## **NEAR-INFRARED SPECTROSCOPY (NIRS) IN FUNCTIONAL RESEARCH OF PREFRONTAL CORTEX**

Topic Editors:

**Nobuo Masataka,** Kyoto University, Japan **Leonid Perlovsky,** Harvard University and Air Force Research Laboratory, USA **Kazuo Hiraki,** The University of Tokyo, Japan

This e-book includes the latest outcomes produced by a broad range of fNIRS research with activation of prefrontal cortex, from methodological one to clinical one, providing a forum for scientists planning functional studies of prefrontal brain activation. Reading this book, one will find the possibility that fNIRS could replace fMRI in the near future, and realize that even our aesthetic feeling is measurable. This will serve as a reference repository of knowledge from these fields as well as a conduit of information from leading researchers. In addition it offers an extensive cross-referencing system that will facilitate search and retrieval of information about NIRS measurements in activation studies. Researchers interested in fNIRS would benefit from an overview about its potential utilities for future research directions.

**Citation:** Masataka, N., Perlovsky, L., Hiraki, K., eds. (2016). Near-Infrared Spectroscopy (NIRS) in Functional Research of Prefrontal Cortex. Lausanne: Frontiers Media. doi: 10.3389/978-2-88919-944-0

# Table of Contents

*05 Temporal hemodynamic classification of two hands tapping using functional near—infrared spectroscopy* Nguyen Thanh Hai, Ngo Q. Cuong, Truong Q. Dang Khoa and Vo Van Toi *17 NIRS-measured prefrontal cortex activity in neuroergonomics: strengths and weaknesses*

Gérard Derosière, Kévin Mandrick, Gérard Dray, Tomas E. Ward and Stéphane Perrey


Laura Ferreri, Jean-Julien Aucouturier, Makii Muthalib, Emmanuel Bigand and Aurelia Bugaiska


Yusuke Moriguchi and Kazuo Hiraki


*112 Prefrontal cortex activation during story encoding/retrieval: a multi-channel functional near-infrared spectroscopy study*

Sara Basso Moro, Simone Cutini, Maria Laura Ursini, Marco Ferrari and Valentina Quaresima

*123 Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS*

Christian Herff, Dominic Heger, Ole Fortmann, Johannes Hennrich, Felix Putze and Tanja Schultz


Hiroki Sato, Thomas Dresler, Florian B. Haeussinger, Andreas J. Fallgatter and Ann-Christine Ehlis

*149 Negative emotion modulates prefrontal cortex activity during a working memory task: a NIRS study*

Sachiyo Ozawa, Goh Matsuda and Kazuo Hiraki


Anthony C. Ruocco, Achala H. Rodrigo, Jaeger Lam, Stefano I. Di Domenico, Bryanna Graves and Hasan Ayaz


## Temporal hemodynamic classification of two hands tapping using functional near—infrared spectroscopy

#### *Nguyen Thanh Hai 1, Ngo Q. Cuong2, Truong Q. Dang Khoa1 \* and Vo Van Toi <sup>1</sup>*

*<sup>1</sup> Biomedical Engineering Department, International University of Vietnam National Universities in Ho Chi Minh City, Ho Chi Minh City, Vietnam*

*<sup>2</sup> Department of Electronics and Telecommunications, Faculty of Electrical and Electronics Engineering, University of Technical Education HCMC, Ho Chi Minh City, Vietnam*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Carlo Cattani, University of Salerno, Italy Thang M. Hoang, University of Canberra, Australia*

#### *\*Correspondence:*

*Truong Q. Dang Khoa, Biomedical Engineering Department, International University of Vietnam National Universities in Ho Chi Minh City, Quarter 6, Linh Trung Ward, Thu Duc District, Ho Chi Minh City, Vietnam e-mail: khoa@ieee.org*

In recent decades, a lot of achievements have been obtained in imaging and cognitive neuroscience of human brain. Brain's activities can be shown by a number of different kinds of non-invasive technologies, such as: Near-Infrared Spectroscopy (NIRS), Magnetic Resonance Imaging (MRI), and ElectroEncephaloGraphy (EEG; Wolpaw et al., 2002; Weiskopf et al., 2004; Blankertz et al., 2006). NIRS has become the convenient technology for experimental brain purposes. The change of oxygenation changes (oxy-Hb) along task period depending on location of channel on the cortex has been studied: sustained activation in the motor cortex, transient activation during the initial segments in the somatosensory cortex, and accumulating activation in the frontal lobe (Gentili et al., 2010). Oxy-Hb concentration at the aforementioned sites in the brain can also be used as a predictive factor allows prediction of subject's investigation behavior with a considerable degree of precision (Shimokawa et al., 2009). In this paper, a study of recognition algorithm will be described for recognition whether one taps the left hand (LH) or the right hand (RH). Data with noises and artifacts collected from a multi-channel system will be pre-processed using a Savitzky–Golay filter for getting more smoothly data. Characteristics of the filtered signals during LH and RH tapping process will be extracted using a polynomial regression (PR) algorithm. Coefficients of the polynomial, which correspond to Oxygen-Hemoglobin (Oxy-Hb) concentration, will be applied for the recognition models of hand tapping. Support Vector Machines (SVM) will be applied to validate the obtained coefficient data for hand tapping recognition. In addition, for the objective of comparison, Artificial Neural Networks (ANNs) was also applied to recognize hand tapping side with the same principle. Experimental results have been done many trials on three subjects to illustrate the effectiveness of the proposed method.

**Keywords: polynomial regression algorithm, support vector machines, artificial neural networks, hand tapping recognition, functional Near-Infrared Spectroscopy**

#### **INTRODUCTION**

Human brain has a complex structure with around 100 billion neurons, so it is a big challenge for all scientists in biological computing (Wolpaw et al., 2002). These neurons can communicate from one to another with or without external excitations to make typical decisions (pattern recognition, cognition, motion, and others; Critchley, 2009). Moreover, in prefrontal cortex of human brain plays an important role in social activity for both adults and children. Tobias Grossmann represented a review related to the role of prefrontal cortex of human brain, in which specific areas in the adult human brain as social brain could process the social world (Aydore et al., 2010; Grossmann, 2013) and also Tila Tabea Brink et al. investigated about orbitofrontal cortex in children with 4− to 8-year-old through processing empathy stories (Brink et al., 2011). The result is that children could passively follow these stories presenting social situations. Regarding prefrontal cortex, EEG electrodes were mounted on frontal positions of human brain for wheelchair control (Ahmed, 2011). In particular, user could move eyes to drive the electrical wheelchair to reach the desired target.

In recent decades, a lot of achievements have been obtained in imaging and cognitive neuroscience of human brain. Brain's activities can be explored using different kinds of non-invasive technologies, such as: Magnetic Resonance Imaging (MRI), Near-Infrared Spectroscopy (NIRS), and ElectroEncephaloGraphy (EEG; Wolpaw et al., 2002; Weiskopf et al., 2004; Blankertz et al., 2006; Ince et al., 2009). Many researchers have been attracted by these technologies with many approaches to find out problems related to human brain for rehabilitation and treatment. For the rehabilitation problem, information obtained from human brain using EEG technique could be employed to perform shared control of motion wheelchairs (Tanaka et al., 2005). A brain simulator can lead to improve or to recover the cognitive/motor functions of tetraplegic patients with degenerative nerve diseases spinal cord injuries (Kauhanen et al., 2006). In these non-invasive technologies, the NIRS technology is often applied to measure Oxygen Hemoglobin (Oxy-Hb), deOxy-Hb, and Total-Hb concentration changes. These changes allow us predict brain activations related to body behaviors.

fNIRS has become the convenient technology for experimental brain purposes. This non-invasive technique emits near infrared light into the brain to measure cerebral hemodynamics as well as to detect localized blood volume and oxygenation changes (Tsunashima and Yanagisawa, 2009). The change of oxy-Hb along task period depending on the location of channels the cortex has been studied: sustained activation in the motor cortex, transient activation during the initial segments in the somatosensory cortex, and accumulating activation in the frontal lobe (Gentili et al., 2010). Oxy-Hb concentration at the aforementioned sites in the brain can also be used as a predictive factor allows prediction of subjects' investigation behavior with a considerable degree of precision (Shimokawa et al., 2009).

fNIRS technique is a non-invasive technique which is applied to monitor human body for diagnosis and treatment (Bozkurt et al., 2005; Macnab et al., 2011; Reher et al., 2011). Hiroshi Taniguchi et al. investigated six subjects with unilateral spatial neglect (USN)-positive (+) and 6 others with USN-negative (Taniguchi et al., 2012). In this research, brain activity was simulated by prism adaptation tasks using fNIRS. The result showed that there was a typically great reduction in Oxy-Hb of the USN (+). For monitoring carotid endarterectomy, one was applied the NIRS technique to evaluate its reliability in the detection of clamping ischemia (Pedrini et al., 2012). The result found that there were three patients who represented transient ischemic deficits at awakening and no case of perioperative stroke or death.

In addition, fNIRS technique has been appeared as an alternative brain-based experimental technique (Lloyd-Fox et al., 2010) to measure human thoughts and activities for rehabilitation. For evaluating behaviors related to human brain during experiments, subjects feel free for performing his or her brain activities. In particular, this technique has been successfully used to study brain functions such as assessment of motor task from everyday living, athletic performance, recovery from neurological illness (Hu et al., 2010), assessment of verbal fluency (Schecklmann et al., 2010), and quantification of brain function during finger tapping (Sato et al., 2007). However, to the best of our knowledge, there have been a few applications of the fNIRS technique to quantify the motor control signals leading to brain simulator for rehabilitation (Chunguang et al., 2010; Gentili et al., 2010).

Neural networks can be used for cognition brain tasks as a classification module, in which wavelet decomposition can be used as feature extractions (Khoa and Nakagawa, 2008); wavelet can be used to remove artifacts (Molavi and Dumont, 2010). Base on the slope of straight line, hand side tapping can be distinguished (Ngo et al., 2012). Oxy-Hb and Deoxy-Hb can also be used directly with SVM algorithm for the recognition of hand tapping (Sitaram et al., 2007).

Savitzky–Golay (SG) filters have been used to smooth signals and images with noises as well as artifacts in recent years. In the SG filters, the coefficients of the local least-square polynomial fit are pre-computed to preserve higher movements and then the output of the filter is taken at the center of the window (Savitzky, 1964; Steinier et al., 1972; Gorry, 1990). In this paper, the SG filter was applied to reduce spike noises of Oxy-Hb signals. The Oxy-Hb signals after filtering allow us be easier in recognizing left (LH) or right hand (RH) tapping status. Moreover, a Polynomial Regression (PR) approach has been applied for estimation of signals and images with noise (Cui and Alwan, 2005; Cai et al., 2007; Zhang et al., 2009; Khan et al., 2011). In our research, in order to estimate Oxy-Hb signals, the PR algorithm was used to produce polynomial curves with their features. Based on these features, one can classify tapping hand tasks.

Support Vector Machine (SVM) algorithms have been applied for classification problems in the machine learning community in recent years. In this case, the SVM was employed to classify hypothyroid disease based on UCI machine learning dataset (Chamasemani and Singh, 2011). Another application related to medical images is that the SVM was utilized to recognize the leaf spectral reflectance with different damaged degrees in the image processing and spectral analysis technology (Dake and Chengwei, 2006). In this project, the SVM algorithm (Sitaram et al., 2007) was applied to recognize hand tapping tasks using fNIRS technology. Oxy-Hb signals after reducing noise will be extracted features using a PR algorithm. Based on coefficients obtained from the PR, the SVM algorithm will be applied for the recognition of the LH and RH tapping tasks.

Another algorithm for classification is that a recursive training algorithm for EEG signals using Artificial Neural Networks (ANNs) to generate recognition patterns from EEG signals was proposed to control electric wheelchair (Tanaka et al., 2005; Singla et al., 2011). Mental tasks were classified for wheelchair control using prefrontal EEG (Rifai Chai, 2012). The relevant mental tasks used in this paper are mental arithmetic, ringtone imagery, finger tapping, and words composition with additional tasks which are baseline and eyes closed. The feature extraction is based on the Hilbert Huang Transform (HHT) energy method and then the ANNs with the Genetic Algorithm (GA) optimization (Subasi et al., 2005) was applied for classification. The result is that the accuracy of the proposed classification algorithm with the five subjects participated was between about 76 and 85%.

In this paper, we proposed the recognition algorithm for developing a brain computer interface using fNIRS. First of all, Savitzky–Golay filter is used to reduce noises as well as artifacts. Coefficients, which are features of Oxy-Hb signals, are found by using a PR algorithm. For the recognition of tapping hands related to the left and right brain activation, ANN and SVM algorithms were used. These two methods will be compared to find out the best one. The results and discussion about tapping hand activity will be shown to illustrate the effectiveness of the proposed approaches. This process is shown in **Figure 1**.

#### **MATERIALS AND METHODS**

#### **SUBJECTS AND THE EXPERIMENTAL SETUP**

A multichannel fNIRS instrument, FOIRE-3000 (SHIMAZU Co. LTD, Japan), is used to acquire brain Oxy-Hb. This machine was located at Lab-104 of Biomedical Engineering Department, International University, VNU, Vietnam. The FOIRE 3000 system with the eight pairs of the probes, consisting of the illuminator and detector optodes, produces 24 channels as shown in **Figure 2A**. These probes were placed on the scalp to collect fNIRS data, in which the detectors were installed at a 3 cm distance from the illuminators. The optodes were arranged to install at the left hemisphere on the head of the subject.

Oxy-Hb concentration changes in motor control area of human brain was captured from a set of the holder with 24 channels for both hemispheres using the fNIRS technique as shown in **Figure 2B**. In particular, when the subject performs one typical activity, brain signals will be obtained from the fNIRS system and then calculated to produce three types of signals [Oxy-Hb (red), Total-Hb (green) and Deoxy-Hb (blue)] corresponding to three wavelengths (780, 805, and 830 nm), in which [Total-Hb] = [Oxy-Hb] + [Deoxy-Hb]. Moreover, the distance between pairs of emitter and detector probes was set at 3 cm and all probes were attached with holders arranged on different sides of human brain hemispheres depending on users. Concentration changes of three signals produce time points in an output. In this research, Oxy-Hb changes are calculated in the following formula (Shimadzu Corporation, 2010):

$$\text{Oxy} = -3.6132 \ast \text{Abs}[780 \,\text{nm}] + 1.1397 \ast \text{Abs}[805 \,\text{nm}]$$

$$+ \text{ 3.0153} \ast \text{Abs}[830 \,\text{nm}] \tag{1}$$

in which Abs: Absorbance.

Three subjects (male, average: 25 years old, 60 kg weights, right-handed) were participated into this study. All participants

coefficients. Finally, Artificial Neural Network or Support Vector Machines is used to determine whether left hand or right hand is tapped.

were healthy and showed no musculoskeletal or neurological restrictions or diseases. Before participating into the experiments, each subject was asked to fill out a questionnaire consisting of patient's identification, age and gender, which was kept confidential. The tenets of the Declaration of Helsinki were followed; the local Institutional Review Board approved the study. These subjects informed consent agreement after reading and understanding of the experiment protocol and the fNIRS technique.

After reading and understanding the experiment protocol and the fNIRS technique, he will start doing hand tapping. The subject was required to perform hand tapping motions, both left and right sides as motor activities. In these hand tapping motions, a protocol includes 20 s (Rest)—20 s (Task)—20 s (Rest), it means that the subject relaxed in 20 s, tapped his hand up/down about 10 times in 20 s, and then rested 20 s, as shown in **Figure 3**.

Oxy-Hb data were collected on 20 channels, in which 10 channels are of the left brain side and that of the opposite side will be obtained for hand tapping recognition. However, we just chose 4 channels of each side which focus on hand and leg motion area to analyze and to estimate features. In particular, the left brain channels are 2, 5, 6, 9, and the 12, 15, 16, 19 channels are of the right brain side as in **Figures 4A,B**. In this research, Oxy-Hb data obtained from these channels will be processed to recognize hand tapping tasks. Without loss of generality, the natural architecture is different from person to person. The probes are allocated on the holder, in which the transmitter probes and receiver probes are predicted to cover as much as area of brain based on the physical structure of each subject. The authors (Aihara et al., 2012) combined EEG and NIRS for estimation of cortical current source. The probes position using stylus marker to allow co-registration

**FIGURE 2 | (A)** fNIRS FOIRE-3000 system. This system operates at three different wavelengths of 780, 805, and 830 nm. **(B)** Subject' head with installed probes. The distance between pairs of emitter and detector probes was set at 3 cm and all probes were attached with holders.


of EEG and NIRS results. In this paper, we also used marker to find out the average positions of motor area of human brain cortex. To achieve more accuracy, the NIRS activity was mapped onto cerebral cortex using fusion software (Shimadzu Corporation, 2010). From this evidence, we proposed the selection of channels 2, 5, 6, 9 and 12, 15, 16, 19 for hand tapping recognition with the 20-channel NIRS system configured above.

#### **DATA PRE-PROCESSING**

Brain data of a subject acquired from the channels have noise and artifacts. In order to obtain more smoothly brain data, a filter as the Savitzky–Golay filter was applied in this paper. The Savitzky– Golay filters (Orfanidis, 2010) are also known as polynomial smoothing. It means that the idea of the polynomial smoothing is replacing samples of signal by the values that lie on the smoothing curve. In moving an average FIR filter, the output is a simply average version of its inputs, in which this filter has the response of the low-pass filter. In practice, NIRS signals fluctuate along time corresponding to excitations and have the unknown specific frequency. Therefore, it could not be the average of inputs with the arbitrary FIR filter length. In this research, to track the acquired signal, the Savitzky–Golay filter as the FIR filter can be used.

In general, we can evaluate a polynomial with the order of *d* to smooth the length-*N* data *x* with the condition *N* ≥ *d* + 1. Assume that, the data *x* is the type of a vector

$$\mathbf{x} = [\mathbf{x}\_{-M}, \dots, \mathbf{x}\_{-1}, \mathbf{x}\_0, \mathbf{x}\_1, \dots, \mathbf{x}\_M]^T \tag{2}$$

in which *N* samples of *x* are replaced by the polynomial with the order of *d* as follow:

$$\hat{\mathbf{x}}\_m = \mathbf{c}\_0 + \mathbf{c}\_1 m + \dots + \mathbf{c}\_d m^d, \quad -M \le m \le M \tag{3}$$

where *c*0, *c*1, ..., *cd* denote polynomial coefficients. *M* is the number of points on either side of *x*<sup>0</sup>

In this case, there are *d* + 1 based on the vector *si*, *i* = 0, 1, ..., *d* as follows:

$$s\_i(m) = m^i, \quad -M \le m \le M \tag{4}$$

Thus, we can write the vector *S* as follows:

$$S = [s\_0, s\_1, \dots, s\_d] \tag{5}$$

in which s0, s1, ..., s*<sup>d</sup>* are the polynomial basic vectors.

The smooth values in (3) can be re-written in the following equation:

$$\hat{\mathbf{x}} = \sum\_{i=0}^{d} c\_i \mathbf{s}\_i = [\mathbf{s}\_0, s\_1, \dots, s\_d] \begin{bmatrix} c\_0 \\ c\_1 \\ \vdots \\ c\_d \end{bmatrix} = \mathbf{S} \mathbf{c} \tag{6}$$

Coefficients of the desired filters are obtained as follows:

$$B = SG^T = GS^T = SF^{-1}S^T = [b\_{-M}, \dots, b\_0, \dots, b\_M] \tag{7}$$

in which, *b*−*M*, ..., *b*0, ..., *bM* are the column filters of the Savitzky–Golay filter set.

$$\begin{cases} F = \mathbb{S}^T \mathbb{S} \\ G = \mathbb{S} \mathbb{P}^{-1} \end{cases} \tag{8}$$

Finally, the values to create more smoothly signals are estimated in the following equation:

$$\hat{\mathbf{x}}\_m = b\_m^T \mathbf{x}, m = -M, \dots, 0 \dots, M \tag{9}$$

in which, *b<sup>T</sup> <sup>m</sup>* are the transpose version of *bm*.

In this paper, the Savitzky–Golay filter will be utilized to smooth spikes of brain Oxy-Hb signals for identifying hand tapping tasks. The filtered Oxy-Hb signals allow us extract features with reliable information.

#### **FEATURE EXTRACTION**

In general, the first step in classification work is to find the features of data samples. For this purpose, there are many methods such as Principle Component Analysis (PCA), Independent Component Analysis (ICA) and etc. However, hemodynamic response of human brain changes in time domain. Moreover, we want to evaluate the Oxy-Hb concentration corresponding to hand tapping tasks based on analyzing numeric as well as having a look in graphical figures.

PR algorithm (Montgomery and Runger, 2003) presents the relationship between amplitude and time of a signal. In this paper, the PR algorithm was applied to analyze brain Oxy-Hb data in blood flow corresponding to hand tapping tasks. From the processed data, one can distinguish the difference between the LH and RH tapping times corresponding to the difference of the Oxy-Hb concentration changes.

Assumed that we have the set of two-dimensional data, (*x*1, *y*1), ..., (*xn*, *yn*), where each of *x* and *y* has no information about the other. Our problem is fitting a polynomial curve generated by a typical data. Thus, the relationship between *x* and *y* can be found out. Based on the coefficients of the regression curve with the order of 5, one can estimate the hand tapping. In particular, the PR equation between independent variable *x* and *y* fitted can be expressed as:

$$
\hat{\chi} = \hat{h}\_0 + \hat{h}\_1 \mathfrak{x} + \hat{h}\_2 \mathfrak{x}^2 + \dots + \hat{h}\_m \mathfrak{x}^m \tag{10}
$$

in which, *h*ˆ0, *h*ˆ1, *h*ˆ2, ··· , *h*ˆ*<sup>m</sup>* are estimated values of *h*0, *h*1, *h*2, ··· , *hm*. There are *m* regressors and *n* observations, *xi*1, *xi*2, ··· , *xim*, *yi* , *i* = 1, 2, ..., *n* corresponding to *xi*, *x*<sup>2</sup> *<sup>i</sup>* , ··· , *<sup>x</sup><sup>m</sup> <sup>i</sup>* , *yi* . In this equation, the powers of *x* play the role of different independent variables.

The PR model can be re-written as a system of linear equations

$$y = Xh + \varepsilon \tag{11}$$

where: ε = [ε1,ε2, ··· ,ε*n*] *<sup>T</sup>* is a vector of error.

The ordinary least square *h*ˆ of *h* given by the arguments that minimize the residual sum of squares and the distributive law is employed. One obtains the equation,

$$\text{RSS}(h) = \mathbf{y}'\mathbf{y} + h'(\mathbf{X}'\mathbf{X})h - \mathbf{2}\mathbf{y}'\mathbf{X}h \tag{12}$$

Equation 12 is minimized by taking <sup>∂</sup>RSS <sup>∂</sup>*<sup>h</sup>* and set the result to zero. This leads to

$$X'Xh = X'y\tag{13}$$

The ordinary least square in the case of the inverse of X X exists is given by

$$
\hat{h} = \left(X^\prime X\right)^{-1} X^\prime \chi \tag{14}
$$

From these coefficients, one can determine problems of the LH tapping or RH tapping tasks with the measured brain data using the fNIRS technology. **Figure 5** represents the regressed signal of the channel-2 corresponding to Equation 15. Similarly, the regression signals of channels 5, 6, 9, 12, 15, 16, and 19 can be shown.

$$y\_{C2} = -0.0001x^5 + 0.0023x^4 - 0.0114x^3 + 0.0182x^2$$

$$+ 0.0043x - 0.0329 \tag{15}$$

In this figure, the blue Oxy-Hb signal after the Savitzky–Golay filter was calculated to produce the red regressive curve. Each hand tapping creates the regressive Oxy-Hb curves which contain information or its feature coefficients. For recognition of hand tapping types, these coefficients will be given the input of the identification system or called the identification algorithms.

The regressed polynomial must represent the original signal with the best fit. The smaller error between the origin (here is the filtered NIRS signal) and the regressed signal is higher than the order of the polynomial is. It means that one should choose the order not only to fit the origin but also to show the general trend and the characteristic of NIRS signal. In practice, the NIRS signal can not change immediately at the moment of tapping hand. For example, one hand moving up or down will make an excitation to both hemispheres. Therefore, in 20 s of tapping hand, one person could take 10 times of moving hand up and down. In this case, Oxy-Hb level, which will flow from the lowest to highest level in short time, is not the "trend" of overall signal. This is the reason for choosing the polynomial with the order of 5.

#### **ARTIFICIAL NEURAL NETWORK**

ANNs are the very powerful tools for the problems of classification and pattern recognition. We can use the estimated coefficients as the features from the PR algorithm by connecting with a multilayer feed forward network for recognition. The architecture of this network used here consists of an input layer, one hidden layer, and the output layer as shown in **Figure 6**. In particular, input samples are the features from channel coefficients corresponding to Oxy-Hb concentration changes. The number of hidden nodes is carefully chosen for this case to obtain higher performance. Therefore, it can be chosen as an average of number of the input nodes and the output nodes. With the hidden layer, we used the double sigmoid function and this sigmoid function was also used for the output layer.

In general, standard back propagation is used for training the network with three layers. It is a gradient descent algorithm, in which the network weights are moved along the negative of the gradient of the performance function. With this argument, the training is based on the minimization of the following error function:

$$E = \sum\_{n=1}^{N} \left( o\_n - d\_n \right)^2,\tag{16}$$

where *N* is number of samples, *o* is network output and *d* is desired output.

Suppose that the network has *I* nodes of the input layer, *J* nodes of the hidden layer and the output layer is *K* nodes. Call *w*(1,0) *<sup>j</sup>*,*<sup>i</sup>* is weight from the *i*th node of the input layer to the *j*th node of the hidden layer and *w*(2,1) *<sup>k</sup>*,*<sup>j</sup>* is weight from the *j*th node of the hidden layer to the *k*th node of the output layer. The backpropagation learning of the 3-layers network is shown in **Table 1**. The application is that with the LH tapping, the output is desired to get the value of [1; 0] and [0; 1] is the desirable value of the right tapping. The ANN is one of the approaches which is often used for recognition. In this research, the SVM is also applied to identify hand tapping tasks through Oxy-Hb flowing in brain blood.

#### **SUPPORT VECTOR MACHINES**

In order to estimate hand tapping tasks, after determining coefficients of hand tapping times using the PR algorithm, we also used the linear SVM algorithm (Shawe-Taylor, 2000) to validate the coefficient data. In the linear SVM algorithm, assume that the training data are {x*i*, *yi*}, *i* = 1,...,l ,*yi* ∈ {−1, 1},x*<sup>i</sup>* ∈ *Rd*. The points *x* which lie on the hyperplane satisfy w.x + *b* = 0, in which | *b* |/ *w* is the distance from the hyperplane to the origin (where *w* is the Euclidean norm of *w*). Let *d*<sup>+</sup> (*d*−) be the shortest distance from the seperation hyperplane to the closest positive (negative) samples corresponding to the coefficients of LH tapping and RH tapping, respectively. This is showed in **Figure 7**.

#### **Table 1 | The three-layers network with backpropagation learning.**

#### Random initial weights

While the Mean Square Error (MSE) is unsatisfied or the number of epochs is not exceed,

For each input *xp*, 1 ≤ *p* ≤ *P*, (\*)

Compute the inputs of hidden layer net(1) *p*, *j* ;

Compute the outputs of hidden layer *x*(1) *p*, *j* ;

Compute the inputs of ouput layer net(2) *<sup>p</sup>*, *<sup>k</sup>* ;

Compute the outputs of network *op*, *<sup>k</sup>* ;

Modify outer weights

$$
\Delta \mathsf{w}\_{k,\stackrel{\scriptstyle \mathsf{T}}{\dot{\mathsf{T}}}\_{\cdot,\cdot}}^{(2,\ 1)} = \mathfrak{n} \left( \mathsf{c}\_{\mathsf{p},\ \mathsf{k}} - \mathsf{c}\_{\mathsf{p},\ \mathsf{k}} \right) \mathcal{S}' \left( \mathsf{n} \mathsf{e} \mathsf{t}\_{\mathsf{p},\ \mathsf{k}}^{(2)} \right) \mathsf{x}\_{\mathsf{p},\ \mathsf{k}}^{(1)}
$$

Modify weights between input layer and hidden layer

$$\begin{split} \Delta \boldsymbol{\nu}\_{j,\,\,\widetilde{\boldsymbol{\mu}}}^{({\uparrow},{\ {\text{O}}})} &= \eta \sum\_{k=1}^{k} \left( \left( \boldsymbol{\sigma}\_{\mathbb{P},k} - \boldsymbol{\sigma}\_{\mathbb{P},k} \right) \boldsymbol{S}' \left( \boldsymbol{\mathsf{net}}\_{p,\,\,\,k}^{({\text{2}})} \right) \boldsymbol{\nu}\_{k,\,\,\,\widetilde{\boldsymbol{\mu}}}^{({\text{2}},{\text{1}})} \right) \boldsymbol{S}' \left( \boldsymbol{\mathsf{net}}\_{p,\,\,\,\widetilde{\boldsymbol{\mu}}}^{({\text{1}})} \right) \times\_{p,\,\,\widetilde{\boldsymbol{\mu}}} \\ \boldsymbol{\mathsf{E}} \boldsymbol{\mathsf{nd}} \left( \boldsymbol{\mathsf{t}} \right) & \end{split}$$

End While

Where: *S*() is the active function, η is the learning rate.

Margin of the hyperplane is *d*+ + *d*−. In the linear case, the support vector looks for the separating hyperplane with the largest margin using the primal Lagrangian. Suppose that all training data satisfy the following contraints:

$$x\_i \cdot w + b \ge +1, \quad \text{for} \quad y\_i = +1 \tag{17}$$

$$x\_i \cdot w + b \le -1, \quad \text{for} \quad y\_i = -1 \tag{18}$$

The optimization problem is considered to transform Equations 17 and 18 using the primal Lagrangian as follows:

$$L\_p(\mathbf{w}, b, \mathbf{a}) = \frac{1}{2} ||\mathbf{w}||^2 - \sum\_{i=1}^{l} \alpha\_i \mathbf{y}\_i (\mathbf{x}\_i \cdot \mathbf{w} + b) + \sum\_{i=1}^{l} \alpha\_i \quad (19)$$

where α*<sup>i</sup>* ≥ 0 are the Lagrange multipliers.

Differentiating *Lp* with respect to *w* and *b* and then getting the results to zeros, we have the following equation:

$$\frac{\partial L\_{\mathcal{P}}(\boldsymbol{w}, \boldsymbol{b}, \boldsymbol{\alpha})}{\partial \boldsymbol{w}} = \boldsymbol{w} - \sum\_{i=1}^{l} \boldsymbol{\gamma}\_{i} \boldsymbol{\alpha}\_{i} \boldsymbol{x}\_{i} = \boldsymbol{0} \tag{20a}$$

**FIGURE 8 | Raw data and its smoothed version. (A)** Raw and smoothed NIRS data of channel 2, 5, 6, 9 of the left hemisphere. **(B)** Raw and smoothed NIRS data of channel 12, 15, 16, 19 of the right hemisphere.

$$\frac{\partial L\_{\mathcal{P}}(\boldsymbol{w}, b, \boldsymbol{\alpha})}{\partial b} = \sum\_{i=1}^{l} \boldsymbol{\nu}\_{i} \boldsymbol{\alpha}\_{i} = \mathbf{0} \tag{20b}$$

D<sup>+</sup> contains the coefficients and *y* = +1 is of the left tapping; similarly D<sup>−</sup> has the coefficients and *y* = −1 is of the right tapping.

Equations can be re-written to calculate the support vector as follows:

$$\mathbf{w} = \sum\_{i=1}^{l} \mathbf{y}\_i \alpha\_i \mathbf{x}\_i \tag{21}$$

The regressed data will trained using the SVM method, in which the hyperplane is a linear function and divided into two planes: **RESULTS AND DISCUSSION**

Oxy-Hb raw signals (blue) were collected from the fNIRS system using the proposed protocol (see **Figure 5**) which plays an important role during measure tasks. In particular, each subject tapped his hand up or down 10 times in 20 s. Therefore, we could separate this task into 10 parts, in which each part has 1 s up and 1 s down as shown in **Figure 9**. Before analyzing Oxy-Hb signals,

the Savitzky–Golay (SG) filter was chosen to produce the smooth Oxy-Hb signals (red) as shown in the **Figures 8A,B**. In this filter, if the size of the window is too small, noises still affect upon the Oxy-Hb signals. Otherwise, if the large window size is chosen, the useful information may be lost. As mentioned before, Oxy-Hb signals using fNIRS technique are the concentration of Oxy-Hb in blood flow of human brain related to excitations or activities of human body. Therefore, choosing the window size as well as the order of the filter is very important and also depends on each typical case. For this reason, the SG with the window size of 11 and the order of 3 was applied (see **Figures 8A,B**).

After smoothing Oxy-Hb signal by using the SG filter, the features of Oxy-Hb signal corresponding to hand tapping are extracted using a PR algorithm. In this case, the PR algorithm with the order-5 polynomial produces six coefficients and its equation is represented as follows:

$$y = h\_5 \mathbf{x}^5 + h\_4 \mathbf{x}^4 + h\_3 \mathbf{x}^3 + h\_2 \mathbf{x}^2 + h\_1 \mathbf{x} + h\_0 \tag{22}$$

where *x* represents time from 0 to 7 s with the resolution of 0.07.

The fact is that choosing the window size as well as the order of the polynomial plays an important role due to avoidance of loosing information of signals. In **Figures 8**, **9**, the red Oxy-Hb signals are the smoothed signals, in which the window size and the order were carefully calculated and chosen so that the peaks of the signals removed after filtering do not affect consequences on the analysis.

This equation was applied to determine the regressed Oxy-Hb signals of the channels 2, 5, 6, 9 (left hemisphere) and the channels of the right hemisphere, 12, 15, 16, 19. Thus, the obtained results of the RH tapping and the LH tapping as showed in **Figures 10A,B** are compared together. However, these features of the Oxy-Hb signals obtained at two hemispheres are very hard to distinguish between are the right tapping and the left tapping. For this reason, training data, which are coefficients of the regressed polynomials as shown in **Table 1**, were applied to identify hand tapping tasks. In particular, in each time of hand tapping, Oxy-Hb concentration changes of two hemispheres allow us obtain the regressed coefficients using the PR algorithm. Moreover, six coefficients of each channel as shown in **Table 2** are arranged to be a vector. For classification of hand tapping tasks, the vector was employed to the algorithms such as the ANN or SVM for training data.

Assume that *vr* is the vector of the RH tapping and the vector of the LH tapping is *vl*. In one run of experiment, the subject performed a hand tapping task 20 times, in which 10 times for the LH tapping and 10 times for the RH one. Therefore, a set of the LH tapping coefficients *Sl* includes 10 vectors (from *vl*<sup>1</sup> to *vl*10) and that of the RH tapping coefficients *Sr* is 10 vectors (from *vr*<sup>1</sup> to *vr*10). With 80 sample vectors obtained from subjects, the recognition algorithm was worked out by splitting the sample vectors to be 4 runs of 20-fold cross recognition. For identifying the LH tapping, one used 9 vectors of the *Sl* set combined with the 10 vectors of the *Sr* set and the remaining vector of the *Sl* set is used to be a sample vector for identification. In the case of identifying the RH tapping, 9 vectors of the *Sr* set combined with the 10 vectors of the *Sl* and the remaining one is used to be the sample vector for identifying. As known, Oxy-Hb signals obtained from human brain have many noises and artifacts. Therefore, identifying hand tapping tasks corresponding to Oxy-Hb concentration changes is not easy. For this reason, the identification algorithms such the ANN and SVM are reliable in this research. In the SVM method, the linear hyperplane was chosen. In each training process, the values α (having 15 values of α) are produced, also there are 15 support vectors *w* (each vector *w* is 48 elements) and *b* is 0.068. In similarity, the ANN algorithm with the hidden layer of 100 nodes was applied to obtain the training result, in which the goal of training is set up of 0.001 and the number of epochs is 5000.

From the data sets of the hand tapping tasks, the SVM algorithm was applied for learning to analyze data and recognize patterns. In understanding this SVM training algorithm, data vectors from the hand tapping tasks are given the input of the classifier with a hyperplane which forms two possible classes of the output. In this method, experimental results to the LH tapping of Subject-1 and Subject-3 are the same and Subject-2 showed the lower performance with just 72.5% of the accuracy compared with 82.5% of Subject-1 and Subject-3 as shown in **Table 3**. While basically its results are the same to that of tapping the RH side.

The ANN algorithm used for identifying hand tapping tasks here consists of one input layer, one hidden layer of 100 nodes and the output layer with two nodes. In addition, the second method in this paper is one of recognition methods which have been applied in recent years. Although this algorithm is used very popular for recognition problems, it still uses here due to giving the good performances and also being a reliable method. The result is that classification using the ANN method gave the around 83% performance of tapping the RH side is higher than the performance of around 73% for the RH tapping as shown in **Table 4**.

**FIGURE 10 | Regressed polynomial corresponding to hand tapping. (A)** Regressed polynomial of 8 channels of the left hand tapping. Blue curves are the signals on the left brain side and red curves are of the right

brain side. **(B)** Regressed polynomial of 8 channels of right hand tapping. Blue curves are the signals on the left brain side and red curves are of the right brain side.



All the results of hand tapping tasks obtained here have the accuracy of more than 70%. In this research, two methods were applied to find the best one. The first method is that the SVM algorithm is used for recognition on three subjects and produce different performances. In particular, Subject-3 with tapping the RH has the best result with over 80% of the accuracy, while the accuracy of Subject-2 is only 75% for the case of the RH tapping and 72.5% for that of the LH tapping. While the second method


**Table 4 | Experiment result of 3 subjects with ANN.**


using the ANN is that the accuracy in the case of the RH tapping is equal to or greater than 82.5%. In particular, Subject-1 has the best accuracy of the right tapping, while the accuracy of the left tapping just stops at 72.5% for both Subject-1 and Subject-3. Moreover, the result is that Subject-2 has the best accuracy in the case of the LH tapping. It is clear that two methods used in this research give a little bit different performance. In general, the SVM method is better in this case. We also observed that Subject-2 produced the best accuracy in the case of the LH tapping. The right tapping accuracy is greater than the left tapping of all three subjects in the case of the SVM is performed. Each classification network has a different response to the same inputs. It can give the good accuracy in some cases of the right tapping, but it can show the poor in others. Because of this selective problem, one should more carefully choose the classification network type to obtain the higher performance.

In recent years, researchers have proposed different algorithms in exploring body activities related to human brain. The poor spatial resolution of NIRS made it difficult to distinguish two closely located cortical areas from each other. A combination of the multi-channel NIRS and a Center of Gravity (CoG) approach widely accepted in the field of Transcranial Magnetic Stimulation (TMS) could be used to discriminate between closely located cortical areas activated during hand and foot movements of the subject (Koenraadt et al., 2012). Hemodynamic responses were measured using a NIRS system of 8 channels. For estimating adapt of Oxyhemoglobin (OHb) and Deoxyhemoglobin (HHb), a CoG algorithm was determined for each condition using the mean hemodynamic responses and the coordinates of the channels. Therefore, significant hemodynamic responses were found for hand and foot movements. This is the interesting methods which can be applied to develop for identifying hand tapping. Based on this information, the proposed algorithms in our research can be improved with some thresholds to find out which channel gives the valuable information. The order of the filter we had chosen here belongs to the pulses time of moving hand up and down. Thus, the method to quantitatively estimate the start and end timing of the hand movement using the neural network was proposed.

In (Muroga et al., 2006), the authors measured regional cerebral blood flow during tapping movement of the RH using NIRS technique. The following tendencies of total-Hb were observed, in which Hb increased within 10 s from the movement start time, decreased within 10 s from the movement end time. The direction of arm force from hemoglobin concentration changes measured by using NIRS technique was discriminated. A Self-Organizing Map (SOM) was used to classify the force direction information obtained from the NIRS signals. The results confirmed that the direction of the arm force is discriminable through the NIRS signal. In the simple classification approach, the average discrimination rate gave the performance of 87.5% for two directions. The experimental results showed that the NIRS signal from arm force contained information related to the force directions (Sato et al., 2009). This research is from our research about the proposed methods and experimental tasks. While the SOM method, possibly called the ANN, evaluated the arm force directions with the 87.5% large performance is a little bit higher than that compared with the SVM and the ANN for the LH and RH tapping tasks. This is one of methods which we need to apply for our experimental tasks to determine the best one.

In using NIRS technology, the local distribution of fingers (right thumb and ring finger, respectively) was distinguished to hemodynamic responses on Somatosensory cortex by the electrical stimuli intensity (SI), whose results showed in good accordance with the anatomical arrangement of hand area (Xu et al., 2007). Another application is that in NIRS-based brain activation mapping, a novel real-time NIRS signal analysis framework based on the General Linear Model (GLM) and the Kalman estimator was proposed (Ge et al., 2010). A set of simulated data was processed using the proposed framework. The results obtained suggested that the method can effectively locate brain activation areas in real-time, thereby demonstrating its potential for real-time NIRS-based brain imaging applications. Both these researches, the authors were proposed the same experiment with finger movements using different methods. It is clear that the NIRS technology is not only used to distinguish hand tapping tasks in this paper, but also applied for finger movement tasks.

From the previous researches, we have realized that the proposed algorithm can be accompanied with other algorithms for finding more accuracy. The NIRS technology has been used to obtain Oxy-Hb signals in recent years. However, these Oxy-Hb signals always exist noises and artifacts due to subject movements, noisy environments, human biological changes and others. Proposing a good method for estimating Oxy-Hb concentration changes related to brain activities is always necessary to researchers. In particular, the poor resolution in spatial domain needs to be overcome and also applications in real time are an interesting field for research developments using the NIRS technology.

#### **CONCLUSION**

In this paper, original brain signals of hand tapping tasks were filtered by the Savitzky–Golay filter to produce the smooth signals. Moreover, the smoothed signals of the LH and RH tapping tasks corresponding to Oxy-Hb concentration changes in human brain were analyzed using the PR algorithm. Based on different

#### **REFERENCES**


based on the high order polynomial regression," in *The IEEE Conference on Geoscience and Remote Sensing Symposium*, (Barcelona), 3100–3103.


coefficients of the curves obtained from the PR algorithm, the ANN and SVM algorithms were employed to validate Oxy-Hb data for the recognition of the hand tapping times. Experimental results with hand tapping times showed that one could distinguish the LH or RH tapping tasks of the subject. In addition, from the obtained results of two methods, it was realized that the SVM algorithm is faster than the ANN one in term of time recognition. Based on the proposed algorithms, future work is that experiments will be developed on many subjects to investigate more accuracy and to apply for treatment and rehabilitation.

#### **ACKNOWLEDGMENTS**

We would like to thank Vietnam National Foundation for Science and Technology Development-NAFOSTED for supporting research grant No. 106.99-2010.11. Furthermore research was partly supported by a research fund from the Vietnam National University in Ho Chi Minh City. Finally, an honorable mention goes to our volunteers, families and friends for their supports on us in completing this project.


Rantanen, P., et al. (2006). EEG and MEG brain-computer interface for tetraplegic patients. *IEEE Trans. Neural Syst. Rehabil. Eng.* 14, 190–193. doi: 10.1109/TNSRE.2006.875546


removal for functional near infrared spectroscopy," in *The 32nd Annual International Conference of the IEEE EMBS*, (Buenos Aires), 5–8. doi: 10.1109/IEMBS.2010.5626589


767–791. doi: 10.1016/S1388-2457 (02)00057-3


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 19 July 2013; paper pending published: 29 July 2013; accepted: 12 August 2013; published online: 02 September 2013.*

*Citation: Thanh Hai N, Cuong NQ, Dang Khoa TQ and Van Toi V (2013) Temporal hemodynamic classification of two hands tapping using functional near—infrared spectroscopy. Front. Hum. Neurosci. 7:516. doi: 10.3389/ fnhum.2013.00516*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Thanh Hai, Cuong, Dang Khoa and Van Toi. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### NIRS-measured prefrontal cortex activity in neuroergonomics: strengths and weaknesses

#### *Gérard Derosière1,2, Kévin Mandrick1, Gérard Dray3, Tomas E. Ward2 and Stéphane Perrey1 \**

*<sup>1</sup> Movement to Health, Montpellier-1 University, EuroMov, Montpellier, France*

*<sup>2</sup> Biomedical Engineering Research Group, National University of Ireland Maynooth, Co Kildare, Ireland*

*<sup>3</sup> LGI2P, Ecole des Mines d'Alès site EERIE, Parc Scientifique Georges Besse, Nîmes, France*

*\*Correspondence: stephane.perrey@univ-montp1.fr Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

**Keywords: mental workload, human-computer interaction, cognitive science, passive BCI, attention**

### **INTRODUCTION**

Contemporary daily life is more and more characterized by ubiquitous interaction with computational devices and systems. For example, it is commonplace for a person walking a busy street, to be engaged in conversation with a distant person using telephony, while simultaneously receiving directions via a GPS-enabled web application on their mobile device. This overwhelming increase in humancomputer interactions has prompted the need for a better understanding of how brain activity is shaped by performing sensorimotor actions in the physical world. In this context, neuroergonomics aims at bridging the gap between the abundant flow of information contained within a person's technological environment and related brain activity in order to adapt machine settings and facilitate optimal human-computer interactions (Parasuraman, 2013). One way to achieve this goal consists in developing adaptive systems. In neuroergonomics, adaptive automation relies on passive brain-computer interfaces (BCI) capable of spotting brain signatures linked to the operator's cognitive state in order to adjust in real-time the operator's technological environment. With the growing area of interest in this topic, the need for neuroimaging methods properly suited to ecological experimental settings has risen. In this vein, near-infrared spectroscopy (NIRS) presents some advantages as compared to other neuroimaging methods.

In this opinion article, we first concentrate on the benefits of utilizing NIRS for investigation in neuroergonomics. Recent neuroergonomics investigations have used NIRS recordings in a number of laboratories (e.g., Ayaz et al., 2012; Mandrick et al., 2013a,b). It is particularly worth noting that most of these investigations have reported NIRS data from the prefrontal cortex (PFC). We provide a brief review of these recent studies and their impact in the field by presenting a detailed analysis of the applicability of NIRS-measured PFC activity to discriminate cognitive states in real life environments. In this paper, we will address two main questions: are NIRS-derived hemodynamic variables sufficiently sensitive to changes in sustained attention when measured over the PFC area? Are these measures useful for delineating different levels of mental workload?

#### **NIRS-MEASURED PREFRONTAL CORTEX ACTIVITY IN NEUROERGONOMICS: STRENGTHS**

In 1977, Jöbsis published the first paper exploiting near-infrared light to investigate hemodynamic and oxygenation changes in the human brain. Since then, the technique has garnered immense interest across a multitude of fields of research in neuroscience including, recently, neuroergonomics. What makes this technique attractive for neuroergonomics investigators? The answer lies in a set of technical advantages offered by NIRS compared to other neuroimaging methods when performing experiments requiring ecological validity.

One such advantage is that subjects can engage in experimental tasks without the noise and movement limitation constraints associated for instance with magnetic resonance imaging (MRI) investigations. In the same vein, the possibility of conducting experiments with the subjects in a sitting or standing position is a specific advantage of NIRS, since lying down in the magnet has been demonstrated to increase the risk of subject drowsiness (Kräuchi et al., 1997) and so the level of attention. In addition, the MRI environment severely limits the establishment of ecologically valid experimental conditions despite the efforts of simulation paradigms [e.g., virtual reality in Calhoun and Pearlson (2012)]. Instead, NIRS or electroencephalography (EEG) may be exploited to counteract all these magnet-related issues. As stipulated by Di Nocera et al. (2007), although specific aspects of the EEG are sensitive to mental workload their robust acquisition and analysis in real-time is still a problematic area. Also, unlike EEG, NIRS recordings are not affected by electrooculographic or facial electromyographic activity and environmental electrical noise—which are undoubtedly ubiquitous in human-computer interactions. Additionally, investigators having exploited both NIRS and EEG techniques may attest that the former technique is less sensitive to movement-related artifacts than the latter. NIRS signals are, however, not totally free of artifacts. Solovey et al. (2009) investigated the effect of physical behaviors inherent to computer (e.g., mouse) usage during NIRS acquisition and proposed guidelines to further limit and correct artifact sources while using this modality in a neuroergonomics context. All the aforementioned technical advantages of NIRS are at the basis of its increasing use in neuroergonomics (e.g., Ayaz et al., 2012).

NIRS in neuroergonomics has been focused predominantly on the PFC. Focusing on a single specific cortical area stemmed from the goal of reducing the number of measurement channels required at the scalp level which is linked to the aim of developing ambulatory passive BCI. However, once this measurement simplicity is recognized, why should NIRS probes be placed over the PFC rather than, say, over parietal cortices? Firstly, the PFC is well-known as being involved in a large amount of cognitive and motor activities (Miller and Cohen, 2001) and is therefore a good candidate for the investigation of the interrelationships between cognition, action and the physical world. Secondly, there is an undeniable practical benefit of setting-up NIRS probes over this hairless scalp area as compared to other—more dorsal—scalp areas, since the presence of hair may impact on both photon absorption (Murkin and Arango, 2009) and the coupling of the probes with the underlying scalp.

The use of NIRS offers then, technical advantages in the field of neuroergonomics, and especially so, when probes are placed over the PFC. However, one could be skeptical regarding the focus on one particular area over others as it may reduce the amount of relevant information for quantifying the operator's cognitive state. This concern gives rise to the following question: does PFC activity measurement through NIRS-derived hemodynamic variables allow for quantifying the cognitive-states of the people/operator with sufficient sensitivity in real life environments?

#### **RELIABILITY OF NIRS-MEASURED PREFRONTAL CORTEX ACTIVITY FOR QUANTIFICATION OF OPERATOR'S COGNITIVE-STATE**

A primary issue in neuroergonomics concerns the assessment of mental workload. Mental workload reflects "*how hard the brain is working to meet task demands*" (Ayaz et al., 2012). Since excessive or insufficient mental workload can be associated with decreased efficiency and safety of human-computer interactions, mental workload has to be assessed for designing new systems or adapting them in real time. Based on NIRS-measured PFC activity, Ayaz et al. (2012) investigated the sensitivity of HbDiff—corresponding to the difference between oxy- (O2Hb) and deoxyhemoglobin (HHb) values—for distinguishing three levels of mental workload during an Air Traffic Control task. In their study, augmenting the number of aircraft—6, 12, or 18—that the subjects had to manage in a given time resulted in three levels of difficulty. For each level, the changes (-) in PFC activity as the difference between HbDiff values measured at the end of the task and during the pre-task baseline were computed. The authors found significant increases in -HbDiff as a function of the level of difficulty. One of the conclusions drawn was that NIRS-based measurement of PFC activity appeared sensitive to large differences in task difficulty while sensitivity to smaller differences in task difficulty would have to be explored further using finer graduations in task level. However, the capacity of NIRS-based measurement of PFC activity to discriminate large differences in task difficulty has not been unequivocally proven and other published research suggests problems, especially for higher levels of mental workload. Using a similar paradigm, Izzetoglu et al. (2004) showed that the -HbDiff measured over the PFC increased when considering conditions involving 6, 12 or 18 aircraft, but failed to find any increase in -HbDiff when the number of aircraft was increased to 24. By manipulating three degree of difficulty of an arithmetic task, Mandrick et al. (2013a,b) provided further evidence of this finding. Subjective measures (i.e., increase in the perceived difficulty and NASA-TLX scores) and behavioral results (i.e., increase in reaction times and rate of errors) confirmed that three distinguishable levels of workload were produced. However, Mandrick et al. (2013a,b) were only able to find differences in -O2Hb between the "*easy*" and "*medium*" levels of difficulty but not between the "*medium*" and "*difficult*" levels of difficulty. By computing the slope value of the linear regressions fitting for O2Hb signals from the beginning to the end of the task, the authors could distinguish differences in -O2Hb patterns between the "*medium*" and "*difficult*" levels of difficulty. In summary, NIRS-measured PFC activity has been demonstrated to be able to distinguish between large changes in difficulty (Ayaz et al., 2012), especially for low to moderate levels of workload. At higher levels of workload, a plateau effect was found when exploiting -O2Hb or -HbDiff suggesting that alternative data analyses should be exploited (e.g., the slope method). Within active BCI systems—where subjects have to intentionally control external devices (Coyle et al., 2007)—the slope index has been identified as a discriminatory feature of the user's cognitive state (Power et al., 2011; Faress and Chau, 2013). Its application and implementation as passive BCI for quantifying the operator's mental workload constitute a future step in neuroergonomics.

Another area of research in neuroergonomics concerns the monitoring of sustained attention. In this case, the interest is to capture relevant brain signatures so as to detect performance breakdown characterizing the so-called time-on-task (TOT) effect—during sustained attention tasks. In this context, some NIRS-based PFC activity investigations revealed the sensitivity of the method to attention degradation. For instance, Li et al. (2009) showed significant changes in NIRSmeasured O2Hb, total hemoglobin (tHb) and regional oxygen saturation (rSO2) over the left PFC in parallel to TOT effect development during a prolonged driving task of 3 h duration. In the same way, we recently demonstrated (Derosiere et al., 2013) that the O2Hb variable was sensitive to the TOT effect development—in the form of increases over the lateral left and right PFC and decrease over the medial part of the PFC—during a simple reaction time task of 30 min duration. When considered together, these results suggest that certain variables measured using NIRS over PFC may be sensitive to the TOT effect development. These variables include O2Hb and tHb. Finally, the sensitivity of the rSO2 variable to the TOT effect should be questioned since, in some cases, it has been found to remain stable despite performance degradation (Helton et al., 2007; De Joux et al., 2013). In these studies, the shorter duration of sustained attention than in Li et al. (2009) may explain the discrepancies in the results. To put it succinctly, the O2Hb and tHb variables as measured over the PFC—can be considered sensitive to attention decrement regardless of task duration while rSO2 may be sensitive to prolonged tasks only.

Finally it is worth noting that the scope of neuroergonomics is not limited to stressful conditions exclusively but includes positive mental states—such as pleasure—as well. In this vein, hedonomics is defined as "*that branch of science which facilitates the pleasant or enjoyable aspects of human-technology interaction*" (Hancock and Szalma, 2006). The aim is to develop interfaces fostering the emergence of flow states (Csikszentmihalyi, 1990) in which operators are fully engaged in a task while information processing is fluid and almost automatic rather than effortful and controlled. Recently, Peck et al. (2013) demonstrated that, based on changes in -O2Hb, users' movie preferences could be reliably classified. Such a result is encouraging and suggests that NIRS-measured PFC activity may allow for the adaptation of computer interfaces based on the operator's design preferences.

#### **CONCLUSION AND PERSPECTIVES FOR NIRS IN NEUROERGONOMICS**

The investigation of cortical activity by NIRS presents real advantages especially when measurement in ecologically valid conditions is required. Further, the PFC is of interest for NIRS investigation in neuroergonomics due to its acknowledged role in linking cognition, action and the physical world. Neuroergonomics studies have confirmed that NIRS-measured PFC activity can be useful for distinguishing changes in the operator's cognitive state. However, some NIRS-measured hemodynamic variables appeared relatively insensitive to certain changes in mental workload or attentional state. While alternative data analyses method (e.g., the slope index) can be proposed to solve some of these issues, further investigation is required to determine the relevancy of each NIRS-measured hemodynamic variable, as taken independently or in a combined manner, for distinguishing changes in the operator's cognitivestate.

#### **REFERENCES**

Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., and Onaral, B. (2012). Optical brain monitoring for operator training and mental workload assessment. *Neuroimage* 59, 36–47. doi: 10.1016/j.neuroimage.2011.06.023


for functional near-infrared spectroscopy-derived cerebral hemodynamic responses. *Int. J. Ind. Ergon*. 43, 335–341. doi: 10.1016/j.ergon.2013.05.003


*Received: 28 August 2013; accepted: 29 August 2013; published online: 19 September 2013.*

*Citation: Derosière G, Mandrick K, Dray G, Ward TE and Perrey S (2013) NIRS-measured prefrontal cortex activity in neuroergonomics: strengths and weaknesses. Front. Hum. Neurosci. 7:583. doi: 10.3389/ fnhum.2013.00583*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Derosière, Mandrick, Dray, Ward and Perrey. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### Fusion of fNIRS and fMRI data: identifying when and where hemodynamic signals are changing in human brains

#### *Zhen Yuan1 \* and JongChul Ye2*

*<sup>1</sup> Bioimaging Core, Faculty of Health Sciences, University of Macau, Macau SAR, China*

*<sup>2</sup> Department of Bio and Brain Engineering, KAIST, Daejeon, South Korea*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Liangzhong Xiang, Stanford University, USA Yao Sun, Wake Forest Baptist Medical Center, USA*

#### *\*Correspondence:*

*Zhen Yuan, Bioimaging Core, Faculty of Health Sciences, University of Macau, Av. Padre Tomás Pereira, Taipa, Macau SAR, China e-mail: zhenyuan@umac.mo*

In this study we implemented a new imaging method to fuse functional near infrared spectroscopy (fNIRS) measurements and functional magnetic resonance imaging (fMRI) data to reveal the spatiotemporal dynamics of the hemodynamic responses with high spatiotemporal resolution across the brain. We evaluated this method using multimodal data acquired from human right finger tapping tasks. And we found the proposed method is able to clearly identify from the linked components of fMRI and fNIRS where and when the hemodynamic signals are changing. In particular, the estimated associations between fNIRS and fMRI will be displayed as time varying spatial fMRI maps along with the fNIRS time courses. In addition, the joint components between fMRI and fNIRS are combined together to generate full spatiotemporal "snapshots" and movies, which provides an excellent way to examine the dynamic interplay between hemodynamic fNIRS and fMRI measurements.

**Keywords: fMRI, fNIRS, independent component analysis, multimodal imaging methods, cognitive neurosciences**

#### **INTRODUCTION**

Functional near infrared spectroscopy (fNIRS) offers unsurpassed high temporal resolution and provides quantitative hemodynamic information for both oxyhemoglobin (*HbO2*) and deoxyhemoglobin (*HbR*), which plays an important role in the *in vivo* study of cognitive processing in the human brain (Jobsis, 1977; Cope and Delpy, 1988; Hoshi, 2003; Singh et al., 2005; Huppert et al., 2009; Ye et al., 2009; Yuan et al., 2010a,b; Brunno et al., 2011; Egetemeir et al., 2011; Gagnon et al., 2012; Yuan, in press). Similar to its fNIRS counterpart, functional magnetic resonance imaging (fMRI) is also a non-invasive imaging method that measures the hemodynamic responses to even-related neural activity with excellent spatial resolution and low temporal resolution. To take advantages of the complementary information from these two imaging modalities, two broad methods for fNIRS and fMRI/MRI integration have been developed for different clinical cases: (1) "spatial constraint," in which spatial information from fMRI/MRI images are utilized to aid diffuse optical imaging based on fNIRS measurements (Carpenter et al., 2007; Ferradal et al., 2013); (2) "temporal correlation," where fMRI bold signals are processed to generate the correlation with *HbO2* and *HbR* concentration changes converted from fNIRS recordings (Cui et al., 2011; Tak et al., 2011; Gagnon et al., 2012). However, so far the linking between *HbO2*/*HbR* signals and fMRI spatial maps has not been extensively investigated, which represents one of the main challenges for fNIRS and fMRI fusion. Therefore, it is crucial for us to develop new imaging techniques to reveal the connections between these two measurements so that we are able to examine the dynamic interplay between space and time of hemodynamic responses with high spatiotemporal resolution.

Interestingly joint independent component analysis (jICA) method has been developed to compute the linked temporally independent event-related potential (ERP) components and spatial independent fMRI components, which enables inferences to be made using estimated associations between fMRI sources and ERP electromagnetic sources (Calhoun et al., 2006a,b; Sui et al., 2009). However, what has not been tried is a joint estimation of the temporal parts of fNIRS waveforms and the spatial maps of fMRI images. In this study, the jICA is extended to identify the spatiotemporal decompositions composed of fMRI spatial components indicating where the hemodynamic signals are changing and fNIRS components indicating when the hemodynamic signals are changing. The fNIRS-fMRI fusion method will involve calculating for given stimuli, the connection between the timelocked fNIRS waveforms and fMRI activation maps for all participants or different measurement sections from a single subject. It is anticipated that the results derived from jICA could be visualized by computing spatiotemporal "snapshot," which provides an effective way to examine the dynamic interplay between fNIRS and fMRI hemodynamic sources.

### **METHODS**

In terms of jICA, the joint spatial and temporal independences of fNRIS and fMRI are assumed to satisfy the following generative model for the data (Calhoun et al., 2006a,b),

$$\mathbf{X}^{\text{f\$NIRS}} = \mathbf{A}\mathbf{S}^{\text{f\$NIRS}}, \mathbf{X}^{\text{f\$MRI}} = \mathbf{A}\mathbf{S}^{\text{fMRI}} \tag{1}$$

in which *X*fNIRS is the group data from the chromophore concentration change of *HbO2* or *HbR* for *n* subjects/*n* sections of a single subject and *X*fNIRS = - *X*fNIRS <sup>1</sup> , *<sup>X</sup>*fNIRS <sup>2</sup> , ... *<sup>X</sup>*fNIRS *n T* , *X*fMRI is the group data

**FIGURE 1 | (A)** The data review for -*HbO2* measurements; **(B)** the data review for -*HbR* measurements; **(C)** Channel configurations along the scalp for right finger tapping tasks; **(D)** the section averaged fMRI image (left side),


**FIGURE 2 | jICA decomposition of** *-HbO2* **and fMRI joint data for right finger tapping tasks: three components were found to be significantly correlated with finger tapping tasks [Independent components (ICs) 2, 4, and 5] while two were well-correlated with the physiology and body movement noise (ICs 1 and 3).** Each of these components is shown in a separate panel in the figure: IC2 **(A)**, IC4 **(B)**, IC5 **(C)**, IC1 **(D)** and IC3 **(E)**. The fMRI maps are thresholded at |*Z*| > 1.5 for display purposes. *(Continued)*

#### **FIGURE 2 | Continued**

The averaged event-related -*HbO2* time course is shown in yellow (the same for all figures) and the -*HbO2* component is plotted in cyan. Positive (orange) and negative (blue) *Z*-values are shown in the image. The axes (bottom) illustrate the time scale, in ms, whereas the scale (left) records chromophore concentrations in μM for the figures on the left column. The scale for the figures on the right column shows the bold signal intensity. LM, Left primary motor cortex; SMA, Supplementary motor area; P, Parietal cortex; V, Visual cortex; BP, Body movement/physiology noise.

from fMRI for *n* subjects/*n* sections of a single subject and *X*fMRI = - *X*fMRI <sup>1</sup> , *<sup>X</sup>*fMRI <sup>2</sup> , ... *<sup>X</sup>*fMRI *n T* , *S*fMRI are the fMRI sources and *S*fMRI = - *S*fMRI <sup>1</sup> , *<sup>S</sup>*fMRI <sup>2</sup> , ... *<sup>S</sup>*fMRI *n T* , and *S*fNIRS are the fNIRS sources and *S*fNIRS = - *S*fNIRS <sup>1</sup> , *<sup>S</sup>*fNIRS <sup>2</sup> , ... *<sup>S</sup>*fNIRS *n T* . The shared lined mixing matrix *A* are written,

$$\mathbf{A} = \begin{bmatrix} a\_{11} \ a\_{12} \ \dots \ a\_{1n} \\ a\_{21} \ a\_{22} \ \dots \ a\_{2n} \\ \ \cdot \quad \cdot \quad \cdot \quad \cdot \\ a\_{n1} \ a\_{n2} \ \dots \ a\_{nn} \end{bmatrix} \tag{2}$$

It is note Equation 1 can be rewritten as a single matrix equation,

$$\begin{bmatrix} X\_1^{\text{�fINRS}} \ X\_1^{\text{fMRI}}\\ X\_2^{\text{fINRS}} \ X\_2^{\text{fMRI}}\\ \dots \ \dots \\ X\_n^{\text{fINRS}} \ X\_n^{\text{fMRI}} \end{bmatrix} = \mathcal{A} \begin{bmatrix} S\_1^{\text{fINRS}} S\_1^{\text{fMRI}}\\ S\_2^{\text{fINRS}} S\_2^{\text{fMRI}}\\ \dots \ \dots \\ S\_n^{\text{fINRS}} S\_n^{\text{fMRI}} \end{bmatrix} \tag{3}$$

We employ the infomax ICA method for jICA of Equation 3, which utilizes a gradient ascent iteration algorithm to maximize the entropy of the output of a single layer neural network (Bell and Sejnowski, 1995). The resulting updated equation for the algorithm to calculate the shared unmixing matrix *W* (i.e., the inversion of *A*), the fused independent fNIRS sources *u*fNIRS and fMRI sources *u*fMRI is as follows,

$$
\Delta \mathbf{w} = \eta \left\{ I - 2 \mathbf{y}^{\text{fNIRS}} (\boldsymbol{\mu}^{\text{fNIRS}})^T - 2 \mathbf{y}^{\text{fMRI}} (\boldsymbol{\mu}^{\text{fMRI}})^T \right\} \mathbf{W} \tag{4}
$$

in which *y*fNIRS = *g*(*u*fNIRS), *y*fMRI = *g*(*u*fMRI), *u*fNIRS = *WX*fNIRS, *u*fMRI = *WX*fMRI, and *g*(*x*) = 1/(1 + *e*−*x*) is the non-linearity in the neural network. The initial value for *W*, *W*(0) is a matrix composed of random vectors.

The jICA method generally doesn't provide us the details on how the components between fNIRS and fMRI interact with one another. To achieve this, the spatiotemporal "snapshots" of the most significant components are generated in two ways (Calhoun et al., 2006a,b). First we need to calculate the linear combination of the fMRI components weighted by their joint fNIRS time courses for a specific point in time. If the *n* spatial (fMRI) and temporal (fNIRS) components are given by *S* = - *s*<sup>1</sup> *s*<sup>2</sup> ... *s<sup>n</sup>* , and *T* = - *t*<sup>1</sup> *t*<sup>2</sup> ... *t<sup>n</sup>* , where *t<sup>i</sup>* is a *T* × 1 vector containing the fNIRS time points and *s<sup>i</sup>* is a *V* × 1 vector represents the *V* brain voxels, the fMRI movie is computed as *M*fMRI = |*T*| × *ST*. It is noted the absolute value is utilized here because the joint components are fused using a single parameter. And a change in

the amplitude of the fMRI component is directly linked to the change in the fNIRS component by this parameter. Meanwhile the fNIRS movie is estimated by *M*fNIRS = *T* × |*S*| *<sup>T</sup>*, in which the time course for a given fMRI voxel is computed.

the figures on the left column. The scale for the figures on the right column shows the bold signal intensity. LM, Left primary motor cortex; SMA, Supplementary motor area; P, Parietal cortex; V, Visual cortex; BP,

#### **RESULTS**

**FIGURE 3 | Continued**

Body movement/physiology noise.

#### **BEHAVIOR TASKS AND fNIRS-fMRI RECORDINGS**

The fNIRS tests are implemented with a block design for a right finger tapping task. The experiment is performed with a 24 chnnel fNIRS system (Oxymon MKIII, Artinis), which has 8 sources, 8 detectors, and 24 channels. In this system, two continuous wave lights at wavelengths 781 and 856 nm are emitted at each source fiber. In the case of block design for right finger tapping tasks, the onset time for the first trigger was at 42 s, then followed by a 21 s period of activation alternated with a 30 s period of rest. This was repeated 10 times for the subject. As such, the total recording time was 552 s. During the task period, subject was instructed to perform a finger flexion and extension action repeatedly. Data segmentation, which is also known as epoching in signal processing, is utilized to chop up the continuous fNIRS data into small time periods. The general way to do this is to extract segments surrounding the event codes from the experiments, e.g., from −35 s prior to the event onset until 35 s after the event code in this study. The original photon density datasets could be downloaded from (http://bisp.kaist.ac.kr/NIRS-SPM/Sample\_data). The converted -*HbO2* and -*HbR* measurements (the unitless differential path length factor *DPF* = 4; sampling rate: 9.75 Hz) are filtered, segmented and plotted in **Figures 1A,B**, respectively. The configurations of 24 channels located on the scalp are also provided in **Figure 1C**. We found channels 8–12 are very unique because their locations are close to the left motor cortex so that they are able to reveal the hemodynamic responses from both the right finger tapping stimuli and other events. As a result, signal averaging is implemented for the datasets from these five channels to generate the representative -*HbO2*/-*HbR* measurements with reduced noise. In particular, five-section (runs 2, 4, 6, 8, and 10) measurements are selected from the representative ten-run fNIRS recordings for further fusion analysis since they show the best signal noise ratio during the stimulus processing.

For fMRI recordings, the fiber length was 10 m to connect the optodes in the MR scanner to the NIRS instrument in the MR control room. A 3.0 T MRI system (ISOL, Republic of Korea) was used to measure the BOLD response. During the blocked task paradigm, the echo planar imaging (EPI) sequence was used with *TR*/*TE* = 3000/35 ms, flipangle = 80◦, 35 slices, and 4 mm slice thickness (Ye et al., 2009). The fMRI data could be downloaded partly from the same website for the same subject or requested from the KAIST lab. Based on the datasets provided, we generated five mean MRI images which are considered as five-section fMRI measurements (Calhoun et al., 2001; Ye et al., 2009).

figure: IC1 **(A)**, IC3 **(B)**, IC2 **(C)**, IC4 **(D)** and IC5 **(E)**. The fMRI maps are thresholded at |*Z*| > 1.5 for display purposes. The averaged event-related *HbR* time course is shown in yellow (same for all figures) and the -*HbR* component is plotted in cyan. Positive (orange) and negative (blue) *Z*-values are shown in the image. The axes (bottom) illustrate the time scale, in ms, whereas the scale (left) records chromophore concentrations in μM for *(Continued)*

Then we will combine the five-section averaged fNIRS recordings in temporal domain and five mean fMRI images in space to extract the joint independent components for hemodynamic fusion. It should be noted here the mean fMRI images are not generated from the raw fMRI images at each TR. As such, the fMRI image for each section may not accurately match its correlated fNIRS measurements in terms of recording times, which may have some influence on the final results. However, the influences should not be that significant since we use mean data from each section for fusion. The five-section averaged fMRI image and -*HbO2*/-*HbR* results are presented in **Figure 1D** for references.

#### **RESULTS OF fNIRS-fMRI FUSION**

The computed joint components are provided in **Figures 2**, **3**, in which the spatial components and regions of brain activity from the fMRI maps are plotted on the right while the correlated temporal joint components along with the mean -*HbO2*/-*HbR* signals are given on the left. We can see from **Figures 2**, **3** that the time courses of temporal -*HbO2*/-*HbR* components correspond well to different peaks on the mean -*HbO2*/-*HbR*

signals. Interestingly, when put together the fMRI maps and fNIRS signals, we found the five spatial joint components correlate very well with the five temporal joint components. For example, the first negative peak of -*HbO2* in **Figure 2A** basically identifies the physiology noise such as visual activity and eye blinks. Also visible for this waveform of the second temporal joint component in **Figure 2A** is a late positive peak. During the positive peak, the hemodynamic activity is mainly found in the left motor cortex. Then the joint components from -*HbO2*/-*HbR* and fMRI are combined together to generate full spatiotemporal movies to more clearly display the dynamic interplay between the fNIRS and fMRI measurements. In particular, two movies are created to show the spatiotemporal dynamics of the fMRI and -*HbO2* fusion, in which **Movie 1** displays the hemodynamic responses of right finger tapping events (composite spatiotemporal joint components 2, 4, and 5) while **Movie 2** basically reveals brain activity from physiology and body movement(composite joint components 1 and 3). The spatiotemporal hemodynamic changes derived from fMRI and -*HbR* fusion are also recorded by **Movies 3**, **4** for right finger tapping tasks (composite joint component 1 and 3) and physiology and body movement noise (composite joint component 2, 4, and 5), respectively.

**Figure 4** shows seven "snapshots" snipped from **Movie 1** produced from fused -*HbO2* and fMRI data for finger tapping tasks. On the left of the "snapshots" a linear sum of the fMRI maps weighted by their respective -*HbO2* time courses are provided while the mean -*HbO2* waveforms combined the estimated -*HbO2* components are plotted on the right of each "snapshot." **Figure 5** plots five "snapshots" captured from fused the fMRI and -*HbO2* data, which basically reveals the spatiotemporal responses of the physiology and body movement noise. Likewise, the "snapshots" from **Movie 3**, **4** produced from fused fMRI and -*HbR* components are given in **Figures 6**, **7** for finger tapping tasks and physiology and body movement noise, respectively.

### **DISCUSSION**

We show, for the first time, a spatiotemporal reconstruction of the human brain's responses with high temporal and spatial resolution by fusing together the fNIRS signals and fMRI images. The jICA is a very key and unique technique for connecting hemodynamic fNIRS temporal signal and hemodynamic fMRI spatial information according to the joint constraints of temporal and spatial independences. Such direct correlation further provides a very valuable tool to localize non-invasively those high

resolution structures that underlines temporally well-resolved fNIRS responses. Meanwhile, deep brain structures, which are not readily detectable with scalp fNIRS alone, can now be identified by the fusion technique when aided by the correlated spatial components from fMRI.

It is observed from **Figure 2** that the first and third -*HbO2 fMRI* joint components basically identifies body movement and negative eye blink activities in pre-frontal lobe while the second, fourth and fifth -*HbO2-fMRI* joint components are fused with hemodynamic activity in left primary motor cortex (PMC), *supplementary motor area (SMA)*, and motor association cortex such as parietal cortex. In particular, the fifth joint component basically identifies the neural activity in the SMA and parietal cortex. These observations are further validated by **Movies 1**, **2**, which show the dynamic interplay between space and time of -*HbO2 fMRI* hemodynamic responses. It is noted **Movie 1** displays the spatiotemporal changes of neural activation patterns for finger tapping tasks. Further, we can see from **Figure 3** that the second, fourth, and fifth -*HbR*-*fMRI* joint components basically reveal body movement and other physiology noise while the first and third -*HbR*-*fMRI* joint components identify the patterns of brain activity in the left motor cortex and motor association cortex. In particular, the first -*HbR*-*fMRI* joint component reveals brain activity in SMA and parietal cortex. These spatiotemporal

findings from fused -*HbR*-*fMRI* data are further processed to generate **Movies 3**, **4**, in which we can see the hemodynamic changes for a finger tapping task with high temporal and spatial resolution, and we can also see clearly how physiology and body movement noise corresponds to the neural activity. Importantly, we can observe from **Figures 2**, **3** as well as **Movies 1**, **3** that strong brain activity mainly occurs in the left PMC, which validates stimuli in the right finger tapping task will yield the activations in the left PMC.

Alternatively, we can see from **Figure 4** the estimated -*HbO2* time courses (i.e., linear combinations of the significant -*HbO2* components) at specific positions in the cerebral cortex. The first "snapshot" in **Figure 4A** is associated with physiology and body movement noise while the second to fourth "snapshots" show the -*HbO2* concentration change during onset and stimulus processing of right finger tapping, which are validated by the spatial maps of fMRI image on the left. We can further observe from **Figure 4** that the fifth "snapshot" captures the maximized spatiotemporal hemodynamic responses while the sixth and seventh ones show the decreased trends of neural activity. So if we examine the different "snapshots" in **Figure 4**, the spatiotemporal dynamics of the right finger tapping are revealed. In addition, the negative and positive peaks of the estimated -*HbO2* time courses in **Figure 5** directly correspond to the physiology and body movement noise though partly they capture the weak and negative hemodynamic responses of right finger tapping in the case of strong body movement.

The -*HbR-fMRI* "snapshots" in **Figure 6** identify SMA and parietal cortex with early peak hemodynamic responses. Later responses occurring in left PMC show peaks of the -*HbR* responses during stimulus processing. In particular, "snapshots" 2–7 display the hemodynamic changes in detail with high spatiotemporal resolution for finger tapping task. **Figure 7** provides the portions of estimated -*HbR* time courses at specific positions of fMRI images, in which we observe the peaks basically corresponding to the activities of physiology noise, eye blinks, and body movement.

In summary, we have used a combination of hemodynamic fNIRS and fMRI data to visualize the neural activation patterns with high spatiotemporal resolution. The fusion techniques have shown the potential by using the joint hemodynamic data to identify unique neural information that cannot be revealed in either technique alone. In particular, we implemented a joint decomposition of fNIRS and fMRI data, which was linked or fused by a common mixing parameter. The present method does not involve the solution of inverse problem for diffuse optical imaging or involve the use of the threshold for the fMRI data. However, we do need to decompose the combined data into specific components by using jICA, which are composed of fNIRS and fMRI portions. Separating the data into joint components will provide us a useful way to examine component specific differences among different patient groups or different stimulus tasks.

We have to point out the present investigation has significant limitations though we are encouraged that the results we have found seem meaningful and valid. For example, we only validate data from a single subject even without accurate measurements from fMRI. In future we would like to examine our method to incorporate different subject groups and complex neural stimuli as well as multiple time points acquired from multiple channels of fNIRS systems.

#### **REFERENCES**


Cope, M., and Delpy, D. T. (1988). System for long-term measurement of cerebral blood and tissue oxygenation on newborn infants by near infra-red transillumination. *Med. Biol. Eng. Comput.* 26, 289–294. doi: 10.1007/BF02447083


**ACKNOWLEDGMENTS**

This research was supported by SRG Grant from University of Macau in Macau. In particular, we thank Prof. Jong Chui Ye at KAIST for providing us the fNIRS/fMRI datasets for our initial methodology study.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/ 10.3389/fnhum.2013.00676/abstract

**Movie 1 |** *-HbO2* **and fMRI fusion movie for finger tapping tasks.**

**Movie 2 |** *-HbO2* **and fMRI fusion movie for physiology and body movement noise.**

**Movie 3 |** *-HbR* **and fMRI fusion movie for finger tapping tasks.**

**Movie 4 |** *-HbR* **and fMRI fusion movie for physiology and body movement noise.**

D. (2009). Homer: a review of time-series analysis methods for near-infrared spectroscopy of the brain. *Appl. Opt.* 48, 280–298. doi: 10.1364/AO.48.00D280


optical spectroscopy in diagnosis of osteoarthritis: a clinical study. *Biomed. Opt. Express* 1, 74–86. doi: 10.1364/BOE.1.000074

Yuan, Z., Zhang, Q., Sobel, E., and Jiang, H. (2010b). High resolution x-ray guided 3D diffuse optical tomography of joint tissues in hand osteoarthritis: morphological and functional assessments. *Med. Phys.* 37, 4343–4354. doi: 10.1118/1.3467755

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 02 August 2013; paper pending published: 31 August 2013; accepted: 26 September 2013; published online: 16 October 2013.*

*Citation: Yuan Z and Ye J (2013) Fusion of fNIRS and fMRI data: identifying when and where hemodynamic signals are changing in human brains. Front. Hum. Neurosci. 7:676. doi: 10.3389/ fnhum.2013.00676*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Yuan and Ye. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## NIRS as a tool for assaying emotional function in the prefrontal cortex

### *Hirokazu Doi, Shota Nishitani and Kazuyuki Shinohara\**

Graduate School of Biomedical Sciences, Nagasaki University, Nagasaki, Japan

#### *Edited by:*

Nobuo Masataka, Kyoto University, Japan

#### *Reviewed by:*

Stephane Perrey, Montpellier I University, France Lei Xi, University of Florida, USA

#### *\*Correspondence:*

Kazuyuki Shinohara, Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto-Cho, Nagasaki 852-8523, Japan e-mail: kazuyuki@nagasaki-u.ac.jp

Despite having relatively poor spatial and temporal resolution, near-infrared spectroscopy (NIRS) has several methodological advantages compared with other non-invasive measurements of neural activation. For instance, the unique characteristics of NIRS give it potential as a tool for investigating the role of the prefrontal cortex (PFC) in emotion processing. However, there are several obstacles in the application of NIRS to emotion research. In this mini-review, we discuss the findings of studies that used NIRS to assess the effects of PFC activation on emotion. Specifically, we address the methodological challenges of NIRS measurement with respect to the field of emotion research, and consider potential strategies for mitigating these problems. In addition, we show that two fields of research, investigating (i) biological predisposition influencing PFC responses to emotional stimuli and (ii) neural mechanisms underlying the bi-directional interaction between emotion and action, have much to gain from the use of NIRS. With the present article, we aim to lay the foundation for the application of NIRS to the above-mentioned fields of emotion research.

**Keywords: NIRS, emotion, prefrontal cortex, hemispheric asymmetry, reward, autonomic nervous system**

#### **INTRODUCTION**

Since being introduced as a research tool, near-infrared spectroscopy (NIRS) has gained wide support and recognition among cognitive neuroscientists, despite having several disadvantages when compared with other non-invasive measurements of neural activation. For instance, NIRS has poor spatial resolution compared with other neuroimaging techniques that measure neurovascular response, such as functional magnetic resonance imaging and positron emission tomography. Similarly, the temporal resolution of NIRS is much lower than that of electroencephalography (EEG) and magnetoencephalography.

The acceptance of NIRS as a novel technique for measuring neural activation might be partly attributable to several unique characteristics. First, NIRS measurement is thought to impose a considerably less severe physical and psychological burden than that of existing neuroimaging techniques. Thus, this technique is particularly advantageous for measuring neural responses in the elderly and infantile populations (Ichikawa et al., 2010; Ozawa et al., 2011; Kida and Shinohara, 2013b). Second, the ease of NIRS measurement makes it a suitable technique for collecting data from a large participant cohort. Third, the measurement of neural activation using near-infrared light is, in principle, more robust with respect to exogenous noise in the environment. Thus, NIRS is considered to be a useful technique for measuring neural activation under less constrained and more ecologically valid settings (Tuscan et al., 2013).

During the past two decades, a number of researchers have used NIRS to produce novel insights about the neural mechanisms underlying various cognitive and perceptual functions. At the same time, the above-mentioned methodological advantages of NIRS have not been fully exploited. For example, the majority

of existing NIRS studies measured brain activation under severely structured settings, with several exceptions (Suzuki et al., 2004).

One field of research, that has much to gain from the use of NIRS, is emotion research. However, there are several concerns that impact the efficacious use of NIRS in this field. In the first part of this mini-review, we discuss findings from NIRS studies on emotion processing with respect to existing views on emotional function in the prefrontal cortex (PFC). Due to the widely accepted convention of functional NIRS studies, we tentatively treat the increase in the oxygenated-hemoglobin concentration [referred to as (oxy-Hb)] as the primary and genuine indicator of cortical activation in this review. At the same time, the possibility that the [oxy-Hb] change reflects the peripheral responses other than the neural activation is discussed in the later part. In the second half of the article, we describe a potential avenue of emotion research where the unique characteristics of NIRS could be gainfully used. We also discuss several practical problems that researchers might face when conducting emotion research using NIRS. This article is not intended to serve as a comprehensive archive of previous findings. Rather, the goal of this article is to lay a foundation for the use of NIRS in several areas of emotion research.

#### **EMOTION PROCESSING IN THE PFC**

The neural mechanisms underlying emotional experience and mood have been the focus of intensive research in both the fields of cognitive neuroscience and clinical psychiatry. According to the now classic "limbic system" model (MacLean, 1949) of neural mechanisms of emotional response, evolutionally ancient subcortical structures generate primitive emotions, such as fear. Emotions originating in the "reptilian brain" are further elaborated in the diverse brain regions of phylogenetically advanced neural circuits, including the PFC. Consistent with this model,

"fnhum-07-00770" — 2013/11/14 — 21:27 — page 1 — #1

more recent studies have identified the PFC as a key region in the induction and regulation of emotional responses (Davidson and Fox, 1982; Damasio, 1996; Rolls, 1996).

Although there is now a wealth of empirical evidencefor the role of the PFC in emotion processing, the exact function subserved by this region is unclear. At the same time, there are several widely accepted views regarding the function of each subregion of the PFC, as summarized in an insightful review by Dalgleish (2004). First, the orbitofrontal region of the PFC has been closely linked to reward processing and reinforcement learning (Rolls, 1996). Specifically, the orbitofrontal PFC appears to play a pivotal role in associating exogenous stimuli with rewarding reinforcers, thereby promoting the assignment of emotional value and saliency. Second, the ventromedial PFC may act as an interface between visceral reactions and higher cognitive function. This view is championed by the influential "somatic-marker hypothesis" (Damasio, 1996), which proposes that somatic markers, as peripheral reactions to stimuli, are processed in the ventromedial PFC as part of a system that guides higher order cognitive functions. Third, the "valence asymmetry hypothesis" of the PFC (Davidson et al., 1990) suggests that the motivational tendency of a living organism can be conceptualized along the dimension of approach/withdrawal. More specifically, when approach motivation is activated, an organism is strongly motivated to pursue an appetitive or rewarding goal. Contrarily, the activation of withdrawal motivation emphasizes avoidance of harmful situations rather than acquisition of rewards. The core proposition of the valence asymmetry hypothesis is that the right PFC activates withdrawal motivation and the left PFC activates approach motivation, thereby enabling adaptive behaviors.

#### **OVERVIEW OF EXISTING NIRS STUDIES**

Several NIRS studies have examined the role of PFC activation in emotion processing. An overview of these findings could be beneficial for several reasons. First, the existing findings might serve as scaffolding upon which researchers could build novel experimental designs. Second, a summary of existing NIRS data might support or oppose established views about the emotional function of the PFC (Davidson and Fox, 1982; Damasio, 1996; Rolls, 1996). Although different types of hemodynamic response are frequently treated as equal [oxy-Hb] reflects different aspects of task-related hemodynamic responses from the concentration of deoxygenated hemoglobin [referred to as (deoxy-Hb)] whose change is supposed to be closely linked to BOLD response (Song et al., 2006). Therefore, a close examination of NIRS data might produce a more comprehensive picture about neural activation during emotion processing. In the following sections, we briefly review the previous findings in light of the above-mentioned theories about the nature of emotion processing in the PFC (Davidson and Fox, 1982; Damasio, 1996; Rolls, 1996). The details about the major studies covered below are summarized in **Table 1**.

#### **SENSITIVITY TO REWARDING STIMULI**

As for the reward sensitivity of the PFC (Rolls, 1996), at least two studies with adult participants have found that [oxy-Hb] in the vicinity of the orbitofrontal region of the PFC increases following exposure to rewarding stimuli, such as tactile stimulation by velvet (Kida and Shinohara, 2013a) and viewing one's infant smiling (Minagawa-Kawai et al., 2009a). Interestingly, an analogous increase in [oxy-Hb] has also been observed in the same region in infants (Minagawa-Kawai et al., 2009a; Kida and Shinohara, 2013b), suggesting that NIRS is a suitable method for measuring reward system activation in participants of varying ages.

#### **PROCESSING OF VISCERAL REACTIONS**

Few NIRS studies to date have specifically examined the neural mechanisms mediating the influence of visceral"somatic"markers on behavior. This is partly due to a technical limitation of NIRS. The ventromedial PFC, which is considered to be the locus of integration between somatic markers and higher order cognitive functions (Damasio, 1996), is located too far from the cranium surface for accurate measurements of activation using NIRS.

With regard to the link between visceral reactions and the PFC, several NIRS studies have succeeded in revealing an association between activation of the PFC and activation of the autonomic nervous system (ANS) in response to emotional stimulation. For example, Tanida et al. (2007) reported that the degree of rightlateralized asymmetry in PFC activation patterns observed during mental stress was positively correlated with the level of activation of the sympathetic nervous system. Likewise, increased [oxy-Hb] has been positively correlated with heart rate change when viewing trauma-related video clips (Matsuo et al., 2003). Furthermore, Moghimi et al. (2012) have linked the steepness of the peak of [oxy-Hb] to a subjectively reported arousal level, which is a relatively coarse, but widely accepted indicator of ANS activation (for similar findings, see Matsuo et al., 2003; Roos et al., 2011). These studies offer partial support for the view that the PFC processes visceral reactions, or somatic markers, associated with exogenous stimuli. At the same time, these findings are mere correlational, and so caution should be exercised in interpreting such data. Furthermore, if causal relations are present, the direction of causality has yet to be clarified.

#### **HEMISPHERIC ASYMMETRY**

Many NIRS studies have used bilateral probes to measure hemodynamic responses, and thus have datasets that are suitable for examining hemispheric asymmetry in the PFC. In these studies, either one of the following criteria was adapted to judge the hemispheric asymmetry in the cortical activations; (1) the significant increase of [oxy-Hb] from the baseline in only one of the hemispheres, or (2) the significant inter-hemispheric difference in the level of [oxy-Hb] change. Several studies have produced evidence in support of the valence-asymmetry hypothesis (Morinaga et al., 2007; Marumo et al., 2009; Tuscan et al., 2013). For example, Morinaga et al. (2007) reported that anticipation of an electrical shock was associated with a greater increase in [oxy-Hb] in the right compared with the left PFC. Furthermore, increases in [oxy-Hb] in the right PFC were positively correlated with the strength of harm-avoidance tendencies in the participants. At the same time, a number of studies have failed to detect hemispheric asymmetry in task-related activation as predicted by the valence-asymmetry hypothesis (Herrmann et al., 2003; Kobayashi et al., 2007; Yang et al., 2007; Hoshi et al., 2011). Much of the empirical support for the valence-asymmetry hypothesis

"fnhum-07-00770" — 2013/11/14 — 21:27 — page 2 — #2



has been obtained by measuring asymmetry in EEG alpha power (Davidson and Fox, 1982; Hagemann, 2004). However, the relationship between the EEG power and transient neurovascular response (as measured by NIRS) is not straightforward. Thus, it is possible that phasic changes in [oxy-Hb] are less sensitive than EEG with respect to changes in approach/withdrawal motivation.

#### **METHODOLOGICAL PROBLEMS IN THE APPLICATION OF NIRS TO EMOTION RESEARCH**

A standardized method of analyzing NIRS signals has yet to be established. Aside from the general challenges that researchers face when analyzing NIRS data, the application of NIRS to emotion research requires additional considerations.

The first problem concerns noise caused by peripheral responses to emotional stimulation. The induction of an emotional state is often accompanied by changes in bodily state, such as the contraction of facial muscles or increased cardiovascular activity. Although mitigated by homeostatic regulation, such changes in heart rate and blood pressure could potentially mask taskrelated hemodynamic responses in NIRS signals. Likewise, the aerobic process of energy consumption associated with muscle contraction may induce significant changes in measurable [oxy-Hb]. Schecklmann et al. (2010) found no systematic relationship between electromyograph signals and [oxy-Hb] during a verbal fluency task. However, the influence of peripheral responses on NIRS signals has been examined only under limited conditions. A regression analysis conducted using simultaneous measurements of NIRS signals and indicators of peripheral response, such as electromyography, heart rate, and blood pressure, could be used to exclude the influence of these factors (Schecklmann et al., 2010).

Another problem is the temporal course of neural activation induced by emotional stimulation. Both the subjective experience of an emotion and the neural responses elicited by emotional stimulation may last longer than the stimulation itself (León-Carrión et al., 2007). Thus, the application of conventional pre-processing methods, such as the correction of global drift (Minagawa-Kawai et al., 2009b) by linear fitting (for example, Takizawa et al., 2008), carries the risk of eliminating important results.

This point was emphasized by Leon-Carrion et al. (2006), who reported that gender differences in cortical activation were present "after" the offset of emotional stimulation. If the researchers had corrected for global drift of the NIRS signal using the period after stimulation offset as the post-stimulation baseline, the observed gender differences may have been obscured. One potential way to address this issue is the use of subjective ratings of emotional state or ANS activation monitoring to continuously track the temporal course of emotional responses for a prolonged duration. This would enable researchers to empirically define the temporal window, and then quantify the hemodynamic responses on the basis of these data.

The third problem is the selection of the appropriate indicator of the cortical activation. In many of the previous studies, the lasting increase of [oxy-Hb] was taken as the indicator of

"fnhum-07-00770" — 2013/11/14 — 21:27 — page 3 — #3

cortical activation, partly because this parameter is quite sensitive to emotional stimulation as seen from the representative data in **Figure 1** collected in our lab (Nishitani et al., 2011). At the same time, Suh et al. (2006) have reported that the direct cortical stimulation induced vary rapid (within 1–2 s after stimulation) and spatially localized increase in [deoxy-Hb], while the total-hemoglobin concentration change in the later latency-range was only poorly localized. Given this, the conventional analysis method, i.e., averaging the [oxy-Hb] level during the whole stimulation period lasting for several seconds, may have impaired to some extent the power to localize the centroid of cortical activation in the previous research. This may partly explain the failure to find consistent pattern of hemispheric asymmetry in the existing NIRS studies (see Hemispheric Asymmetry). In order to avoid this problem, emotion researchers are advised to statistically evaluate the changes in [deoxy-Hb] as well as [oxy-Hb] with high temporal resolution using analytic methods such as point-by-point testing that is widely conducted in the event-related potential studies (Blair and Karniski, 1993).

#### **POTENTIAL APPLICATIONS OF NIRS**

The above-mentioned theories about the role of the PFC in emotion processing (Davidson and Fox, 1982; Damasio, 1996; Rolls, 1996) all share the view that PFC is one of the key regions where the emotional and motivational reaction bias a wide array of behaviors ranging from attentional allocation, motor function, and decisionmaking. The importance of PFC function in emotion processing makes it an essential target for future emotion research. In the next section, we present a brief overview of potential fields of research in which NIRS could aid investigations of emotion processing.

#### **THE BIOLOGICAL BASIS OF INDIVIDUAL DIFFERENCES IN EMOTIONAL RESPONSE**

Several recent studies have indicated that traits associated with sensitivity to reward and stress may modulate vulnerability to pathological conditions such as depression (Pizzagalli et al., 2005; Bogdan et al., 2013). These findings give weight to investigations about biological predisposition determining the PFC response to emotion-inducing stimuli. Such research may help elucidate the causes of individual differences in emotional reactions, and potentially help in identifying risk factors that lead to psychiatric conditions. The use of NIRS to measure PFC activation is invaluable due to the suitability of this technique for measuring cortical activation in a large cohort of participants.

For example, the application of NIRS in the field of genetic neuroimaging may enable researchers to acquire data from larger numbers of participants in an economically feasible way, and consequently increase the reliability of their findings. In recent years, there has been a surge in the number of multidisciplinary studies focused on the link between genetic polymorphism and neural function (Hariri et al., 2005; Northoff, 2013). This type of research requires data from a large cohort of participants to mitigate the influence of the confounding variables, such as other genetic predispositions and environmental factors. However, previous fMRI and EEG studies have recruited a modest number of participants (for instance, between 40 and 60), which somewhat compromises the reliability of the existing findings.

#### **NEURAL SUBSTRATES MEDIATING THE LINK BETWEEN ACTION AND EMOTION**

Another important topic related to emotion processing in the PFC is the influence of the emotional state on motor function. As stated by Frijda (1987), one of the primary functions of emotion is to guide adaptive motor behavior. However, until very recently, the notion of "action" has been largely absent in research about the neural underpinnings of emotion, with few exceptions (van Peer et al., 2007; Volman et al., 2011). Motor activity has long been known to directly modify emotion and mood (Niedenthal, 2007), and a bi-directional relationship between motor function and the subjective experience of emotion has been established (Zhu and Thagard, 2002). Considering that the PFC is functionally connected with several motor regions like cerebellum (Kipping et al., 2013) and basal ganglia (Kung et al., 2013), it is likely that this region mediates the complex interactions between motor function and emotion.

Many neuroimaging techniques restrict the diversity of actions that can be performed simultaneously while measurements of neural activation are being collected. This, in turn, limits the types of phenomena that researchers can investigate. As NIRS is robust with respect to external noise, it has potential for widening the scope of investigations of the neural mechanisms underlying the interaction between emotion and action.

#### **CONCLUSION**

Despite technical limitations, NIRS is a reliable technique for quantifying several aspects of emotional functioning in the PFC, such as sensitivity to rewarding stimuli (Rolls, 1996) and processing of visceral reactions (Damasio, 1996). There are some practical challenges when using NIRS to research emotion. However, when adequate measures are taken to address these issues, NIRS is an invaluable tool that has the potential to expand the scope of investigations about the emotional function of the PFC.

#### **ACKNOWLEDGMENTS**

This research was supported by Grant-in-Aid for Scientific Research on Innovative Areas "Shitsukan" (No. 25135726) to Hirokazu Doi from MEXT.

**"fnhum-07-00770" — 2013/11/14 — 21:27 — page 4 — #4**

#### **REFERENCES**


"fnhum-07-00770" — 2013/11/14 — 21:27 — page 5 — #5

during interaction and performance tasks. *Brain Imag. Behav.* 7, 140–153. doi: 10.1007/s11682-012-9206-z


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 09 September 2013; paper pending published: 25 September 2013; accepted: 26 October 2013; published online: 18 November 2013.*

*Citation: Doi H, Nishitani S and Shinohara K (2013) NIRS as a tool for assaying emotional function in the prefrontal cortex. Front. Hum. Neurosci. 7:770. doi: 10.3389/fnhum.2013.00770*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Doi, Nishitani and Shinohara. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited andthatthe original publication inthis journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

"fnhum-07-00770" — 2013/11/14 — 21:27 — page 6 — #6

## Music improves verbal memory encoding while decreasing prefrontal cortex activity: an fNIRS study

### *Laura Ferreri 1\*, Jean-Julien Aucouturier <sup>2</sup> , Makii Muthalib3 , Emmanuel Bigand1 and Aurelia Bugaiska1*

<sup>1</sup> Laboratory for the Study of Learning and Development, CNRS UMR 5022, Department of Psychology, University of Burgundy, Dijon, France

<sup>2</sup> Institut de Recherche et Coordination Acoustique/Musique, STMS CNRS UMR9912, Paris, France

<sup>3</sup> Movement to Health, EUROMOV, Montpellier-1 University, Montpellier, France

#### *Edited by:*

Nobuo Masataka, Kyoto University, Japan

#### *Reviewed by:*

Leonid Perlovsky, Harvard University; Air Force Research Laboratory, USA Donna Mae Erickson, Showa University of Music, Japan

#### *\*Correspondence:*

Laura Ferreri, Laboratory for the Study of Learning and Development, CNRS UMR 5022, Department of Psychology, University of Burgundy, Pôle AAFE-11 Esplanade Erasme, 21000 Dijon, France e-mail: lf.ferreri@gmail.com

Listening to music engages the whole brain, thus stimulating cognitive performance in a range of non-purely musical activities such as language and memory tasks. This article addresses an ongoing debate on the link between music and memory for words. While evidence on healthy and clinical populations suggests that music listening can improve verbal memory in a variety of situations, it is still unclear what specific memory process is affected and how.This study was designed to explore the hypothesis that music specifically benefits the encoding part of verbal memory tasks, by providing a richer context for encoding and therefore less demand on the dorsolateral prefrontal cortex (DLPFC).Twentytwo healthy young adults were subjected to functional near-infrared spectroscopy (fNIRS) imaging of their bilateral DLPFC while encoding words in the presence of either a music or a silent background. Behavioral data confirmed the facilitating effect of music background during encoding on subsequent item recognition. fNIRS results revealed significantly greater activation of the left hemisphere during encoding (in line with the HERA model of memory lateralization) and a sustained, bilateral decrease of activity in the DLPFC in the music condition compared to silence. These findings suggest that music modulates the role played by the DLPFC during verbal encoding, and open perspectives for applications to clinical populations with prefrontal impairments, such as elderly adults or Alzheimer's patients.

**Keywords: music, verbal memory, encoding, prefrontal cortex, fNIRS**

#### **INTRODUCTION**

Listening to music engages the whole brain through a diverse set of perceptive and cognitive operations, and equally diverse neural substrates (Altenmüller, 2003). As most of these neural substrates also intervene in other activities, it is increasingly believed that music can benefit non-musical abilities, and most notably language (Abbott and Avins, 2006; Patel, 2011). In particular, there is an ongoing debate in the field of music and cognitive stimulation on whether music can be used to enhance verbal memory. On the one hand, music is a complex auditory stimulus which evolves through time and which has a strong emotional impact (Blood and Zatorre, 2001; Salimpoor et al., 2013). As such, music can provide considerable additional cues which are likely to enrich the encoding of an event. On the other hand, musical information was also claimed to negatively affect memory by attracting participants' attention away from the information to be remembered, generating a dual task situation with poorer memory performance than in a silent situation (Racette and Peretz, 2007; Moussard et al., 2012). In the last 20 years, several studies were conducted in order to understand when and how music can have a positive effect on memory. Research on western music indicates that musical training (Chan et al., 1998; Ho et al., 2003; Franklin et al., 2008) and also simple exposure to music leads to benefits on short- and long-term verbal memory in healthy and clinical populations (Balch et al., 1992; Balch and Lewis, 1996). In 1994, Wallace showed that text is better recalled when heard as a song rather than speech, suggesting that musical context can assist in learning and retrieving words. In clinical settings, short (i.e., music played as a background in a memory task) and long-lasting (i.e., in a music-therapy program) auditory stimulations with music were both shown to improve category fluency in a verbal fluency task in both healthy elderly and Alzheimer's patients (Thompson et al., 2005), speech content and fluency in patients with dementia (Brotons and Koger, 2000), and verbal memory in stroke patients (Särkämö et al., 2008). Additionally, verbal material is more efficiently retrieved when sung than spoken in multiple sclerosis (Thaut et al., 2005), aphasics (Racette et al., 2006), and Alzheimer's patients (Simmons-Stern et al., 2010).

Such evidence suggests that music provides contextual cues that contribute to episodic memory processes. Episodic memory (Tulving, 1972) enables conscious recollection of personal experiences and events from the past (Wheeler et al., 1997). Encoding is a crucial aspect of episodic memory and it is tightly related to the contribution of contextual factors such as location, time, prevailing conditions, and converging multisensory and emotional stimuli (Eich, 1985; Hamann, 2001; Kensinger and Corkin, 2003; Hupbach et al., 2008). Neuroimaging and behavioral data have clearly shown that the capacity to retrieve correct information depends on its successful encoding (e.g., Prince et al., 2005; Hertzog et al., 2010). Furthermore, richer contexts enhance the

"fnhum-07-00779" — 2013/11/21 — 20:44 — page 1 — #1

encoding of contextual information associated to an item and can be subsequently used as mnemonic cues during retrieval, facilitating the access to the target item (Lövdén et al., 2002; Kroneisen and Erdfelder, 2011). It has been shown that enriching the context of encoding through, e.g., enacted encoding (Lövdén et al., 2002) or with emotional valence stimuli (see Hamann, 2001 for a review) can enhance memory performance at retrieval. It is therefore possible that the greatest value of music for memory is to provide mnemonic processes with a particularly rich and helpful context during the encoding phase of episodic memory.

Memory encoding and retrieval processes are supported by a broad brain network that involves the medio-temporal and posterior parietal areas, the hippocampus, and the prefrontal cortex (PFC), the latter being particularly importantfor episodic memory (Spaniol et al., 2009; Manenti et al., 2012). Different sub-regions of the PFC are recruited by different mnemonic processes: according to the hemispheric encoding/retrieval asymmetry (HERA) model (Tulving et al., 1994), left PFC activation is greater for encoding than retrieval, while right PFC activation is greater for retrieval than encoding. Although the PFC sub-region most constantly associated with memory in neuroimaging studies is the ventrolateral prefrontal cortex (VLPFC – BA 44-45-47), the dorsolateral-prefrontal cortex (DLPFC – BA 9 and 46) has recently gained importance for the specific investigation of memory encoding processes. In particular, it has been shown that the DLPFC, mainly in the left hemisphere, plays a crucial role for organizational, associative (Murray and Ranganath, 2007; Ranganath, 2010) and semantic (Innocenti et al., 2010) memory encoding. As discussed byBlumenfeld and Ranganath (2007), DLPFC activation seems to be more specifically sensitive to demands for organizational processing and it may support long term memory by building associations among items that are active in memory.

Functional near-infrared spectroscopy (fNIRS) is an optical neuroimaging technique that can non-invasively monitor cortical tissue oxygenation (oxygenated-O2Hb and deoxygenated-HHb hemoglobin concentration changes) during cognitive, motor, and sensory stimulation (Jobsis, 1977; Ferrari and Quaresima, 2012). In the last 20 years, the use of fNIRS in cognitive neuroscience has constantly increased (Ferrari and Quaresima, 2012). In the field of memory research in particular, fNIRS studies have revealed an increase in PFC oxygenation patterns (i.e., an increase in O2Hb and concomitant decrease in HHb concentrations) during working memory and attention tasks in healthy and clinical populations (seeCutini et al.,2012 for a review). However, the current literature only has a limited number of fNIRS studies (Kubota et al., 2006; Matsui et al., 2007; Okamoto et al., 2011) investigating episodic encoding-retrieval processes. In the field of music cognition, a few fNIRS studies were recently conducted in order to investigate the emotional response to music (Moghimi et al., 2012a; Moghimi S et al., 2012b). However, no fNIRS study has yet looked at a possible role of music in memory encoding.

The previous fNIRS studies that have documented facilitating factors on memory encoding, e.g., strategies to memorize words (Matsui et al., 2007), or pharmacological stimulants such as methylphenidate (Ramasubbu et al., 2012), have repeatedly shown that such factors deactivate, rather than more greatly activate, regions of the PFC – as if they were "less-demanding" (Matsuda and Hiraki, 2004, 2006). Similar reductions of PFC activation were shown using fMRI when there were strong semantic associations between words (Addis and McAndrews, 2006). These evidence suggest that facilitatory cues (e.g., strategies, pharmacological stimulant, strong semantic associations) during verbal encoding could result in less involvement of high cognitive functions mediated by PFC regions (such as DLPFC) known to be usually crucial during memory encoding processes.

The present study addresses the music and memory debate using a source memory paradigm: participants were asked to memorize both lists of words and the context/source (either music or silence) in which words were encoded. The critical new point of the present study was to assess whether the presence of background music during the encoding of verbal material results in different memory-specific cortical patterns of activations than episodic encoding in silence. We used fNIRS to monitor the DLPFC bilaterally during the encoding of verbal material with or without background music context. Our hypothesis is that music may enhance verbal encoding by providing a helpful context which can facilitate organizational, associative, and semantic processes. If so, such an effect of music on memory encoding processes should be linked to both behavioral performance and PFC activity. More specifically, we consider a facilitatory effect of music during verbal encoding should result in a better recognition performance and deactivation of DLPFC activity during the music encoding condition compared to the silent condition.

### **MATERIALS AND METHODS**

#### **PARTICIPANTS**

Twenty-two young healthy students at University of Burgundy (11 female, mean age 23.5 ± 4.3 years) took part in the experiment in exchange of course credits. All the participants were right-handed, non-musicians, French-native speakers, and reported having normal or corrected-to-normal vision. None were taking medication known to affect the central nervous system. Informed written consent was obtained from all participants prior to taking part in the experiment. The study was anonymous and fully obeyed to the Helsinki Declaration, Convention of the Council of Europe on Human Rights and Biomedicine.

#### **EXPERIMENTAL PROCEDURE**

Subjects were seated in a chair in front of a computer in a quiet, dim room. Each participant was subjected to a memory encoding task while their PFC activation was monitored using fNIRS neuroimaging and then behaviorally tested in a retrieval task. After the eight fNIRS probe-set was adjusted on the forehead scalp overlaying the DLPFC (see fNIRS section below for a description) and the in-ear headphones inserted, subjects were informed that they would be presented with different lists of words with two different auditory contexts: music or silence. They were asked to memorize both the lists of words and the context in which words were encoded.

Verbal stimuli consisted of 42 taxonomically unrelated concrete nouns selected from the French "Lexique" database (New et al., 2004, http://www.lexique.org). They were randomly divided into 6 lists of words (7 words each list, 21 words for each encoding condition), equated for word length and occurrence frequency. In

"fnhum-07-00779" — 2013/11/21 — 20:44 — page 2 — #2

the music encoding condition, the background music used in all blocks was an upbeat, acoustic jazz piece ("If you see my mother" by Sidney Bechet), chosen for its positive valence and medium arousal quality.

The encoding phase consisted of three blocks of "music encoding" and three blocks of "silence encoding" intermixed with 30-s rest periods. In each block, seven words were displayed successively in the presence of a music or silence auditory context. The audio stimulation started 15 s before the first word was displayed, continued during the sequential display of words, and ended 15 s after the last word. Words displayed in each block were paced at 4 s per word, amounting to 28 s for the sequential presentation of seven words. Each block therefore had a duration of 58 s (15 s context, 28 s words, 15 s context), and was followed by a 30 s rest (silent) between each block (**Figure 1**). The order of music/silence blocks was counterbalanced, as well as the order of word lists and the order of words in the lists. During the rest periods subjects were instructed to try to relax and not to think about the task any longer; in contrast, during the context-only phases of the blocks (i.e., silence and music blocks), participants were instructed to concentrate on a fixation cross on the screen and to prepare/focus on the task. The entire encoding phase, together with fNIRS recording, took about 10 min.

Prior to the retrieval phase, subjects performed two 5-min interference tasks: a "X-O"letter-comparison task (Salthouse et al., 1997) and a "plus–minus" task (Jersild, 1927; Spector and Biederman, 1976). Subjects were then tested for item and source memory recognition. We used item-memory and source-memory tasks (Glisky et al., 1995) in order to evaluate the subjects' memory for the context of encoding. The retrieval test included the 42 words presented previously, together with 42 new words which were lure items matched for word length and occurrence frequency. For each word, subjects were asked to judge if they had already seen the word before (yes/no button on the keyboard; item-memory task). If they believed they had, they were asked to indicate in which context they saw the word (music/silence/I don't know; sourcememory task). The presentation of task instructions and stimuli as well as the recording of behavioral responses were controlled by the E-Prime software (Psychology Software Tools, Inc.) running on a laptop with a 15-monitor.

#### **fNIRS MEASUREMENTS**

An eight-channel fNIRS system (Oxymon MkIII, Artinis Medical Systems B.V., The Netherlands) was used to measure the concentration changes of O2Hb and HHb (expressed in micromoles) using an age-dependent constant differential path-length factor given by 4.99 + 0.0067\*(age 0.814; Duncan et al., 1996). Data was acquired at a sampling frequency of 10 Hz. The eight fNIRS optodes (four emitters and four detectors) were placed symmetrically over the dorsal part of the PFC (Brodmann Areas 46 and 9, EEG electrodes AF7/8, F5/6, F3/4, and AF3/4 of the international 10/10 system; Okamoto et al., 2004; Jurcak et al., 2007), and the distance between each emitter and detector was fixed to 3.5 cm (**Figure 2**).

To optimize signal-to-noise ratio during the fNIRS recording, the eight optodes were maskedfrom ambient light by a black plastic cap that was kept in contact with the scalp with elastic straps, and

all cables were suspended from the ceiling to minimize movement artifacts (Cui et al., 2011). During data collection, O2Hb and HHb concentration changes were displayed in real time, and the signal quality and the absence of movement artifacts were verified.

#### **DATA AND STATISTICAL ANALYSIS** *Behavioral data*

Each subject's item- and source-memory accuracy (hit) rates (number of hits for each condition during the yes/no recognition) as well as false alarms were calculated for both the silence and music conditions. To examine source memory, we analyzed the proportion of correct source judgments among item-memory hits. A paired *t*-test was used to compare the item- and source- memory scores between the silence and music conditions. One sample *t*-tests were used to ascertain that all the scores were significantly above chance.

#### *fNIRS data*

For each of the eight fNIRS measurement points, the O2Hb and HHb signals were first low-pass filtered to eliminate task-irrelevant systemic physiological oscillations (fifth order digital Butterworth filter with cutoff frequency 0.1 Hz).

In order to ascertain the DLPFC activation during the word encoding task as compared to the rest phase, we first ran a complete timecourse analysis on the O2Hb and HHb signals using a 2(music/silence condition) × 2(left/right hemisphere) × 4(optodes) × 13(successive measures of concentrations, averaged over 5 s windows with the last 10 s of the rest phase as baseline) repeated-measures ANOVA, on which Fisher's LSD *post hoc* comparisons determine which steps of the O2Hb and HHb time course showed significant increase/decrease of O2Hb and HHb as compared to the baseline point set during the rest phase (**Figure 3**).

In order to determine the amount of activation during the encoding phase for the two conditions, data in each of the six experimental blocks was baseline corrected using the mean of the O2Hb and HHb signals during the first 5 s of each block (i.e.,

"fnhum-07-00779" — 2013/11/21 — 20:44 — page 3 — #3

**FIGURE 2 | fNIRS optode localization on the forehead scalp region overlying the dorsal part of the prefrontal cortex.** fNIRS transmitters [**(A)** Tx1–Tx2, **(B)** Tx3–Tx4, yellow circles) and receivers [**(A)** Rx1–Rx2, **(B)** Rx3–Rx4, blue squares) were placed on the **(A)** and **(B)** forehead scalp region, which corresponded to AF7/8, F5/6, F3/4, and AF3/4 EEG channels (international 10/10 system), respectively renamed left/right channels 1, 2, 3, and 4.

during context-only phase, before the start of word encoding). We then sample-to-sample averaged (i.e., 10 samples/s) the baseline-corrected signals over the three blocks of each condition, yielding one average music and silence O2Hb and HHb signal per participant. We then computed the maximum O2Hb (max-O2Hb) value and the minimum HHb (min-HHb) value over the 28 s stimulus window (i.e., from *t* = 15 s to *t* = 42 s), for both the music and silence average block of each participant. The peak concentrations (max-O2Hb, min-HHb) were analyzed using a repeated measure MANOVA with 2(music/silence condition) × 2(left/right hemisphere) repeated factors and optodes (4) as a multivariate. The significance level was set at *p* < 0.05.

#### **RESULTS**

#### **BEHAVIORAL RESULTS**

Both the item and source memory scores were significantly above chance (one sample *t*-test, *p* < 0.003), demonstrating that participants did not encounter strong difficulties to remember the specific context in which words were presented.

There was a statistically significant difference in item recognition performance between the music (mean = 18.36, SD = 2.84) and silence [mean = 16.59, SD = 3.98; *t*(21) = 2.63, *p* = 0.016] conditions, with improved recognition in the musical condition. However, there was no significant difference (*p* > 0.05) in source memory performance between music (0.67 ± 0.22) and silence (0.68 ± 0.22) conditions.

#### **fNIRS RESULTS**

**Figure 3** shows the grand average time course of PFC O2Hb and HHb concentration changes at each of the eight fNIRS channels in the music and silence encoding conditions. The repeated-measure ANOVA on the O2Hb and HHb timecourse series revealed a main effect of condition (*F* = 8.130, *p* = 0.01), corresponding to significantly greater O2Hb increases bilaterally in the silence than music condition. Although the increases in O2Hb are visible bilaterally during the silence condition (especially in bilateral channels 1 and 2 as shown by *post hoc* LSD Fisher comparisons), together with a decrease in HHb (in particular for left channels 1, 2 and right channel 2), the music condition was associated with a strong bilateral decrease of O2Hb (underlined by significant LSD Fisher *post hoc* comparisons especially for bilateral channels 3 and 4).

**Figure 4** shows the group mean of max-O2Hb and min-HHb values recorded at each of the eight fNIRS channels on the PFC for the silence and music conditions. The repeated-measure MANOVA on max-O2Hb values revealed a statistically significant main effect of condition [*F*(4,18) = 4.207, *p* = 0.008], with greater O2Hb increases bilaterally in the silence than music condition, and a significant main effect of channel laterality [*F*(4,18) = 5.006, *p* = 0.003], with greater O2Hb increases in the left hemisphere regardless of condition. Although there was no effect of condition for min-HHb values, which is typical with several other NIRS studies (e.g., Matsui et al., 2007; Okamoto et al., 2011), there was

"fnhum-07-00779" — 2013/11/21 — 20:44 — page 4 — #4

a significant effect of laterality [*F*(4,18) = 3.783, *p* = 0.013], with greater values in the right hemisphere (which is coherent with smaller O2Hb values, i.e., overall smaller O2Hb increases in the right hemisphere).

#### **DISCUSSION**

The present study shows that a background musical context during the encoding of verbal material modulates the activation of the DLPFC and, at the same time, facilitates the retrieval of the encoded material. Despite a few recent studies (e.g., Racette and Peretz, 2007; Moussard et al., 2012) reporting a perturbing effect of music on the memorization of verbal material, a consistent part of the literature (e.g., Balch et al., 1992; Wallace, 1994; Balch and Lewis, 1996; Chan et al., 1998; Brotons and Koger, 2000; Ho et al., 2003; Thaut et al., 2005; Thompson et al., 2005; Racette et al., 2006; Franklin et al., 2008; Särkämö et al., 2008; Simmons-Stern et al., 2010) claims that music can have a positive effect on memory in both healthy and clinical populations. However, most of these studies had remained on a behavioral level. The critical new point of the present study was to also track the brain activation response during the encoding phase with fNIRS: we found that improved word recognition coincides with reduced DLPFC activation in musical encoding compared to silence encoding.

#### **BEHAVIORAL RESULTS**

Our behavioral results showed an effect of music on subsequent item recognition memory performance, although this did not

extend to a source memory performance. We found that music played during encoding facilitates item recognition. The role of background music in learning and memory tasks is still an open and debated question in the literature (e.g., Schellenberg, 2003, 2005; De Groot, 2006; Peterson and Thaut, 2007; Jäncke and Sandmann, 2010). Research in music cognition is increasingly aware of the fact that it is necessary, rather than stating a general and reliable positive effect of music, to disentangle which experimental paradigm can lead to memory improvements through music, when and with whom. Our results are in line with previous studies on healthy subjects (Balch et al., 1992; Wallace, 1994; Balch and Lewis, 1996; De Groot, 2006) and clinical populations (Brotons and Koger, 2000; Ho et al., 2003; Thaut et al., 2005; Thompson et al., 2005; Racette et al., 2006; Franklin et al., 2008; Särkämö et al., 2008; Simmons-Stern et al., 2010) which showed a positive role of music in verbal memory encoding. Consistent with the view that context is crucial during episodic memory encoding, our findings support the idea that music provides rich and helpful contextual cues that are useful for subsequent item recognition.

#### **fNIRS RESULTS**

The novel finding of the study was that DLPFC activation was significantly higher during the silence than music encoding condition (**Figures 3** and **4**). As predicted in our hypothesis, the facilitatory effect of music during verbal encoding resulted not only in better recognition performance, but also deactivation of DLPFC activity. On the one hand, encoding PFC activity in the silent condition followed the classical hemodynamic response

"fnhum-07-00779" — 2013/11/21 — 20:44 — page 5 — #5

to neuronal activation, showing a bilateral increase of O2Hb together with a decrease of HHb as compared to baseline. This result confirms the involvement of the DLPFC in episodic memory encoding (Blumenfeld and Ranganath, 2007; Murray and Ranganath, 2007; Innocenti et al., 2010). On the other hand, encoding in the music condition showed a bilateral reversed PFC hemodynamic response (with a sustained decrease in O2Hb and minimal change in HHb), which only returned to baseline at the end of each music block (see **Figure 3**). This result suggests that the DLPFC was deactivated during word encoding in the musical context and that music can strongly modulate the activity of the bilateral DLPFC. Similar PFC deactivation have already been shown by fNIRS studies investigating human cognition (Matsuda and Hiraki, 2006, 2004), and specifically in verbal learning tasks when subjects were helped to memorize words by a given strategy (Matsui et al., 2007) or by a pharmacological stimulant (Ramasubbu et al., 2012).

However, to the best of our knowledge, none of the previous fNIRS studies which investigated memory processes or music perception reported music-specific PFC deactivation. In the present study, this PFC deactivation during memory encoding with a musical context could be the manifestation of our hypothesis that music plays a facilitating, less-demanding role for the PFC during word encoding.

fNIRS analysis of the maximum O2Hb concentration values reached during word encoding in the music and silence conditions revealed also a significant main effect of lateralization. As predicted, we found greater PFC activation (represented by O2Hb increases) in the left than the right hemisphere during the entire encoding phase (especially for channels 1 and 2, see **Figure 4**). This result is in line with the hemispheric left prefrontal asymmetry during the encoding of verbal material, as predicted by the HERA model (Tulving et al., 1994; Nyberg et al., 1996), and confirms the feasibility of fNIRS neuroimaging for the study of long-term memory processes (Kubota et al., 2006; Matsui et al., 2007; Okamoto et al., 2011).

It is important to discuss the possible mechanisms by which music may act on the PFC during memory encoding tasks. The PFC, specifically the DLPFC, is known to be recruited during tasks demanding organizational (Blumenfeld and Ranganath, 2007) and relational inter-item processing during encoding (Murray and Ranganath, 2007). Therefore, one possible interpretation of the deactivation of the PFC (i.e., O2Hb decrease) during music in the present study is that music helps to generate inter-item and item-source relationships, without demanding high-cognitive PFC processes. Investigating the correlation between PFC activity and semantic associations during word encoding with fMRI, Addis and McAndrews (2006)found that greater semantic associations correlated with reduced activity in the inferior frontal gyrus (IFG) region of the PFC. A recent fNIRS study by Ramasubbu et al. (2012) also supports this explanation. The authors gave methylphenidate (a central nervous system stimulant) or placebo to subjects and measured PFC activation during a working memory task (N-back). They found a reduction in PFC O2Hb from baseline together with better behavioral performance, which the authors suggested was due to methylphenidate improving neuronal efficiency or signal–noise ratio during the memory task. In the present study, the decreased PFC activity observed during the music condition could therefore indicate better neuronal efficiency.

The musical context may afford efficient mnemonic strategies to bind items between one another, and/or to bind items to music, so that less PFC activity is required to drive these associations. In line with the idea of music as an help for cognitive functions which could lead to a deactivation of PFC activity, it has been recently shown how exposure to consonant music improve performance during a Stroop task, suggesting that music may help overcoming cognitive interference (Masataka and Perlovsky, 2013). So how could music represent a facilitatory factor particularly for words encoding? Previous EEG studies underlined how few seconds of music can influence the semantic and conceptual processes of words, showing that both music and language can prime the meaning of a word and determine physiological indices of semantic processes (Koelsch et al., 2004; Daltrozzo and Schön, 2009a,b). It is therefore possible that this semantic priming could also be reflected in easier associations and bindings between items when background music is present. Further investigations on organizational strategies during verbal encoding with music may confirm this explanation and shed new lights on music-verbal memory cognitive processes.

Another possible explanation of the music-specific PFC deactivation is an increase of attentional mechanisms in the music condition. Music is known to modulate attentional processes (Janata et al., 2002; Janata and Grafton, 2003; see also Masataka and Perlovsky, 2013) and previous fNIRS studies reported that the more attention the subjects put to a task, the more greatly rCBF (Mazoyer et al., 2002; Geday and Gjedde, 2009) and O2Hb concentrations (Matsuda and Hiraki, 2004, 2006) were decreased in the PFC. This second interpretation would be in line with previous behavioral studies which attributed improved cognitive performance in the presence of a musical background to higher amounts of arousal and attention (Foster and Valentine, 2001; Thompson et al., 2005; Patel, 2011). However, in apparent conflict with this interpretation, a considerable amount of literature claims the importance of the PFC in attentional processes, mainly for the maintenance and mental manipulation of memory contents (see Ptak, 2012 for a review). In the present study, we observed that PFC O2Hb increase seems to precede the word encoding phase by a few seconds in the most lateral fNIRS channels (i.e., left channels 1 and 2, corresponding to EEG channels AF7 and F3, **Figure 3**), and this, even in the silence condition (**Figure 3**). This may indicate that attentional processes are already in full use when encoding in silence, and put a limit on the potential of even further recruitment of attention specific to music.

Finally, we should also note that it has been repeatedly observed that music-related processing typically recruits more widely distributed networks of cortical and subcortical areas than non-musical verbal function (Halpern, 2001; Parsons, 2001; Peretz, 2002; Altenmüller, 2003; Patel, 2011). If so, PFC deactivation in the music condition could also reflect broader network recruitment during word encoding with music. Further research is needed to test this hypothesis.

fNIRS data interpretation of the present study must be done bearing in mind certain limitations. First, recent studies suggest

"fnhum-07-00779" — 2013/11/21 — 20:44 — page 6 — #6

**FIGURE 4 | Mean (±SD) of the prefrontal cortex max-O2Hb (red lines) and min-HHb (blue lines) concentration values over the left (channels 1, 2, 3, and 4, corresponding to EEG channels AF7, F5, AF3, F3) and right**

**(channels 1, 2, 3, and 4, corresponding to EEG channels AF8, F6, F4, AF4) hemisphere during memory encoding for the silence (solid lines) and music (dotted lines) condition.**

that caution should be exercised when applying fNIRS to infer PFC activation: the task-evoked changes occurring in forehead skin perfusion could represent an overestimation of the cortical changes as measured by fNIRS. Recent reports have raised a question against the assumption that PFC O2Hb/HHb changes originated only from the cortical hemodynamic response (Kohno et al., 2007; Gagnon et al., 2011, 2012; Takahashi et al., 2011; Kirilina et al., 2012). Furthermore, as previously described, fNIRS acquisitions in the present study were limited to eight channels covering the bilateral DLPFC. So, it is not possible to know whether other cortical areas were involved during episodic encoding, especially for the music condition. Despite these limitations, several studies have shown fNIRS feasibility for the study of cognitive processes (Cutini et al., 2012), and this study for the first time applied fNIRS to investigate if and how music can help memory during episodic encoding.

An important perspective for further research is to apply fNIRS monitoring during the retrieval phase. Indeed, research on episodic memory during the past century has demonstrated that a complete understanding of how memories are formed requires appreciation of the many cognitive and neurobiological processes that constitute encoding and retrieval, as well as the interaction among these two stages (Brown and Craik, 2000). The behavioral and fNIRS data we obtained lead us to wonder about what is also happening during retrieval. Further studies with multichannel fNIRS systems during both encoding and retrieval phases are needed to determine which regions are more activated and to clarify how music could act on long term memory processes.

Another interesting perspective for further studies is to extend our paradigm to applications in older adults or patients with dementia. Several studies have highlighted that memory impairments in normal aging as well as several types of dementia (e.g., Alzheimer's disease) are often linked to impairments or damage in frontal lobe functions (e.g., see Maillet and Rajah, 2012 for a review). Our results suggest that music helps verbal encoding by facilitating associative and organizational processes (i.e., generate inter-item and item-source relationships) without demanding the high-cognitive PFC processes which are usually required. At the same time, fNIRS is a non-invasive technique and its features allow it to be used also with special populations by preserving good ecological settings (Cutini et al., 2012; Ferreri et al., in press). Further fNIRS investigations on normal and pathological aging could therefore be pivotal for better understanding of how music can be used as a tool in rehabilitation of memory disorders.

In conclusion, we have shown that background music context during the encoding of verbal material modulates the activation of the PFC during encoding and, at the same time, facilitates the retrieval of the encoded material. This opens interesting perspectives on how music could act on the PFC of subjects with memory disorders for whom the prefrontal lobe is hypo-activated, impaired or damaged, such as older adults or Alzheimer's patients.

#### **ACKNOWLEDGMENTS**

This work was supported by the European Project EBRAMUS (European BRAin and MUSic) ITN – Grant Agreement Number 218357 and the Conseil Régional de Bourgogne. Authors also thank Chris Moulin for insightful comments on the manuscript.

#### **REFERENCES**


"fnhum-07-00779" — 2013/11/21 — 20:44 — page 7 — #7


"fnhum-07-00779" — 2013/11/21 — 20:44 — page 8 — #8


and auditory cortices predict music reward value. *Science* 340, 216–219. doi: 10.1126/science.1231059


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 September 2013; paper pending published: 06 October 2013; accepted: 29 October 2013; published online: 22 November 2013.*

*Citation: Ferreri L, Aucouturier J-J, Muthalib M, Bigand E and Bugaiska A (2013) Music improves verbal memory encoding while decreasing prefrontal cortex activity: an fNIRS study. Front. Hum. Neurosci. 7:779. doi: 10.3389/fnhum.2013.00779*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Ferreri, Aucouturier, Muthalib, Bigand and Bugaiska. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

"fnhum-07-00779" — 2013/11/21 — 20:44 — page 9 — #9

### Monitoring attentional state with fNIRS

### *Angela R. Harrivel 1,2\*, Daniel H. Weissman3, Douglas C. Noll <sup>2</sup> and Scott J. Peltier <sup>2</sup>*

*<sup>1</sup> Bioscience and Technology Branch, NASA Glenn Research Center, Cleveland, OH, USA*

*<sup>2</sup> fMRI Laboratory, Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA*

*<sup>3</sup> Department of Psychology, University of Michigan, Ann Arbor, MI, USA*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Yukiori Goto, Kyoto University, Japan Truong Q. Dang Khoa, International University of Vietnam National Universities in Ho Chi Minh City, Viet Nam*

#### *\*Correspondence:*

*Angela R. Harrivel, Bioscience and Technology Branch, NASA Glenn Research Center, Mail Stop 110-3, 21000 Brookpark Rd., Cleveland, OH, 44135, USA e-mail: angela.r.harrivel@nasa.gov*

The ability to distinguish between high and low levels of task engagement in the real world is important for detecting and preventing performance decrements during safety-critical operational tasks. We therefore investigated whether functional Near Infrared Spectroscopy (fNIRS), a portable brain neuroimaging technique, can be used to distinguish between high and low levels of task engagement during the performance of a selective attention task. A group of participants performed the multi-source interference task (MSIT) while we recorded brain activity with fNIRS from two brain regions. One was a key region of the "task-positive" network, which is associated with relatively high levels of task engagement. The second was a key region of the "task-negative" network, which is associated with relatively low levels of task engagement (e.g., resting and not performing a task). Using activity in these regions as inputs to a multivariate pattern classifier, we were able to predict above chance levels whether participants were engaged in performing the MSIT or resting. We were also able to replicate prior findings from functional magnetic resonance imaging (fMRI) indicating that activity in task-positive and task-negative regions is negatively correlated during task performance. Finally, data from a companion fMRI study verified our assumptions about the sources of brain activity in the fNIRS experiment and established an upper bound on classification accuracy in our task. Together, our findings suggest that fNIRS could prove quite useful for monitoring cognitive state in real-world settings.

**Keywords: near infra-red spectroscopy, attention, default mode network, classification, human performance**

#### **INTRODUCTION**

The ability to distinguish between high and low levels of task engagement is important for detecting and preventing performance decrements during safety-critical operational tasks in the real world. Examples of such tasks include commercial aviation, monitoring for air traffic control, executing space walks, performing surgery, and driving. Since accident-causing errors can be made even by skilled professionals (Dismukes et al., 2007), the ability to monitor cognitive state measures for low levels of task engagement in real time could be useful for developing an "early warning system" for detecting and preventing performance errors before they occur.

The use of cognitive state measures to optimize human performance (for example by informing flight automation or the operator themselves of a hazardous state) has been of particular importance to aviation safety (Pope et al., 1995; Schnell et al., 2004) and to the Augmented Cognition program (Raley et al., 2004) of the Defense Advanced Research Projects Agency of the United States. Additionally, such research is highly relevant to space flight, since adaptation to microgravity can cause performance decrements due to motion sickness, lack of sleep, loss of sensorimotor control, increased stress or mood changes (Cowings et al., 2003). More generally, the ability to monitor cognitive state for low levels of task engagement could be helpful for detecting and preventing vigilance decrements due to sleep-deprivation (Drummond et al., 2005; De Havas et al., 2012) or distraction (Strayer et al., 2011).

Monitoring brain activity may provide an effective means for monitoring cognitive state, and in particular for distinguishing between high and low levels of task engagement. Numerous functional magnetic resonance imaging (fMRI) studies have revealed that activity increases in a so-called "task-positive" network, which includes the dorsolateral prefrontal cortex (DLPFC), dorsal anterior cingulate cortex (dACC), superior and inferior parietal lobe (SPL and IPL), and anterior insula (AI), when participants perform a task as compared to when they rest (MacDonald et al., 2000; McKiernan et al., 2003; Dosenbach et al., 2006). In contrast, activity increases in a so-called "task-negative" network, which includes the anterior medial frontal gyrus (aMFG), posterior cingulate cortex (PCC), and certain regions of lateral parietal cortex (LPC), when participants rest as compared to perform a task (Raichle et al., 2001; Greicius et al., 2003). In other words, activity in the "task-positive" and "task-negative" networks is negatively correlated (Fox et al., 2005; Kelly et al., 2008). For this reason, monitoring activity in key regions of the "task-positive" network alone or monitoring activity in key regions of the "taskpositive" and "task-negative" networks together may distinguish between relatively high and relatively low levels of task engagement (Drummond et al., 2005; Weissman et al., 2006; Chee et al., 2008). Given recent data indicating that interactions between key regions of the "task-positive" and "task-negative" networks vary with task engagement (Prado and Weissman, 2011), we predicted that either approach for monitoring brain activity would allow us to distinguish between relatively high and relatively low levels of task engagement, but that the latter approach would likely prove most effective.

Since hemodynamic activity cannot be monitored with fMRI outside of a laboratory, we employed functional Near Infrared Spectroscopy (fNIRS) to determine whether monitoring brain activity is an effective method for monitoring cognitive state. FNIRS is a portable optical neuroimaging technique that can be used to quantify hemodynamic activations. Moreover, it is relatively low-cost, non-confining, non-invasive, and safe for long-term monitoring (Boas et al., 2004; Gibson et al., 2005; Gratton et al., 2005; Steinbrink et al., 2005; Schroeter et al., 2006). Finally, temporal resolution is sub-second, and spatial resolution is on the order of 1 cm<sup>2</sup> at best (Strangman et al., 2002a; Obrig and Villringer, 2003; Bunce et al., 2006). Measurements have been shown to be consistent with fMRI (Kleinschmidt et al., 1996; Strangman et al., 2002b; Steinbrink et al., 2005; Huppert et al., 2006; Schroeter et al., 2006; Emir et al., 2008) and electroencephalography (EEG) (Moosmann et al., 2003; Li et al., 2007). Critically, the ambulatory nature of fNIRS allows neuroimaging in the field. Thus, fNIRS may evolve into a synergistic complement to EEG and other physiological measures for monitoring cognitive state during operational tasks.

In the present study, we employed fNIRS to determine whether it is possible to distinguish between high and low levels of task engagement (i.e., performing a task vs. resting). Specifically, we monitored brain activity from the DLPFC in the "task-positive" network and from the MFG in the "task-negative" network while participants alternated between performing and not performing a cognitive task. We then employed multivariate pattern classification techniques in an effort to distinguish between periods of task performance and periods of rest.

To facilitate our ability to make this distinction, we asked participants to perform the multi-source interference task (MSIT; Stins et al., 2005; Bush and Shin, 2006). The MSIT is a selective attention task in which optimal performance requires participants to suppress multiple sources of interference (Stroop, Eriksen, and Simon). Thus, it reliably and robustly activates the "task-positive" network, even in individual task blocks from the same participant (Bush and Shin, 2006). Given these characteristics, we reasoned that the MSIT would provide a strong signal with which to monitor task engagement. Consistent with this reasoning, in the present study we were successful at distinguishing between relatively high and relatively low periods of task engagement.

We also conducted a companion fMRI study for verification and comparison purposes. First, given that DLPFC and MFG activity was recorded with fNIRS at the scalp surface, we wished to verify that our paradigm actually elicited hemodynamic activations in these regions. Second, we wished to compare fNIRS with fMRI with regard to the ability to distinguish between high and low levels of task engagement. Given that fMRI detects motor cortex activation reliably enough for routine use (Möller et al., 2005), and given that the motor cortex should be activated by the button presses required by our task, we expected that including such activation as an input to our classification algorithms would produce the highest levels of accuracy. Thus, we reasoned that classification accuracy with fMRI when including motor cortex activation would establish an "upper bound" for classification accuracy expectations. As expected, the fMRI study confirmed both of these predictions.

### **METHODS**

#### **BEHAVIORAL METHODS**

#### *Participants*

Seven participants (three females, four males) completed the fMRI study. Five participants (two female, three male) completed the fNIRS study (including two from the fMRI study). All participants practiced the behavioral task with performance feedback for 1 min at the beginning of the study. Next, they performed a set of four 7-min-long runs. Four runs each were completed for the fNIRS and fMRI experiments, which were performed on separate days. Human participant data were collected according to a protocol approved by both the University of Michigan IRB MED and the NASA IRB. Informed consent was obtained from all participants, who were healthy adults between the ages of 21 and 50 years. All participants were right-handed but one, who was ambidextrous.

#### *Behavioral task*

In each trial of the MSIT (duration, 2 s), participants viewed a horizontally-oriented array of four digits at the center of the screen (duration, 1 s). A target digit was printed in a larger font than each of three distracter digits (72 point vs. 60 point). The stimulus sizes were chosen to make the digits visible within the MRI scanner. Moreover, the large target digit was chosen to be about 20% larger than the small distracter digits to make the task sufficiently difficult. Indeed, Stins et al. (2005) observed a much smaller interference effect in the MSIT when the large target digit was 33% larger than the small distracter digits. Since the goal of the present MRI experiment was to contrast activity during task performance to activity during rest, we wanted to ensure that the task was difficult enough to elicit a good deal of task-related activity.

Participants were instructed to identify the target digit (1, 2, 3, or 4) by pressing a key with one of four fingers (1 = right thumb, 2 = right index finger, 3 = right middle finger, 4 = right ring finger) as quickly as possible without making mistakes. For congruent trials (50%), both the spatial position of the target and the identity of the three distracter digits were mapped to the correct response (1111, 2222, 3333, 4444). For incongruent trials (50%), both the spatial position of the target and the identity of the three distracter digits were mapped to a conflicting response (2111, 2122, 3343, 4443). Participants alternated between trials composed of 1s and 2s and trials composed of 3s and 4s across trials to prevent immediate stimulus and response repetitions (Mayr et al., 2003; Jiménez and Méndez, 2013).

Responses were recorded via the keyboard (the "n," "u," "9," and "0" keys) of a laptop used in fNIRS experiments and via a MR-compatible response device in fMRI experiments. The task was implemented using a combination of MATLAB, 2012 (Natick, MA) and the Psychophysics Toolbox (Kleiner et al., 2007).

#### *Functional neuroimaging experimental design*

We employed a block design. In each of four runs, an initial 16 s rest block was followed by 12 alternations between the MSIT (16 s — 8 trials per block) and rest (16 s). Each run lasted 400 s.

#### **fNIRS METHODS**

#### *fNIRS data acquisition*

Hemoglobin concentration changes were measured using an Imagent NIRS instrument and fiber optic cables (ISS, Inc.). Eleven rigidly-connected source-detector pairs were used, and each source fiber delivered both 690 and 830 nm wavelength light. The data collection rate was 6.25 Hz. During each task run, 2500 time points were collected. Eight sources were located around one detector placed over the DLPFC region, and three sources were located around a second detector placed over the MFG. The array of head probes and fiber optic cables used to interrogate the MFG in this study is shown in **Figure 1**, right. The sources were held in place using both clear and blacked-out plastic at the locations shown with respect to the International 10–20 locations in **Figure 1**, left. The array of probes used to interrogate the MFG contained two sources placed 3 cm from the detector placed between FPz and FP2. The array of probes used to interrogate the DLPFC contained seven sources placed 3 cm from the detector placed near F4. The clear material (not shown) improved visual inspection for placement and hair control at the probeskin interface. Blackout material was applied over and around the probes to block ambient light. Motor regions were not simultaneously interrogated due to instrumentation and fiber optic probe limitations.

The probes were located with the aid of an electroencephalography net (64-channel HydroCel Geodesic Sensor Net by EGI,

**FIGURE 1 | Head probes used to interrogate the DLPFC and MFG. Left**: source channels (red) are arranged at a separation distance of 3 cm from two detectors (green, "D"), shown schematically with respect to the International 10–20 locations. The shallow sources, located at 1 cm from the detectors, provide channel 4 for the DLPFC array, which is near F4, and channel 2 for the MFG array, which is between FPz and FP2. The probes consistently producing the traces with the highest task model fit parameters were DLPFC probe channels 1–3 and MFG channel 1 (as numbered, **left**). **Right**: the skin-side of the array of head probes used for the MFG.

Inc.) applied according to EGI Inc.'s instruction. The nets were used to identify the International 10–20 locations for each participant at each visit, not to record EEG data. A mark was made under the same net pedestals for all participants. The use of the nets made localization reliable, consistent and expeditious. The MFG mark was placed under a pedestal half way between FPz and FP2. This placed the sources for the MFG array (which were 2 cm from each other) at about the midline, with the detector then about 3 cm from the midline, to avoid the superior sagittal sinus as much as possible. The DLPFC mark was placed at the pedestal for EEG channel 59, which is immediately inferior to F4. The DLPFC array was placed by hand such that the detector and the source for channel 3 straddled its mark. The MFG array was placed similarly for channel 2. The probe arrays were secured with Velcro straps. The probes were not moved between the four runs unless the participant requested adjustment for the purposes of comfort. In those cases, care was taken to relieve pressure on the head without translating the probes.

Each probe array included a shallow source located 1 cm from the detector for superficial physiological and nuisance signal regression. The sources located 3 cm from the detectors provided deep traces of interest which sampled brain tissue, while the sources at 1 cm provided shallow traces which sampled only superficial tissue due to the proximity to the detector (Gagnon et al., 2011). The shallow channels were primarily sensitive to physiological changes in the skin, while also being sensitive to nuisance signal contributions such as motion of the rigidlyconnected probe and ambient light exposure.

The Imagent instrument employs photomultiplier tubes for optical intensity detection. Gain settings for these were set on a per-participant basis after applying the probes and ensuring most of the hair was parted under the probe tips. Gain was increased and probes were re-adjusted to make better contact with the skin, iteratively, until as many channels as possible detected continuous wave intensity signal above a threshold of 500 analog-to-digital counts. None of the channels with low signal produced the best fit to the task model, and thus were not passed to the classification step (described below). The shallow channel sources at 1 cm may damage the detectors at gain settings appropriate for sources at 3 cm due to the lack of signal attenuation over their relatively shorter optical path length. To avoid this, delivered optical intensity was reduced for the shallow channel sources by layering optically absorbent pigment on partially-transmissive tape between the ends of the fiber probe tips and the skin.

#### *fNIRS data processing*

Both oxygenated and deoxygenated hemoglobin concentration ([Hb]) changes were calculated from the filtered raw continuous wave intensity measured traces using the Modified Beer Lambert Law (Delpy et al., 1998), then normalized (Huppert et al., 2009). Filtering was set to include 0.008–0.08 Hz to focus on sustained task activations while removing very slow drift and higher frequency physiological and motion contributions.

Standard linear regression was then used to remove physiological contributions from the measured signal to produce the functional task signal, and to select probes from each array. Probe channels, and specifically the oxygenated or deoxygenated [Hb] trace from that probe, were selected for use in classification based on how well the measured deep traces fit the expected functional activation task response, quantified by their beta fit parameter. The expected task response was modeled by convolving the boxcar task onset signal with the hemodynamic response function. The expected deoxygenated [Hb] time series was set to the negative of the oxygenated [Hb]. The shallow trace was smoothed using a 6-timepoint moving average, and task-like response in it was removed (by a separate regression step) prior to use as a nuisance regressor in the design matrix. The shallow trace removal was performed within-species. That is, the oxygenated [Hb] trace from the shallow channel was regressed from the measured oxygenated trace of each deep channel on that detector's array, and similarly for the deoxygenated traces. The functional task signal, which was effectively the measured signal minus the fitted nuisance signal, was then passed to the classification step.

Within- and across-network regional correlations, respectively, were defined as the time-series correlation coefficients relating activity between different regions of the DLPFC in the task-positive network and between the DLPFC and the MFG in the task-negative network. Each of these correlation values was averaged across all four runs in each participant, before it was averaged across participants. All statistical tests on correlation and accuracy values were one-tailed, with comparisons paired by participant.

#### *fNIRS classification*

Classification was performed using two traces as input features to Support Vector Machines (SVM). Scripts were implemented in MATLAB, 2012 using LibSVM (Chang and Lin, 2011). We choose SVM for good performance with ease of implementation, processing speed in the interest of future real-time application, and the ability to use tuning parameters to optimize feature separation for each participant. First, the best two traces for each participant were selected from the array of seven deep DLPFC channels (a within-network pair, noted as DLPFC and DLPFC2 in **Figure 2**), based on which traces (oxygenated or deoxygenated) best fit the functional activation model as described above (which had the highest beta fit parameter considering all four runs). The traces which best fit the task model for each participant were assumed to make the best input features for producing the highest classification accuracy. The same best DLPFC trace was then paired with the best trace for each participant from the array of two deep MFG probes (an across-network pair, noted as DLPFC and MFG in **Figures 2**, **3**) and a separate, second classification step was performed. Trace selections were not changed across runs.

A SVM model was trained to discriminate high from low levels of task engagement using three of each participant's four runs. Its prediction accuracy was tested on the participant's fourth run. Thus, training and prediction were always conducted within participants. All permutations were computed for each participant, such that the SVM model's prediction accuracy could be determined for each of the four runs. As described above, this was done separately for within- and across-network input pairs. The truth labels used for training and accuracy determination purposes were determined from the boxcar task onset signal, but shifted 4 s later to account for the delay of the hemodynamic response (the green trace in **Figure 3**).

**FIGURE 2 | Classification accuracy averaged across four runs for each of five participants.** Two fNIRS time traces were used as support vector machine input features in each case. **Left-hand bars**: within-network pairs (green). **Right-hand bars**: cross-region pairs (red). Accuracy is the number of time points for which the prediction matched the truth label out of all 2500 time points. An accuracy of 50% represents prediction by chance. Error bars represent ± one standard deviation.

A radial basis function kernel was employed for the SVM, which involves two tuning parameters. Kernel parameter gamma (g) determines the non-linearity of the mapping of our few features into a multidimensional space in which the prediction class labels are determined. A lower cost of error (c) allows more flexibility and generalizability of the SVM model by reducing the cost of misclassifications during model training (Chang and Lin, 2011). Non-linear mapping may improve performance depending on g. To determine the best possible prediction accuracy achievable with the methods of this study, c was tested at 0.1, 1, and 10, and g was tested at 0.001, 0.01, and 0.1 (a total of nine cases) in each classification step to optimize accuracy. These values were selected after running a test case varying c and g each across nine orders of magnitude. Accuracy did not change appreciably across four orders of magnitude, and the c, g values used were selected from this region. The highest classification accuracies produced using the same set of parameters across all four runs (the same c and g, given in **Table 1**) were selected for each participant and reported (see **Figure 3**). The best c, g parameters found upon optimization are not extreme in value and are not identical across participants. Accuracy was determined by comparing the work or rest state predicted for each time point to the known states (i.e., performing the task or resting).

#### **fMRI METHODS**

#### *fMRI data acquisition*

fMRI data were acquired using a 3T GE Discovery MRI scanner with a T2∗-weighted spiral BOLD sequence with pulse sequence parameters TR/TE/FA = 2 s/30 s/90◦. The FOV was 22 cm in a 64 × 64 matrix for 40 slices with 3 mm thickness. The total time for each scan (400 s) was matched to the functional task at 200 volumes, after discarding 5 volumes at the beginning of each scan. Physiological signals were collected concurrently using a pulse oximeter and chest plethysmograph. For anatomical reference in the functional data analysis, a high-resolution T1-weighted anatomical image was collected using spoiled-gradient-recalled acquisition (SPGR) in steady-state imaging with pulse sequence parameters TR/TE/FA = 12.2 ms/5.2 ms/15◦. The FOV was 26 cm in a 256 × 256 matrix for 136 slices at thickness 1.2 mm.

#### *fMRI data processing*

After slice-timing and motion correction, and physiological noise removal with RETROICOR (Glover et al., 2000), fMRI data were co-registered to the participant's anatomical scan, normalized to the MNI template (Collins et al., 1994), smoothed, modeled and estimated with SPM8 (Wellcome Department of Cognitive Neurology, London, UK). Smoothing was performed with a Gaussian kernel of 8 × 8 × 8 mm at full width half maximum. The resulting image files contained BOLD activation time traces for each voxel of the brain and were used in the classification step.

Both "rest minus task" and "task minus rest" contrasts were generated and used in second level random effects analyses performed across all runs for seven participants to inform fNIRS probe placement, and separately for each participant (across that individual participant's four runs) to guide voxel selection for fMRI classification.

Across-network, within-network and co-activating correlations were defined, respectively, as the correlation coefficients between across-network (i.e., the DLPFC and the MFG), withinnetwork (i.e., DLPFC and DLPFC2), or co-activating (i.e., the DLPFC and motor cortex) pairs of functional task signals. The

**Table 1 | The best** *c***,** *g* **parameters found after optimization for each participant for fNIRS classification, and the time trace correlations (***r***) averaged across that participant's four runs.**


fMRI traces were additionally smoothed across 10 s to reduce the impact of noise in the signal, before the correlation coefficient was calculated. These were averaged across runs by participant, then averaged across participants.

#### *fMRI classification*

All fMRI traces used as SVM inputs were processed BOLD responses, averaged across clusters centered on local maxima within the regions of interest (DLPFC, primary motor cortex, and MFG). Eighteen voxels were symmetrically selected around participant-specific centers to be included in the average. Participant-specific locations were selected using the second-level statistical maps generated using that individual participant's four runs. The voxels used were not contiguous, and the region of interest spanned 1 cm per side in MNI space. In this way, the SVM input features for fMRI were restricted to traces averaged across local tissue. This is analogous, for fairness of comparison between the modalities, to the volume of tissue interrogated by one fNIRS probe, which is on the order of centimeters (Boas et al., 2004).

Classification was performed with a fMRI trace selected from the contralateral motor area, which was paired with one from the DLPFC region (a co-activating pair, noted as DLPFC and Motor in **Figure 4**). The same DLPFC trace was paired with one from the MFG region (an across-network pair, noted as DLPFC and MFG in **Figure 4**), and classification was performed again. The same DLPFC trace was then paired with a second DLPFC trace from a region 2 cm superior and 2 cm medial to the first DLPFC trace in MNI space (a within-network pair, noted as DLPFC + DLPFC2 in **Figure 4**), and classification was performed a third time. To optimize accuracy, c was tested at 0.0001, 0.001, 0.01, 0.1, 1, 5, 10,100,1000, and g was tested at 0.00001, 0.0001, 0.001, 0.01,0.1,1, 5, 10, 100 (a total of 81 cases) in each classification step. Otherwise, all methods for classification were the same as those

**FIGURE 4 | Classification accuracy averaged across four runs for each of seven participants.** Two fMRI time traces were used as support vector machine input features in each case. Accuracy is the number of time points for which the prediction matched the truth label out of all 200 volumes. An accuracy of 50% represents prediction by chance. Error bars represent ± one standard deviation. **Left-hand bars**: co-activating pairs (blue). **Center bars**: within-network pairs (green). **Right-hand bars**: cross-region pairs (red).

**Table 2 | The best** *c***,** *g* **parameters found after optimization for each participant for fMRI classification, and the time trace correlations (***r***) averaged across that participant's four runs.**


for the fNIRS traces described above. The SVM tuning parameters producing the best classification accuracies after optimization are given in **Table 2**. The accuracies are summarized in **Figure 4**.

The motor region was selected for its robust and reliable response during task periods. We treat classification based on time traces from the motor region as a gold standard. That is, we do not expect fNIRS classification accuracies to exceed those attainable using motor cortex activations measured with fMRI.

#### **RESULTS**

#### **BEHAVIORAL TASK RESULTS**

As described above, five participants completed the fNIRS study and seven participants completed the fMRI study. One participant performed the task incorrectly for incongruent trials. Thus, this participant's data were excluded from the behavioral analyses. However, since this participant was engaged in the task and responding to stimuli, these trials were not excluded from the fNIRS and fMRI analyses. Including these data was appropriate because the fNIRS and fMRI analyses were aimed at distinguishing between performing a task and resting, rather than distinguishing between incongruent and congruent trials. Mean accuracy was 98% (*SD* = 2*.*2%, *N* = 4) for the four fNIRS participants who performed the task correctly, and 97% (*SD* = 4*.*7%, *N* = 6) for the six fMRI participants who performed the task correctly. As expected, mean accuracy was relatively high.

Also as expected, performance was worse in incongruent than in congruent trials (the analysis of the data included only the participants who performed correctly in most incongruent trials). In a random effects analysis, mean reaction time was significantly higher in the incongruent condition (fNIRS: *M* = 0.639 s, *SD* = 0*.*035, *N* = 4; fMRI: *M* = 0.789 s, *SD* = 0*.*114 s, *N* = 6) than in the congruent condition (fNIRS: *M* = 0.552 s, *SD* = 0*.*033 s, *N* = 4; fMRI: *M* = 0.714 s, *SD* = 0*.*096 s, *N* = 6), [fNIRS: *t(*3*)* = 26, *p <* 0*.*0005; fMRI: *t(*5*)* = 9*.*1, *p <* 0*.*0005]. Likewise, mean error rate was significantly higher in the incongruent condition (fNIRS: *M* = 2.0%, *SD* = 1*.*8%, *N* = 4; fMRI: *M* = 4.9%, *SD* = 6*.*4%, *N* = 6) than in the congruent condition (fNIRS: *M* = 0.38%, *SD* = 0*.*81%, *N* = 4; fMRI: *M* = 2*.*1%, *SD* = 3*.*5%, *N* = 6), [fNIRS: *t(*3*)* = 2*.*9, *p <* 0*.*05; fMRI: *t(*5*)* = 2*.*2, *p <* 0*.*05). Errors of omission were rare and thus not analyzed.

#### **fNIRS RESULTS**

The four-run average classification accuracy for each of five participants is presented in **Figure 2**. As expected, we were able to distinguish between task engagement and rest. In particular, both averages differed significantly from chance at 50% [across: *t(*4*)* = 9*.*65, *p <* 0*.*0005; within: *t(*4*)* = 4*.*95, *p <* 0*.*005]. Also in line with predictions, there was a non-significant trend toward greater classification accuracy for across-network pairs (*M* = 69.1%, *SD* = 4*.*4%) than for within-network pairs (*M* = 66.0%, *SD* = 7*.*2%), [*t(*4*)* = 1*.*48, *p <* 0*.*25]. The probe channels resulting in oxygenated or deoxygenated [Hb] traces with the highest task model fit parameters were 1, 2 and 3 (as numbered in **Figure 1**, left; data not shown). The [Hb] species of the bestfitting traces were nearly evenly split: eight were oxygenated and seven were deoxygenated [Hb] traces. Thus, we found no universally best Hb species for fitting the task model. This result is consistent with prior suggestions that a probe's sensitivity to one species or the other depends on whether it mostly samples the arterial or venous compartment (Strangman et al., 2003), which is likely to change for every probe application.

Prior work indicates that activity in key regions of the "task-positive" network is positively correlated while activity in key regions of the "task-positive" and "task-negative" networks is negatively correlated (Fox et al., 2005; Kelly et al., 2008). Consistent with such findings, the group-averaged correlations for within-network pairs were significantly greater than zero [*r* = 0*.*75, *SD* = 0*.*29; *t(*4*)* = 5*.*72, *p <* 0*.*005], while those for acrossnetwork pairs were significantly less than zero [*r* = −0*.*12, *SD* = 0*.*08; *t(*4*)* = 3*.*41, *p <* 0*.*025]. Further, the correlation averages for within-network pairs were significantly higher than those of the across-network pairs [*t(*4*)* = 5*.*56, *p <* 0*.*005; see **Table 1** for a participant-specific list of correlation values]. These findings suggest that our fNIRS probes accurately measured activity in the "task-positive" and "task-negative" networks.

Finally, since across-network correlation was determined on a per-participant basis, we also wished to verify that negatively correlated across-network activity was observed at the group level. To this end, we averaged the across-network functional task signals across all 20 runs. For the purposes of this group average, all deoxygenated traces were first inverted, consistent with a reduction of deoxygenated [Hb] during activation. The group-averaged time traces are presented in **Figure 3**, which shows filtered, normalized and corrected [Hb] changes [see fNIRS methods; truth labels (green trace) show task at +1 and rest at −1]. As expected, the across-network correlation determined in this fixed effects analysis was significantly less than zero [*r(*18*)* = -0.58, *p <* 0*.*01]. This finding illustrates that, even at the group level, DLPFC activity (blue trace) increased during task performance while MFG activity (red trace) decreased.

#### **fMRI RESULTS**

The four-run average classification accuracy for each of the seven participants is presented in **Figure 4**. Replicating the fNIRS results, group averaged classification accuracy was significantly greater than chance at 50% for all three types of region pairs [across: *t(*6*)* = 3*.*56, *p <* 0*.*01; within: *t(*6*)* = 2*.*56, *p <* 0*.*025; coactivating: *t(*6*)* = 7*.*96, *p <* 0*.*0005], indicating we were able to distinguish task engagement from rest. Also as expected, group averaged classification accuracy was significantly higher for the co-activating (DLPFC and motor) pairs (*M* = 74*.*1%, *SD* = 8*.*0%) than for the across-network pairs (*M* = 65.6%, *SD* = 11*.*6%), [*t(*6*)* = 4*.*83, *p <* 0*.*005], consistent with the high reliability of motor cortex activation detection in fMRI studies (Möller et al., 2005) and with the motor cortex activation associated with button press responses in the present task. Within-network pair accuracy (*M* = 63.0%, *SD* = 13*.*5%) tended to be lower than across-network pair accuracy, but not significantly [*t(*6*)* = 1*.*57, *p <* 0*.*1].

Also consistent with the fNIRS results, the group-averaged correlations for co-activating (*r* = 0*.*53, *SD* = 0*.*23) and withinnetwork pairs (*r* = 0*.*54, *SD* = 0*.*26) were significantly greater than zero [within: *t(*6*)* = 6*.*07, *p <* 0*.*0005; co-activating: *t(*6*)* = 5*.*49, *p <* 0*.*005]. In contrast, those for the across-network pairs (*r* = 0*.*17, *SD* = 0*.*25) did not differ from zero [*t(*6*)* = 1*.*81, *p <* 0*.*1]. As predicted, however, they were significantly lower than those for the within-network pairs [*t(*6*)* =2.29, *p <* 0*.*05; see **Table 2** for a participant-specific list of correlation values].

The locations of statistically significant activations for the MSIT, after second-level analysis across four runs each from seven independent participants, are shown for the DLPFC [(52, 14, 32) in MNI space; **Figure 5**, left] and for the MFG [(22, 66, 0) in MNI space; **Figure 5**, right]. The *t*-statistic is mapped, with the threshold set at an uncorrected significance level of *p <* 0*.*001 for the work minus rest contrast shown on the left, and at *p <* 0*.*01 for the rest minus work contrast shown on the right. The MFG activation was not present at the higher threshold but appeared at the lower threshold. Of importance, the expected "task-positive" and "task-negative" hemodynamic activations occurred in the same regions that were interrogated by the fNIRS probes. Notably, even with only seven participants, the DLPFC survived a family wise error correction at *p <* 0*.*05; the MFG, however, did not survive this correction.

Finally, we note that although other regions of the "tasknegative" network were identified in the fMRI analysis, they may be less useful for monitoring task engagement with fNIRS. First, lateral parietal regions were not consistently activated bilaterally across participants. Thus, monitoring both sides with fNIRS would present greater difficulty due to the increased number of optical probes. Second, although precuneus and PCC regions of the task-negative network (Raichle et al., 2001; Greicius et al., 2003) were reliably activated (**Figure 5**, right), they are too deep to be accessible via fNIRS probes, which can interrogate only the outer layers of the cortex (Boas et al., 2004).

#### **DISCUSSION**

In the present study, we investigated whether functional neuroimaging methods (i.e., fNIRS and fMRI) can be employed to distinguish periods of task engagement from periods of rest. As described below, our findings support this view. They also provide valuable information about which brain activations may prove most useful for monitoring task engagement in the field.

Our first set of findings came from fNIRS experiment. Here, we found that multivariate pattern classification techniques could distinguish between periods of task performance and periods of rest based on brain activity recorded from (a) different regions of the DLPFC in the task-positive network (a within-network pair) or (b) the DLPFC in the task-positive network and the MFG in the task-negative network (an across-network pair). Further, there was a trend toward higher classification accuracy for acrossnetwork pairs than for within-network pairs. Indeed, accuracy with across-network pairs approached 70%, even with the basic processing methods described here (with adaptive physiological filtering and additional probes, accuracy may further improve). This result fits with prior data suggesting that variability in task engagement is associated with variability in activity and/or functional connectivity involving both the "task-positive" and the "task-negative" networks (e.g., Weissman et al., 2006; Prado and Weissman, 2011). Most important, our fNIRS findings indicate that online recordings of brain activity via fNIRS may provide a valuable tool for detecting varying levels of task engagement in the real world.

Our second set of findings came from an fMRI study. Of importance, these findings both verified and extended the results

expected "task-positive" and "task-negative" hemodynamic activations occurred in the same regions that were interrogated by the fNIRS probes. **Left**: the *t*-statistic is mapped, with the height threshold set at an uncorrected significance level of *p <* 0*.*001 and the extent threshold

*t*-statistic is mapped, with the height threshold set at an uncorrected significance level of *p <* 0*.*01 and the extent threshold set at 10 voxels. MFG is shown at [22, 66, 0] in MNI space for the rest minus work contrast. (Crosshair: *x* = 0, *y* = 66, *z* = 14).

of the fNIRS study discussed earlier. First, we observed activations and deactivations, respectively, in the DLPFC and the MFG, which verified that our functional neuroimaging paradigm engaged the task-positive and task-negative networks. Second, further analyses revealed that these activations occurred in the same DLPFC and MFG regions that were activated in the fNIRS experiment, wherein activity was measured with probes on the scalp. Third, the trend toward higher classification accuracy for across-network pairs than for within-network pairs observed with fNIRS was also observed with fMRI. Fourth, the fMRI findings indicated a possible upper bound on classification accuracy for distinguishing between task performance and rest: as expected, the highest classification accuracy was observed when the DLPFC trace and co-activating (for this task) motor cortex trace served as inputs to the SVM classifier. Together, these fMRI findings verified that our fNIRS recordings reflected activity in the key regions under investigation (DLPFC and MFG). They also replicated the fNIRS results and extended them by suggesting an upper bound for classification accuracy based on relatively focal hemodynamic activity.

#### **NOVEL CONTRIBUTION OF THE PRESENT WORK**

The present findings make an important contribution to the field. Specifically, they show, for the first time, that it is possible to detect negative correlations between activity in key regions of the "task-positive" and "task-negative" networks with fNIRS. Further, they show that the detection of activity in the "task-negative" network is useful for distinguishing between high and low levels of task engagement. This capability might prove useful in future applications of fNIRS that are aimed at discriminating between optimal behavioral performance (where a negative correlation is expected) and internally-guided thought (where co-activation and, hence, a positive correlation is expected) (Christoff et al., 2009; Smallwood et al., 2012). Thus, it could function to improve the predictive power of a fNIRS-based cognitive state monitoring system. Finally, although our findings make a novel contribution to the field, it is important to note that they build on previous work showing that frontal oxygenation is sensitive to workload (Izzetoglu et al., 2004) and that fNIRS can reliably detect both resting state physiology and functionally-connected networks (White and Culver, 2008; Mehnert et al., 2009; Mesquita et al., 2010).

#### **LIMITATIONS**

While across-network pairs were associated with stronger negative correlations and higher classification accuracy, relative to within-network measures (data not shown), negative correlations between DLPFC and MFG activity were not observed in every participant. This lack of consistency may stem from a variety of sources, including non-optimal fNIRS probe localization, variation in participant compliance or strategy, interference from physiological or motion artifact, variable fMRI voxel selection, and co-activation of key regions in the "task-positive" and "task-negative" networks during mind wandering or internallyguided thought (Christoff et al., 2009; Smallwood et al., 2012). Future studies should be conducted to distinguish among these possibilities and to determine which methodologies provide more consistent measures of negatively correlated activity in the taskpositive and task-negative networks.

Also regarding the consistency of our measures, classification accuracy varied considerably across runs (see the error bars in **Figures 2**, **4**). Future studies might therefore be conducted to investigate the source(s) of this variability as well as the impact of other sources of variability (e.g., across-visit, across-participant and across-task) on classification accuracy. Such studies might also investigate the impact of using the known task model to clean the measured traces when producing functional task signals for use in classification (see fNIRS data processing), which may have biased the classifier toward higher accuracy in the present study.

Another study limitation stems from the fact that some taskevoked systemic signals are measureable on the scalp surface, and that at least one such signal—skin blood volume—depends on cognitive state (Kirilina et al., 2012). Since fNIRS is sensitive to hemodynamics in superficial tissue at all source-detector separation distances, it is possible that systemic signals in the superficial tissue may have driven the negative across-network correlations that we observed. To investigate this possibility, we quantified correlations for the traces taken from the across-network shallow source-detector pairs (see fNIRS data acquisition). Of importance, no association was observed between the correlation values for the superficial traces and the correlation values for the deep traces, whether corrected or not (data not shown). Thus, the negative correlations that we measured between the across-network deep traces likely reflected [Hb] related changes in brain tissue rather than superficial physiological signals. However, if skin blood changes provide additional information about the task engagement, then it could be useful in future studies to include such changes directly as classifier inputs.

Finally, we note that motor activation is not always a reliable component of task engagement as some tasks require sustained attention over long periods in the absence of overt responses (e.g., instrument cross-checking and visual display searches). Thus, future studies may wish to focus on our fNIRS finding that classification accuracy was slightly higher for across-network pairs (i.e., pairs in which one region came from the task-positive network while the other came from the task-negative network) than for within-network pairs (i.e., pairs in which both regions came from the task-positive network). As we mentioned earlier, this finding fits with the view that task engagement is determined by interactions between these networks (Fox et al., 2005; Weissman et al., 2006; Kelly et al., 2008; Prado and Weissman, 2011).

#### **FUTURE WORK**

Future work could address whether adaptive filtering of fNIRS traces over smaller time windows improves the activation-based classification measures reported here. This may include investigating adaptive physiological noise removal driven by the correlation between the deep and shallow traces (Harrivel et al., 2012) or motion artifact reduction based on the frequency domain phase signal (Harrivel and Hearn, 2012). An increase in probe density could also be used to improve localization within the regions of interest. Finally, measures of network correlation could be quantified over shorter time scales to determine whether transient internally-guided thought can be distinguished from periods of "zoning out." Ongoing simultaneous fNIRS/fMRI studies are further examining these and other possible methods for improving our ability to discriminate between varying levels of task engagement.

### **CONCLUSION**

In the present study, we used a combination of fNIRS and fMRI results to show that online recordings of brain activity from the task-positive and task-negative networks can be used to detect moment-to-moment changes in task engagement. We hope that future studies combining fNIRS recordings with multivariate classification methodologies will build upon the present work to produce robust systems for detecting changes in task engagement in real-world settings.

#### **AUTHORS CONTRIBUTIONS**

Angela R. Harrivel is the PI on the human subject study. She recruited and consented participants, designed fNIRS head probes, wrote all scripts for the fNIRS data processing, collected and processed all data, performed analyses and wrote the manuscript. Daniel H. Weissman wrote the code for the MSIT presentation, provided guidance regarding psychological aspects of the study and fMRI analyses, and edited the manuscript. Douglas C. Noll provided funding support, guidance regarding all fMRI aspects of the project and all data processing methods, and reviewed the manuscript. Scott J. Peltier is Co-I on the study. He contributed to analyses, edited the manuscript, and provided guidance regarding data collection and processing techniques, pattern classification methods, and resting state analyses.

#### **ACKNOWLEDGMENTS**

This work was supported by the University of Michigan fMRI Laboratory and NASA's Aviation Safety Program. Colleagues at the NASA Glenn and Langley Research Centers are appreciated, especially Jeffrey Mackey, Daniel Gotti and Padetha Tin for head probe design and assembly, and Tristan Hearn and Alan Pope for helpful review. We are grateful for the assistance of the fMRI laboratory at the University of Michigan for the collection and pre-processing of the fMRI data, and Ted Huppert of the University of Pittsburgh for review and invaluable guidance regarding general fNIRS techniques and analyses.

#### **REFERENCES**


utive system contributions to mind wandering. *PNAS*. 106, 8719–8724. doi: 10.1073/pnas.0900234106


stress. *Proc. Hum. Factors Ergon. Soc. Annu. Meet.* 48, 1150–1154. doi: 10.1177/154193120404801001


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 September 2013; paper pending published: 22 October 2013; accepted: 26 November 2013; published online: 13 December 2013.*

*Citation: Harrivel AR, Weissman DH, Noll DC and Peltier SJ (2013) Monitoring attentional state with fNIRS. Front. Hum. Neurosci. 7:861. doi: 10.3389/fnhum. 2013.00861*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Harrivel, Weissman, Noll and Peltier. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Identifying and quantifying main components of physiological noise in functional near infrared spectroscopy on the prefrontal cortex

#### *Evgeniya Kirilina1 \*, Na Yu2, Alexander Jelzow3, Heidrun Wabnitz 3, Arthur M. Jacobs <sup>1</sup> and Ilias Tachtsidis <sup>2</sup>*

*<sup>1</sup> Department of Education and Psychology, Dahlem Institute for Neuroimaging of Emotion, Free University of Berlin, Berlin, Germany*

*<sup>2</sup> Department of Medical Physics and Bioengineering, University College London, London, UK*

*<sup>3</sup> Physikalisch-Technische Bundesanstalt (PTB), Berlin, Germany*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Felix Scholkmann, Biomedical Optics Research Laboratory, Switzerland*

*Takusige Katura, Hitachi, Ltd., Japan*

#### *\*Correspondence:*

*Evgeniya Kirilina, Department of Education and Psychology, Dahlem Institute for Neuroimaging of Emotion, Free University of Berlin, Habelschwerdter Allee 45, 14195 Berlin, Germany e-mail: kirilina@zedat.fu-berlin.de*

Functional Near-Infrared Spectroscopy (fNIRS) is a promising method to study functional organization of the prefrontal cortex. However, in order to realize the high potential of fNIRS, effective discrimination between physiological noise originating from forehead skin haemodynamic and cerebral signals is required. Main sources of physiological noise are global and local blood flow regulation processes on multiple time scales. The goal of the present study was to identify the main physiological noise contributions in fNIRS forehead signals and to develop a method for physiological de-noising of fNIRS data. To achieve this goal we combined concurrent time-domain fNIRS and peripheral physiology recordings with wavelet coherence analysis (WCA). Depth selectivity was achieved by analyzing moments of photon time-of-flight distributions provided by time-domain fNIRS. Simultaneously, mean arterial blood pressure (MAP), heart rate (HR), and skin blood flow (SBF) on the forehead were recorded. WCA was employed to quantify the impact of physiological processes on fNIRS signals separately for different time scales. We identified three main processes contributing to physiological noise in fNIRS signals on the forehead. The first process with the period of about 3 s is induced by respiration. The second process is highly correlated with time lagged MAP and HR fluctuations with a period of about 10 s often referred as *Mayer waves*. The third process is local regulation of the facial SBF time locked to the task-evoked fNIRS signals. All processes affect oxygenated haemoglobin concentration more strongly than that of deoxygenated haemoglobin. Based on these results we developed a set of physiological regressors, which were used for physiological de-noising of fNIRS signals. Our results demonstrate that proposed de-noising method can significantly improve the sensitivity of fNIRS to cerebral signals.

#### **Keywords: fNIRS, physiological noise, wavelet coherence, de-noising methods**

#### **INTRODUCTION**

Functional Near-Infrared Spectroscopy (fNIRS) is a powerful tool to study functional organization of the prefrontal cortex (Scholkmann et al., 2013c). Due to absence of hair on the forehead, and relatively short distance between the forehead surface and the frontal cortex (Okamoto et al., 2004) this cortical area is well accessible by near-infrared light. However, despite these beneficial biophysical circumstances physiological noise generally limits overall fNIRS sensitivity and specificity on the prefrontal cortex (Tachtsidis et al., 2009; Aletti et al., 2012; Gagnon et al., 2012; Kirilina et al., 2012). Physiological fNIRS noise is induced by fluctuations in both blood volume and blood oxygenation in the extra- and intracranial tissues. Global and local blood flow regulation processes in these anatomical compartments lead to oscillations on multiple time scales. As a result fNIRS sensitivity to functional neuronal signals on the single subject level deteriorates and additional variance is added on the group level, due to inter-subject variability of the systemic and skin physiology.

In literature a number of methods were proposed for the separation of physiological noise from the cerebral activation (Saager and Berger, 2005; Katura et al., 2008; Gregg et al., 2010; Saager et al., 2011; Aletti et al., 2012). Comprehensive review of these methods can be found in Scholkmann et al. (2013c). Here we briefly summarize some of them. The majority of these approaches may be subdivided into three classes. Among them the most powerful methods are based on the idea of superficial signal regression. These methods require additional fNIRS channels with short spatial source-detector separation (Saager and Berger, 2005; Gregg et al., 2010; Saager et al., 2011; Gagnon et al., 2012; Funane et al., 2013). Nevertheless, these approaches still fail in removing physiological noise components originating from the cerebral compartment. Secondly, transient differences of functional signals and physiological noise have been exploited by a variety of methods (Kohno et al., 2007; Zhang et al., 2009, 2012b; Tanaka et al., 2013). These methods typically assume statistical independence between physiological noise and cerebral signal and fail to separate task-evoked responses of physiological parameters. In the third class, separation of forehead noise from cerebral fNIRS signals is achieved by exploring additional information from concurrent recordings of global systemic physiology, e.g., blood pressure and heart rate (HR) or local physiological parameters, e.g., skin blood flow (SBF) (Franceschini et al., 2006; Rowley et al., 2007; Tachtsidis et al., 2010; Minati et al., 2011; Patel et al., 2011; Takahashi et al., 2011; Sato et al., 2013). However, this separation is challenging, since the precise physiological fNIRS noise mechanisms are unknown. This shortcoming results from the lack of robust physiological models capable of relating physiological parameters like e.g., blood pressure or SBF with the fNIRS observable, tissue haemoglobin concentration. Further complications arise from the complex structure of physiological noise. Physiological noise involves components originating from both cerebral and extra-cerebral tissue, and global systemic as well as local tissue specific regulatory processes. Moreover different physiological processes dominate at different time scales.

The goal of the present study is to improve this situation by identifying the global systemic and local tissue-specific physiological noise processes in fNIRS recordings at the forehead and to quantify their relative impact in a group analysis.

Below we shortly review the literature on physiological processes which could potentially contribute to the physiological noise in fNIRS at different time scales. The most prominent, however, unproblematic, component is the heart pulsation with frequencies around 1 Hz, which may be effectively removed by low-pass filtering, since the corresponding frequency band is well distinguished from that of hemodynamic responses. Simple detrending or high pass filtering also removes very slow drifts on the time scale of several minutes. The major concerns for fNIRS data quality are oscillations in the range from 0.2 to 0.005 Hz, since they account for the main components of physiological noise. Moreover, these fluctuations may in the worst case be synchronized with the task and might thereby lead to false positives in fNIRS activation maps (Tachtsidis et al., 2009).

Based on physiological considerations it is common to distinguish three distinct bands: high frequency oscillations (HFOs) ranging from 0.2 to 0.5 Hz, low frequency oscillations (LFOs) from 0.08 to 0.15 Hz and very low frequency fluctuations (VLFOs) from 0.02 to 0.08 Hz. In the following we will first focus on global systemic and then on local regulatory processes specific to cerebral or to extra-cerebral compartments.

HFOs in fNIRS signals, with frequencies around 0.3 Hz, are predominantly induced by direct or mediated influence of respiration. In the following we therefore refer to the frequency band around 0.3 Hz as R-band.

High variability in global systemic parameters of blood pressure and HR was found in the frequency band of LFOs (Julien, 2006). First reported by Mayer in 1876 (Mayer, 1876), these slow blood pressure waves are usually referred to as Mayer waves and the corresponding frequency band as M-band. In how far Mayer waves are propagated into cerebral and skin compartments remains controversial (Tong and Frederick, 2010; Tong et al., 2011).

In the cerebral compartment LFOs, independent from blood pressure fluctuations, were observed with Laser Doppler flowmetry (LDF) (Hudetz et al., 1992) and optical imaging in animals (Mayhew et al., 1996) and humans (Rayshubskiy et al., 2013) and with fNIRS on the human cortex (Elwell et al., 1999; Obrig et al., 2000; Tachtsidis et al., 2004).

Also SBF exhibits LFOs and VLFOs. LFOs in the M-band were observed even in isolated vessels (Johansson and Bohr, 1966) and were shown to be related to the activity of vessel walls. The SBF fluctuations in VLFO band originate most likely from sympathetic control of the peripheral vasculature (Kastrup et al., 1989; Söderström et al., 2003). The forehead skin vasculature constitutes a separate layer of sympathetic and parasympathetic innervations (Drummond, 1996, 1997). This fact allows for a synchronization between skin VLOFs and certain type of tasks in the forehead (Tachtsidis et al., 2009; Kirilina et al., 2012). Since the typical period of stimulation used in block design fNIRS studies often corresponds to VLFO frequency band, we refer to this band as A-band (activation) throughout this manuscript.

In summary, physiological fNIRS noise originates from different global systemic and local regulatory processes, each dominating at a different time scale. The goal of the present study was to identify the dominating processes and quantify their relative impact. We furthermore aim to develop a method to account for the contribution of each noise process in the fNIRS analysis, and therefore provide a method for physiological de-noising of fNIRS data. In order to achieve this challenging task we combined depth selective time-domain fNIRS measurements with concurrent recordings of global systemic peripheral physiological measurements of respiration, mean arterial blood pressure (MAP) and HR as well as a local SBF recordings on the forehead. Temporal disentanglement of the different physiological processes was realized by an advanced WCA. This method, originally developed in the field of geo science and meteorology (Torrence and Webster, 1999; Grinsted et al., 2004) but also applied to fNIRS signals analysis (Rowley et al., 2007; Li et al., 2010, 2013; Zhang et al., 2012a). WCA allows to target the coherent content of two temporal signals specifically at multiple time scales. Based on results of the WCA we developed a physiological de-noising method for fNIRS signals based on General Linear Modeling (GLM) (Kiebel and Holmes, 2007) and auxiliary physiological regressors (Tachtsidis et al., 2010). This approach allowed us to develop a novel versatile but yet robust physiological fNIRS de-noising method.

In the Theory section we briefly describe the theoretical background of wavelet transform and WCA used in this study to decompose physiological noise in fNIRS into components coherent with particular physiological processes. In the Methods section we describe the setup used for timedomain fNIRS experiments on the forehead. Based on the findings summarized in the Results section we discuss the possible origin of three processes contributing to the physiological noise and analyse the performance of the developed de-noising technique.

#### **THEORY**

#### **WAVELET TRANSFORM**

Continuous wavelet transform allows for constructing timefrequency representations of a signal with optimal resolution for each frequency band. Continuous wavelet transform of a time dependent signal *S*(*t*) is defined as (Mallat, 2009):

$$W\_{\mathcal{S}}(\alpha, t\_0) = \frac{1}{\sqrt{\alpha}} \int\_{-\infty}^{\infty} S(t) \psi^\* \left(\frac{t - t\_0}{\alpha}\right) dt\tag{1}$$

Where *t* and *t*<sup>0</sup> are time and time shift, respectively, α is a scale (of dimension time), and *WS* (α, *t*0) represents the wavelet transform of signal *S*(*t*). ψ(*t*) is the mother wavelet function. In the present study a Morlet wavelet (Goupillaud et al., 1984) was used as mother wavelet function. This complex function may be expressed as product of a harmonic function and a Gaussian:

$$
\psi(s) = \frac{1}{\sqrt{\pi f\_b}} e^{i2\pi f\_s s} e^{-\frac{s^2}{f\_b}} \tag{2}
$$

where *fc* and *fb* are dimensionless parameters, determining the wavelet center frequency and wavelet bandwidth, respectively, and *s* = *<sup>t</sup>* <sup>−</sup> *<sup>t</sup>*<sup>0</sup> <sup>α</sup> is a dimensionless variable. We used *fc* = 1 and *fb* = 3, a parameter set that provides optimal trade-off between time and frequency resolution. *WS* (α, *t*0) is a complex function of scale α and position *t*0. If scale α is greater than one, the mother wavelet function ψ is stretched in the time domain. If α is smaller than one (but positive) the function is compressed in time. The scaling factor √ 1 <sup>α</sup> in Equation (1) is to ensure the energy normalization for all values of scale α.

From Equation (2) one can see that the relationship between wavelet scale α and pseudo-frequency *fa* is given by:

$$f\_a = \frac{f\_c}{\alpha} \tag{3}$$

In our case the relationship between the scale α and the pseudofrequency simplifies to *fa* = <sup>1</sup> α .

To exemplify the above described algorithm, the wavelet transform of a synthetic signal *S*(*t*) is outlined from left to right in **Figure 1A**.

The synthetic signal *S*(*t*) is a sum of three weighted sinusoidal signals with frequencies of 0.3 Hz (R-band), 0.1 Hz (M-band), and 0.034 Hz (A-band) and white Gaussian noise. The signal *S*(*t*) models a hypothetical physiological noise process with three components of different physiological origin.

The absolute values of *WS* (α, *t*0) are depicted as 2D wavelet scalogram in the right part of **Figure 1A**. This representation has clear maxima at the three scales 3, 10, and 33 s, corresponding to the inverse frequencies of the components present in the signals.

Please note that the width of the large-scale (low-frequency) component is proportionally larger, than that of the low-scale

**FIGURE 1 | (A)** Scheme of the wavelet transform. The synthetic signal *S*1(*t*) (left) is convolved with complex mother wavelet function stretched with the scale parameter α (middle) to provide 2D wavelet scalogram on the right side. The real part of mother Morlet wavelet is plotted with red and imaginary part plotted with a blue line. Sign <sup>∗</sup>indicates cross-correlation operation. The magnitude of the complex 2D wavelet scalogram is shown in color code. The synthetic curve *S*2(*t*) contains three harmonic components with the frequencies 0.3, 0.1, and 0.033 Hz, respectively, as well as additional white noise. The scalogram on the right hand side clearly demonstrates three components at scales 3, 10, and 34 s corresponding to the inverse

frequencies of the signal components. **(B)** Example of a 2D wavelet coherence calculated for two synthetic signals with three coherent components. WCTS1 on the left and WCT<sup>∗</sup> S2 in the middle are complex 2D wavelet scalograms of two signals (plotted as color coded magnitude values), both containing phase shifted coherent components with frequencies 0.3, 0.1, and 0.034 Hz, respectively, and additional non-coherent white noise. Sign × indicates scalar product operation. On the right side the magnitude value of wavelet coherence WCOH(*S*1, *S*2, α) is shown. Three peaks on the WCOH plot correspond to the three coherent components in the two signals in R-, M-, and A bands.

(high-frequency) component. This reflects an intrinsic wavelet transform property, which adjusts the frequency resolution to the frequency band.

#### **WAVELET COHERENCE ANALYSIS**

WCA is a tool to evaluate the correlation of two time dependent signals separately at different time scales.

Wavelet coherence of two signals *S*1(*t*) and *S*2(*t*) is defined as follows (Torrence and Webster, 1999; Grinsted et al., 2004):

$$\overline{\text{WCOOH}}(\alpha) = \frac{R\_{X,Y}\left(W\_{S\_1}, W\_{S\_2}, \alpha\right)}{\sqrt{|R\_{X,X}\left(W\_{S\_1}, \alpha\right) \cdot R\_{X,X}\left(W\_{S\_2}, \alpha\right)|}} \tag{4}$$

*RX*, *<sup>Y</sup>* (*WS*1, *WS*2, α) indicates the covariance or scalar product of the wavelet coefficients, *WS*<sup>1</sup> (α, *t*0), and *WS*<sup>2</sup> (α, *t*0) of signals *S*<sup>1</sup> and *S*<sup>2</sup> at scale α. *RX*, *<sup>X</sup>*(*WS*<sup>1</sup> , α) and *RX*, *<sup>X</sup>*(*WS*<sup>2</sup> , α) denote the autocorrelation or power of *WS*<sup>1</sup> (α, 0) and *WS*<sup>2</sup> (α, 0). In other words, the complex function WCOH(α) represents the crossspectral power in two time series as a fraction of the total power of the series. The absolute value of wavelet coherence WCOH(α) ranges from 0 to 1 with 1 indicating strongest correlation. The phase of the wavelet coherence provides information about the time lag between two signals. We define a time lag between two signals at scale α as the product of α and the wavelet coherence phase at this scale (expressed in radians).

Note that instead of conventionally performed smoothing in both time and scale dimensions (Torrence and Webster, 1999), we calculated only an average over the whole measurement period, but omitted smoothing in the scale dimension.

To illustrate the ability of WCOH(α) to detect coherent content in two signals at multiple time scales, **Figure 1B** provides an example of WCA of two synthetic signals with coherent components in three frequency bands. Two signals *S*1(*t*) and *S*2(*t*) are hypothetical physiological signals containing coherent oscillations in R-, M-, and A-bands. From left to right 2D wavelet scalograms of *S*1(*t*) and *S*2(*t*) are plotted together with the wavelet coherence (outmost right plot). The first signal *S*1(*t*) is identical to the example provided in **Figure 1A**. The second signal *S*2(*t*) also contains white Gaussian noise added to the sinusoidal components. *S*2(*t*) contains the same frequencies as *S*1(*t*) (0.3, 0.1, and 0.034 Hz), but with different phases and weights for the three components.

The wavelet coherence plot in **Figure 1B** shows three distinct maxima at 3, 10, and 34 s (equal to inverse frequencies of signal components), corresponding to three coherent components contained in the signals *S*1(*t*) and *S*2(*t*). In contrast to a simple correlation analysis WCOH(α) provides detailed frequency information about the coherent content in both signals.

In the current study we explore this property of WCA in order to analyze coherence between global systemic and local physiological processes and fNIRS signals at different time scales. In this way we are able to decompose physiological noise in fNIRS into components induced by several physiological processes.

Note the different widths of the maxima at the three different scales in the **Figure 1B**. Due to the wavelet transform properties, the frequency resolution degrades with the increasing scale α. However, the ratio between the resolution and α is independent of α.

#### **METHODS**

fNIRS and concurrently recorded physiological signals were acquired as part of a comparative fNIRS/fMRI experiment previously reported in (Kirilina et al., 2012). Here we only briefly describe one part of experimental settings including fNIRS and peripheral physiological measurements, which were used in the analysis presented. A detailed description of subject population, stimulation paradigm, NIRS instrumentation and fNIRS data pre-processing can be found in (Kirilina et al., 2012).

#### **SUBJECTS**

Fifteen healthy subjects (5 female, age 34.9 ± 7.2 years) took part in the present study. Due to technical reasons data from one subject was excluded from further analysis. During the experiment subjects were sitting in upright position in front of a computer monitor, while responding with right hand button presses. All subjects gave their informed consent to the experimental protocol, which was approved by a local ethics committee.

#### **STIMULUS**

A variation of continuous performance task (CPT) combined with a semantic categorization task was used to achieve the cerebral activation in the frontal lobe (bilateral Brodmann Area 10). A series of German words representing either concrete or abstract categories were continuously presented to the subjects in semantic (sem-CPT) and control (word-CPT) tasks. In sem-CPT the subjects were instructed to press the left button if a concrete word was presented after an abstract word. In any other case, they should press the right button. In word-CPT the subjects were instructed to press the left button if one particular target word 1 (VORZUG, German for "preference") followed target word 2 (KOFFER, German for "suitcase"), and right button in any other case. Both tasks were performed in nine 34.15 s long blocks in alternating order, interleaved by 31.11 s long baseline blocks. During the baseline blocks a fixation cross was presented in the middle of the screen. Pre- and post-baselines of 120 s were recorded for each subject.

#### **DATA ACQUISITION**

Concentration changes in oxygenated and deoxygenated haemoglobin, in the following referred as HbO and HbR, respectively, were measured by the PTB time-domain optical brain imager (Wabnitz et al., 2005, 2010). This device provides three wavelengths 689, 797, and 828 nm. The laser power was split to obtain two sources. Diffusely reflected light was collected by four detection fiber bundles and detected by fast photomultipliers connected to a multi-board time-correlated single photon counting (TCSPC) system. Distributions of photon time of flight (DTOFs) were acquired with time bins of 24.4 ps width and at a 20 Hz rate. A set of one source fiber and two detection fiber bundles was placed on the left and right forehead, respectively, along the Fp1–Fp2 line defined by the international 10–20 system, as illustrated in **Figure 2**. A source-detector separation of 3 cm was chosen for all fNIRS channels. To exploit the potential of time-domain fNIRS for the separation of extracranial and

cerebral signals, the measured DTOFs were analyzed in terms of statistical moments (Liebert et al., 2004). HbO and HbR time courses were extracted from changes in three different measures. These were (i) the 0th moment of DTOF, *m*0, corresponding to the total photon count, (ii) the 1st moment of DTOF, *m*1, i.e., the photon mean time of flight and (iii) the 2nd central moment of DTOF, the variance *V*. A detailed description of the procedure is given in Appendix 1 of (Kirilina et al., 2012). We would like to note that haemoglobin concentration changes based on *m*<sup>0</sup> are analogous to signals measured in conventional continuous wave (cw) fNIRS experiments. HbO and HbR based on *m*<sup>1</sup> and on the variance signal *V* have the advantage to be more sensitive to deeper and less sensitive to superficial absorption changes (Liebert et al., 2004; Wabnitz et al., 2005).

#### **PERIPHERAL PHYSIOLOGY**

Alongside with fNIRS recordings, the following global and local physiological signals were recorded: HR (from electrocardiography ECG), blood pressure, scalp blood flow, and respiration chest movement.

Respiration (RSP) and ECG were recorded using a Nexus-10 system (Mind Media, The Netherlands). The respiration was recorded with a Hall-sensor based respiratory belt placed over the subject's lower ribs. ECG was recorded with a sampling rate of 256 Hz, using two electrodes placed on the upper right and lower left part of the chest, respectively, and a ground electrode placed on the upper left chest.

HR was defined for each heartbeat as the inverse time interval between two subsequent R-peaks in the ECG time trace. A PortaPress system (TNO TPD Biomedical Instrumentation) was used to continuously measure MAP at the left hand index finger. HR and MAP signals originally measured for time points corresponding to heartbeats were re-sampled onto the equidistant 1 Hz time grid.

Changes in SBF were recorded by a floLAB Laser Doppler Perfusion Monitor (Moor Instruments) operating at an emitterdetector distance of about 1 mm. The laser Doppler probe was placed on the right forehead 15 mm above the NIRS source (see **Figure 2**). Due to the close placement of fNIRS detectors and laser Doppler probe the light emitted by the floLAB device was detectable by the td fNIRS system. This light appeared as a constant uncorrelated background in the td-NIRS measurement and was subtracted during the data preprocessing procedure. The additional contribution of the background to photon noise was insignificant.

#### **DATA ANALYSIS**

#### *Pre-processing*

Pre-processing, wavelet analysis and GLM analysis of fNIRS and physiological data was performed by own software written in MATLAB (R2012a, Mathworks Inc.). All signals were first filtered by a low-pass filter, with a cutoff frequency of 0.8 Hz to remove the variability due to the cardiac cycle. In addition, a high-pass filter with a cutoff frequency of 0.008 Hz was used to remove very slow signal and baseline shift variations (Mitsis et al., 2004). In both cases fifth order Butterworth filters in forward and backward direction were used for further data filtering. After filtering, all recorded measurements were down sampled to 1 Hz.

#### *Wavelet coherence analysis*

After preprocessing of HbO and HbR signals from four channels and three moments as well as the time dependent physiological traces for each subject, WCA was employed to investigate the coherence between physiological noise in fNIRS and peripheral physiological traces.

We used a complex Morlet mother wavelet as defined by function *cmor3-1*, [see Equation (2)] in the MATLAB Wavelet Toolbox. A complex wavelet transform was performed on an equidistant scale grid ranging from 0.2 to 50 s with 0.2 s steps, corresponding to pseudo-frequencies from 5 Hz down to 0.02 Hz. For each subject, wavelet coherence was calculated between four physiological traces and twelve haemoglobin concentration changes measured for the four channels and based on three moments. Magnitude and phase of the group average was calculated in the next step for each pair of signals. Based on the observed maxima of the magnitude of the group wavelet coherence we identified three bands. Mean phase differences and corresponding time lags between each signal pair were calculated within each scale band. The above described procedure of WCA between fNIRS and physiological signals is illustrated as a flowchart in **Figure 3**.

Both fNIRS and physiological signals contain a certain amount of measurement noise. Therefore, the obtained wavelet coherence values include a stochastic component induced by experimental noise. The noise propagation in wavelet coherence is not linear due to the presence of the denominator in Equation (4). Therefore, proper statistical analysis of the group average wavelet coherence values and their comparison becomes a highly complex challenge. This situation is further complicated by the fact that fNIRS signals based on different moments have different photon noise contributions and that the physiological traces have different frequency spectra.

In order to overcome this non-linear problem, we numerically estimated the impact of different levels of photon noise and uncorrelated physiological noise components on the group wavelet coherence values. This was achieved by adding artificial noise to experimental data. In the following the numerical

procedure to estimate the influence of these noise components is described in detail.

*Influence of photon noise on wavelet coherence.* In order to be able to correctly compare wavelet coherences obtained for fNIRS signals based on different moments, their different levels of photon noise were taken into account. In general, photon noise has increasing influence for haemoglobin concentrations based on higher order moments. We first theoretically estimated the standard deviation of the photon noise component in the measured HbO and HbR signals based on the three different moments. Theoretical and numerical details of this estimation are described in the Appendix. In a second step we performed numerical experiments, adding a controlled amount of synthetic white Gaussian noise to the (least noisy) measured *m*0-based HbO and HbR signals. The noise amplitudes were chosen in a way, that the photon noise in the synthetic *m*0-based signal matched the level of photon noise in *m*1- and *V*-based HbO and HbR signals, respectively. The group wavelet coherence, between synthetic "noise matched" signals and physiological traces, was calculated and compared to the wavelet coherence obtained for experimentally measured *m*1 and *V*-based signals.

*Uncorrelated physiological noise and wavelet coherence.* Furthermore, we numerically estimated the noise levels of the magnitude of group wavelet coherence obtained for uncorrelated physiological noise. For this purpose we calculated the wavelet coherence on the single subject level, between the fNIRS signals measured for each subject and physiological signals measured for different randomly chosen subjects. Additionally we circularly shifted the physiological traces with a random time shift, different for each subject in order to avoid inter-subject coherence effects potentially induced by the task. The wavelet coherences of individual subjects were subjected to a group analysis by calculating the absolute value of the group mean wavelet coherence value.

#### *General linear modeling and physiological de-noising*

As was described above, the proper statistical analysis of group wavelet coherence is mathematically challenging. Therefore, in this study we used the results of the group WCA only in a qualitative manner. Based on the maxima in wavelet coherence we identified the main processes contributing to physiological noise in fNIRS. We then used this information in order to create a set of auxiliary regressors for physiological noise modeling as described in the next session. Finally with the help of the GLM analysis informed by WCA we were able to statistically analyze the impact of each physiological process at the group level.

The GLM analysis was performed on the fNIRS signal time traces from the entire experimental sessions. Our model to analyze fNIRS signals included one regressor modeling task-related brain activation and auxiliary regressors modeling physiological noise. To create a regressor modeling task-related brain activation, boxcar functions of the task presentation were convolved with the standard haemodynamic response function as it is implemented in SPM8 software package (Friston and Stephan, 2007). The physiological noise regressors were constructed based on the results of the WCA.

For each physiological trace we determined a frequency band demonstrating maximal wavelet coherence with fNIRS signals. The time delay between fNIRS and physiological signals was determined in these bands, based on the phase of the group averages of wavelet coherence.

Physiological signals of the entire experimental session of each participant were then band-pass filtered at these frequency bands using a fifth order Butterworth filter in forward and backward direction. After filtering the signals were time shifted by the delay determined as described before. Filtered and time-shifted signals were used as regressors in the GLM analysis. The above described GLM de-noising procedure is illustrated as a flowchart in **Figure 4**.

Signal amplitudes corresponding to each regressor, obtained in the first level analysis for each subject, were subjected to a second level analysis, by calculating one-sample *T*-tests. For those values, which significantly differed from zero, with a significance level of *p* = 0.05 the group mean and standard deviation were calculated.

#### **RESULTS**

#### **WAVELET COHERENCE ANALYSIS**

**Figure 5A** shows time traces of the single channel HbO and HbR signals, as well as time traces of four physiological signals for one representative subject.

**Figure 6** shows the 2D wavelet scalograms of HbO and MAP and wavelet coherence between these two signals for one channel of a representative subject. In this particular case an increased coherence is observed for scales around 10 s, corresponding to the M-band.

**Figure 7** depicts absolute values of the group average of the wavelet coherence between transient haemoglobin

concentrations, e.g., (HbO and HbR) and the physiological signals MAP, HR, SBF, and RSP, respectively. The black solid curves in each subplot in **Figure 7** indicate the level of coherency obtained between corresponding fNIRS signal and uncorrelated physiological noise as described in section Uncorrelated Physiological Noise and Wavelet Coherence. Therefore, all values within gray area under the black curve can be considered as not significant.

There are three distinct peaks of the wavelet coherence between fNIRS and physiological signals within three different frequency bands. The peak with the lowest scale around 3 s (corresponding to the pseudo-frequency of 0.33 Hz in R-band) is most clearly observed for the respiration signal, but also present in the wavelet coherence with MAP, HR, and SBF.

The peak with scale in the range of about 10 s (corresponding to the pseudo-frequency of 0.1 Hz in M-band) is strongly present in both global signals (MAP and HR) and is also detectable for SBF. In comparison to HR and SBF, MAP demonstrates higher correlation and a broader peak in the M-band, ranging up to scale values of 20–25 s.

The third and broadest peak at 34 s (corresponding pseudofrequency 0.029 Hz in A-band) is only detectable for wavelet coherence between *m*0-based HbO and SBF.

Wavelet coherence values obtained between all physiological signals and HbO based on different moments are clearly different, with *m*0-based signals exhibiting the highest correlation at all scales, followed by *m*1-based and then by *V*-based signals. In contrast, wavelet coherence values obtained for HbR, based on all three moments, are not significantly different from each other. Generally higher wavelet coherence values were observed for HbO in comparison to HbR for *m*0- and *m*1- based signals. Interestingly, the *V*-based HbO and HbR wavelet coherence values are similar at all scales, while the *m*0- and *m*1- based HbO coherence is always higher than that of HbR for all signals (see **Figure 7**).

The lower wavelet coherence obtained for fNIRS signals based on higher moments may be rationalized by two different explanations. The higher coherence of *m*0-based signals may originate from larger contributions of physiological fluctuations in the extra-cerebral skin tissue in these signals. On the other hand, the lower coherence in *m*1- and *V*-based signals may be due to higher photon noise levels as compared to *m*0-based signals. The estimation of latter effect is presented in **Figure 8**. It illustrates a hypothetical case assuming that the difference between signals based on different moments is only attributed to the different photon noise level. The blue solid lines show the experimentally obtained wavelet coherence for the *m*0-based signal. The dashed lines represent the same signal, but numerically matched to the level of photon noise present in *m*1- and *V*-based signals. A slight influence of photon noise is clearly visible for low scales in the Rband. In contrast, the impact of photon noise may be neglected for scales higher than 5 s (see **Figure 8**).

In the following we describe the time lags between physiological signals and fNIRS signals in the three main frequency bands exhibiting high impact of physiological noise. These results are presented for the R-, M-, and A-bands in **Tables 1**–**3**, respectively.

*m*0-based HbO and HbR, MAP, HR, SBF, and respiration signals. All signals are arbitrary scaled. Gray areas correspond to the durations of the stimulation blocks; **(B)** Time traces of regressors used in de-noising GLM analysis. From

The time delay between the respiration signal and HbO was around 1 s, with no significant difference between delays obtained for HbO based on different moments. The time delays for HbR based on the three moments were not significantly different from zero (see **Table 1**).

The delay between HbO and MAP was around −0.6 s and was significantly shorter for the *m*1- and *V*-based signals in comparison with the *m*0-based signal (see **Table 2**). The difference in time delay between *m*<sup>0</sup> and *V* was (0.27 ± 0.1) s. The time delay between HbR and MAP was around 3.3 s and was not significantly different for signals based on different moments.

The time delays between HbO and SBF in the A-band are presented in **Table 3**. A significant delay of −6.0 s was observed for the *m*0-based HbO signal.

The results of group wavelet coherence between the four physiological signals are presented in **Figure 9**. A maximum in wavelet coherence in R-band can be seen for all pairs of physiological signals, with highest coherence in this band observed between MAP and RSP and between HR and RSP. Very high coherence (mean value 0.85) was observed between MAP and HR signals regressors obtained by appropriate filtering and time shifting of skin blood flow and respiration traces, respectively. The arrows between **(A,B)** parts indicate which signals were used to generate the corresponding regressor.

in the M-band. Maxima of lower amplitude in the M-band were observed for coherence between MAP and SBF as well as between HR and SBF. No considerable coherence was observed between any pair of physiological signals in the A-band. **Table 4** shows the group average time delays between physiological signals in R- and M-bands.

#### **PHYSIOLOGICAL DE-NOISING**

Based on the qualitative analysis of group wavelet coherence presented in **Figure 7**, we identified the most relevant frequency bands and most relevant sources of physiological noise in each band. We then constructed auxiliary physiological regressors modeling impact of each process on fNIRS signals. Taking into account the mutual correlation of physiological signals that is obvious from the results presented in **Figure 9**, only a single physiological signal, the one showing the highest coherence, was used in each band for the de-noising procedure. In this way we avoid redundancy in our model. The RSP signals was used in order to model physiological noise in the R-band, the MAP signal was used in order to model the physiological noise component in the Mband. The physiological noise modeling in the A-band was based on the SBF signal.

In the following we summarize filter functions and time shifts employed for each physiological auxiliary regressor. This parameter choice was based on maxima in the group wavelet coherence (see **Figure 7**) and group mean time shifts for the corresponding bands (see **Tables 1**–**3**). To model respiration-induced noise, RSP signals were filtered with a bandpass filter [bandwidth (bw) 0.2 to 0.5 Hz] and time shifted by 1 and 0 s for HbO and HbR, respectively.

The impact of physiological noise at M-band frequencies was taken into account by bandpass filtering of MAP signals (bw 0.15–0.08 Hz) and a time shift of −0.69 s for HbO and 3.4 s for HbR signals.

To account for physiological noise in the A-band, SBF signals were band pass filtered (bw 0.02–0.04 Hz) and shifted in time by −7 s for both, HbO and HbR signals.

Time shifted and band-pass filtered physiological signals of each subject were normalized to unit power and used to create an

M-bands for this subject.

**FIGURE 7 | Magnitude values of group average of wavelet coherence between haemoglobin concentration changes** *-***HbO (left) and** *-***HbR (right) and the four physiological signals.** In the columns from top to bottom results for mean arterial blood pressure, heart rate, skin blood flow, and respiration are shown, respectively. Haemoglobin concentration changes extracted from m0, m1, and V are plotted in blue, green, and red, respectively. The gray area under the black curve indicates the noise level for the magnitude of the group mean value of the WCOH between HbO or HbR and the physiological signals. Light blue, light, red and light green bars indicate the following three time scale ranges: R-band (scales around 3 s and pseudofrequencies around 0.3 Hz), M-band (scales around 10 s and pseudo-frequencies around 0.1 Hz) and A-band (scales around 35 s and pseudo-frequencies around 0.033 Hz), respectively. The colored shadowed areas represent standard error of mean for each curve.

**FIGURE 8 | Influence of photon noise on wavelet coherence.** The wavelet coherence between HbO **(left)** or HbR **(right)** signals based on different moments and four physiological signals. In the rows from **top** to **bottom** results for MAP, HR, SBF, and RSP are shown, respectively. Haemoglobin concentration changes extracted from *m*0, are plotted with a blue line. Green and red dashed lines correspond to experimental *m*0–based signals matched in the level of photon noise to *m*<sup>1</sup> and *V* based signals by adding synthetic noise. R-, M-, and A-bands are marked as in **Figure 4**. Slight effects of matching photon noise is observed in R-band only.

individual set of four auxiliary physiological regressors for each subject. **Figure 5B** shows the time traces of four GLM regressors for one representative subject. GLM including one functional (cerebral) and four physiological regressors was performed for the four detector channels and two haemoglobin concentrations for each of the 14 subjects. In order to quantify the impact of physiological de-noising, we additionally performed GLM modeling with a reduced model, including the cerebral regressor only.

#### **Table 1 | Mean group time lag between fNIRS signals and the respiration signal in the R-band.**


*Asterisks indicate values significant on the group level analysis, and the T-values are listed.*

**Table 2 | Mean group time lag between fNIRS signals and physiological signals in M-band.**


*Asterisks indicate values significant on the group level analysis, and the T-values are listed. The arrows indicate the significant difference between parameters.*

**Table 3 | Mean group time lag between fNIRS signals and SBF signal in A-Band.**


*Asterisks indicate values significant on the group level analysis, and the T-values are listed.*

The results of subject level analysis were subjected to a group *T-*test to determine the significance of activation at the group level.

The results of the second level analysis are presented in **Table 5**, and in **Figures 10**, **11** for HbO and HbR, respectively. In **Figures 10A**, **11A** HbO and HbR attributed to cerebral activation are shown. Significant positive HbO and significant negative HbR concentration changes were observed in channel 4 for *m*0-, *m*1- as well as for *V*-based signals. For the *V*- based HbR signal significant negative changes were also observed in channel 3.

Subpanels B–D in **Figures 10**, **11** show significant HbO and HbR values attributed to the three components of the physiological noise related to M-wave, SBF and respiration. One can see that the impact of physiological noise is generally much stronger for HbO than for HbR. Noise related to M-wave and respiration is present in HbO based on all three moments, while essentially only *m*0-based HbR is affected. The physiological noise related to the SBF changes is present only in *m*<sup>0</sup> and *m*1 based signals and is stronger pronounced in the two medial channels. In the *V*-based HbR no significant contribution of physiological noise was detected.

The results of the reduced GLM including only one cerebral regressor are presented in **Table 6**. No significant cerebral activation was observed with this reduced model for *m*0-based HbO. Significant activation was identified for channel 4 only, for the *m*1- and *V*-based HbO signals. In HbR signals significant activation was observed in channel 4 for the signals based on all three moments, however, the obtained *T*-values were always lower than that observed with the full model (compare **Table 5**, **6**).

#### **DISCUSSION**

The wavelet coherences of fNIRS signals and physiological processes, presented in **Figures 7**, **9**, reveal a strong impact of the physiological noise at three main frequency bands: R-band with scales around 3 s (0.3 Hz), M-band with scales around 10 s (0.1 Hz) and A-band with scales around 34 s (0.034 Hz). These three distinct bands indicate the presence of at least three distinct physiological mechanisms dominating physiological fNIRS noise. In the following we summarize the results obtained at each band and discuss possible underlying physiological mechanisms.

#### **R-BAND (0.2 TO 0.5 HZ)**

The R-band correlation peak is most obvious for the wavelet coherence between *m*0-based HbO signal and the respiratory signal (see **Figures 7**, **10D**). Its peak frequency (0.3 Hz) corresponds to the mean respiration rate, thus indicating the direct influence of the respiration on the fNIRS signal. Clear coherence at the R-band was also found for *m*0-based MAP, HR and SBF signals for both HbO and HbR concentration changes. However, the peak is not always clearly detectable, due to an overlay from neighboring M-band maxima in these signals.

Coherence at R-band demonstrates a clear difference between signals based on different moments. Highest coherence was primarily observed for *m*0-based signals. Significantly lower values were observed for *m*1-based HbO and HbR signals. The coherence in the R-band for *V*-based signal does not exceed the noise level. However, this dependence on the moment, which at first glance seems to be related to different physiological noise levels at different depths, has to be interpreted with care.

The analysis of the photon noise influence presented in **Figure 8** indicates that the wavelet coherence at small scale values is sensitive to photon noise. Taking into account an increasing photon noise contribution in *m*1- and *V*-based fNIRS signals, the observed dependence might be partly due to an artifact induced by different photon noise levels in the three measurements based on different moments.

In literature two different physiological mechanisms leading to R-band fluctuations are described. First, there is a direct influence of the respiration on the venous blood flow. Negative intrathoracic pressure during inspiration leads to increased venous outflow, which modulates the venous blood volume with the respiration frequency. The second mechanism is the respiratory modulation of the HR, mediated by the parasympathetic nervous system. Consequently the arterial input is modulated via HR by the respiratory cycle, a phenomenon usually referred to as respiratory sinus arrhythmia (Hirsch and Bishop, 1981). High correlation between respiration and HR observed in our study (see **Figure 9**) also indicates the presence of this phenomenon.

**FIGURE 9 | Magnitude values of group average of wavelet coherence between four physiological signals.** Left column: in the three rows from top to bottom results for wavelet coherence between MAP and HR, SBF, and RSP are shown, respectively. Right column: in the three rows from top to bottom results for wavelet coherence between HR and SBF, HR and RSP, and SBF and RSP are shown. Light blue, light red, and

**Table 4 | Mean group time lag between physiological signals.**


Based on our data it is difficult to conclude, which of the two mechanisms dominates fNIRS physiological noise—direct influence of the respiratory pump, or the indirect influence of the HR variability. It is possible that the impact of both mechanisms is different for the two haemoglobin concentrations.

#### **M-BAND (0.05 TO 0.15 Hz)**

The highest wavelet coherence between fNIRS signals and MAP and HR traces was observed in the M-band. This coherence is induced by Mayer waves, correlated fluctuation between HR and MAP in this frequency band (Elstad et al., 2011). As one can see in **Figure 7**, Mayer waves are present in both HbO and HbR concentration changes. However, both haemoglobin concentrations demonstrate different amplitudes, different time shifts and different dependence on moment, indicating different depth localization in the tissue. As one can see from the results of the WCA presented in **Figure 7** and from GLM results in **Figure 10**, *m*0-based HbO signals show the highest M-band contribution. The amplitude of M-band physiological noise is higher than that of the cerebral signal. HbO signals based on *m*<sup>1</sup> and *V* show significantly lower Mayer wave contributions. Since the influence of photon noise is negligible in the M-band, light green bars indicate the following three time scale ranges: R-band (scales around 3 s and pseudo-frequencies around 0.3 Hz), M-band (scales around 10 s and pseudo-frequencies around 0.1 Hz) and A-band (scales around 35 s and pseudo-frequencies around 0.033 Hz), respectively. The gray shadowed areas around the curves represent standard error of mean for each curve.

this moment dependence can be clearly assigned to Mayer waves with different amplitudes at different depths. A large fraction of the M-band contribution in the HbO signal originates from superficial extra-cerebral tissue, however, there is an additional cerebral contribution as well. The presence of two separate Mband components (extra- and intracranial) is further supported by the phase difference detected between *m*0-, *m*1-, and *V*-based signals and reported in **Table 2**. Indeed, if the Mayer waves appear both in the brain and in the scalp, one might expect a time delay between these two compartments, caused by different vascular path lengths as well as possible delays in sympathetic mediating signals between the two compartments (Tong et al., 2011). Although cerebral auto-regulation mostly acts at lower frequencies (Latka et al., 2005; Rowley et al., 2007), there might be a time shift due to this vascular property of brain vessels as well. Since *m*0-, *m*1-, and *V*-based signals reflect a linear combination of signals from skin and cerebral compartments with different weights, they would then show different time lags relative to MAP.

Interestingly, wavelet coherence between HbR and MAP and between HbR and HR is very similar for *m*0, *m*1, and *V*-based signals. In addition, no significant time lag was obtained for these signals. This means that the main source of M-band physiological noise in HbR is localized in deeper intracranial tissue. However, GLM analysis failed to detect significant HbR M-band contribution in *V*-based signals (see **Figure 11**), although significant M-band signals were obtained in *m*<sup>1</sup> and *V*-based HbO signals. Therefore, we can conclude that both extra- and intracranial M-band signals are mostly present on arterial side. This fact might explain why Mayer waves are not considered as important source of physiological noise in fMRI (Birn et al., 2006; Chang and Glover, 2009). Since BOLD signal exploited in fMRI mostly reflects changes in HbR concentration, it seems to be not strongly affected by Mayer waves. Moreover, the cerebral compartment might exhibit an additional contribution in the M-wave band which is not coupled with blood pressure fluctuations as


#### **Table 5a | The results of second level GLM analysis of the** *m***0-,** *m***1- and** *V* **-based** *-***HbO signals.**

*Only values significant on the group level analysis are shown, and the T-values are listed.*

#### **Table 5b | The results of second level GLM analysis of the** *m***0-,** *m***1, and** *V* **-based** *-***HbR signals.**


*Only values significant on the group level analysis are shown, and the T-values are listed.*

demonstrated recently by (Rayshubskiy et al., 2013). The presence of an additional component that is uncorrelated with MAP could also explain the experimentally observed lower wavelet coherence in the deeper tissue.

Another interesting fact is the high coherence between fNIRS signals and HR time traces in the M-band. Although the direct correlation between HR and signals is low, it accounts for up to 40% fNIRS signal variance for conventional cw (in our case *m*0-based) signals, for HR being shifted in time. Despite this number is lower than that obtained for MAP, it is still enough to significantly improve fNIRS sensitivity when used in a denoising procedure. This may become a fact of high importance in fNIRS experimental practice. Indeed, continuous blood pressure measurements are challenging and dedicated hardware more rare and cost intensive than conventional ECG or pulse plethysmography devices. Those are relatively cheap and available in many labs. Thus, physiological de-noising based on HR measurements is much more easily applicable than approaches based on continuous MAP recordings, although it might be slightly less efficient.

High correlation between MAP, HR, and fNIRS signals was reported and emphasized by several studies (Franceschini et al., 2006; Katura et al., 2006; Minati et al., 2011; Li et al., 2013). The time shift between fNIRS signals and MAP and HbO signals reported by (Katura et al., 2006) are very close to those presented in the **Table 2**.

Finally we would like to discuss the possible physiological mechanism inducing Mayer waves in fNIRS signals. Synchronized oscillations with frequencies around 0.1 Hz are typically observed in blood pressure and HR in humans and animals and are most likely induced by an interplay between sympathetically driven

**FIGURE 10 | fNIRS GLM group analysis of** *-***HbO signals.** The 12 gray shaded areas indicate the forehead region. Circles indicate the positions of the four fNIRS channels. The color of the circles represents the mean of significant HbO concentration changes related to one of the four regressors in the GLM model. An empty circle indicates non-significance of the

corresponding parameter. From left to right the four columns represent: HbO concentration changes related to cerebral activation and to physiological noise: **(A)** cerebral activation, **(B)** Mayer waves, **(C)** skin blood flow, **(D)** respiration. The three rows (from top to bottom) correspond to: HbO based on *m*0, *m*1, and *V*.

HR variations and sympathetic vasoconstriction of peripheral resistive vessels (Pagani et al., 1986; Malliani et al., 1991; Stauss et al., 1998; Cohen and Taylor, 2002; Nilsson and Aalkjaer, 2003). The high coherence between MAP and HR in M-band observed in our experiment (see **Figure 9**) is in agreement with these findings.

A mechanism that directly links local blood pressure and local HbO and HbR concentration changes is vasoconstriction of peripheral resistance vessels. These vessels are situated prior to the capillary bed on the arterial side and therefore reveal high concentration of oxygenated haemoglobin. The diameter of resistive vessels is sympathetically regulated, and critically influences the overall hydrodynamic resistance of the vascular system and thereby the blood pressure. Regulatory changes in the vessel diameter are connected to changes of the arterial blood volume, which is the most probable source of the observed tissue haemoglobin concentration changes. These changes occur on the arterial side are further propagated to the venous side due to induced fluctuations in blood flow (Tong et al., 2011).

#### **A-BAND (0.02 TO 0.04 Hz)**

As one can see in **Figure 7** the WCA reveals synchronous oscillations in the very low frequency band between the *m*0*-*based HbO signal and SBF.

With the help of GLM analysis presented in **Figures 10**, **11** we detect a significant contribution of this signal in all four channels in *m*0-based HbO, and in the two medial channels in *m*1-based HbO and *m*0-based HbR and in one lateral channel in *m*1 based HbR. Since *m*<sup>0</sup> is much more sensitive to superficial tissue then *m*<sup>1</sup> and *V,* we can conclude that activation in the A-band is predominantly localized in the skin compartment, and is due to the local SBF regulation mechanisms.

The stronger influence of the SBF related artifact on the medial forehead is supported by our earlier findings from a comparative fMRI/ fNIRS study (Kirilina et al., 2012). In this study we observed a task-evoked response in the two medial veins draining the forehead.

The scale of maximum coherence corresponds to the period of stimulation (34 s) used in our study. Therefore, we hypothesize,



that observed SBF responses might be induced by task-induced cognitive stress.

VLFOs with frequencies from 0.021 to 0.052 Hz were previously reported in LDF and NIRS measurements of the skin (Li et al., 2010) and were linked to sympathetic control of the peripheral vasculature (Kastrup et al., 1989; Söderström et al., 2003). Since increased level of sympathetic activity might be induced by cognitive or emotional tasks used in fNIRS experiments, SBF oscillation in this band might be synchronized with the task. Therefore, special caution has to be taken in fNIRS signal analysis in order to separate these superficial skin signals from cerebral activation.

#### **PHYSIOLOGICAL DE-NOISING OF fNIRS DATA**

The results of GLM-modeling presented in **Figures 10A**, **11A** demonstrate significant activation in the lateral left channel. An increase in HbO and decrease in HbR concentration was observed in this area when physiological de-noising was applied. This result is consistent with the results of an fMRI study performed using the same task and the same subject group (Kirilina et al., 2012). In this study activation in bilateral Brodman Area 10 was observed, lateralized on the right side (Talairach coordinates [32 48 11]). The left BA10 activation was too deep from the cortical surface to be detected by fNIRS.

Although HbO demonstrates the signals of higher absolute value than that of HbR, it is also much more strongly affected by physiological noise (compare **Figures 10B–D**, **11B–D**). Moreover, for the *m*0-based signal we failed to detect cerebral activation in HbO when the reduced GLM model (**Table 6**) was used. This means that we would not detect any cerebral activation with conventional cw fNIRS, if not corrected for physiological noise. In contrast, HbR signals show robust activation in all three moments even without physiological noise correction (see **Figure 11**).

The above described results demonstrate that HbR signals are more reliable in detecting cerebral activation, and that the denoising procedure developed in the current study can significantly improve the sensitivity of fNIRS to cerebral activation for HbO.

#### **LIMITATIONS OF THE CURRENT STUDY AND OUTLOOK**

One limitation of the present approach is that not all possible contributions to fNIRS physiological noise might be detected by our approach, but only those, which also manifest themselves in MAP, HR, and SBF. In particular, Lased Doppler Flowmetry measures only capillary SBF and not that of the large vessels. We also did not account for the possible impact of fluctuations in arterial CO2 concentration (Scholkmann et al., 2013a,b). Thus, some important sources of signal variance might be missing in our consideration.

Moreover the wavelet analysis is a linear transformation method, thus with the present approach we might miss more complex interrelations between physiological parameters and fNIRS signals, which are not captured by a simple linear relationship. In our WCA we used single physiological trace at a time. In the future the multi-variable coherence and correlation analysis such as the canonical correlation analysis could provide more complete picture (Caicedo et al., 2013).

Another limitation of the proposed de-noising strategy is the necessity to use additional physiological sensors and monitors. In particular, continuous monitoring of the blood pressure as well as laser flowmetry are not widely available in most labs. However, we have shown that even applying widely available and cost efficient sensors for respiration and HR might significantly improve fNIRS data quality.

An important interesting step in the further investigation might be to compare and combine the presented method of physiological de-noising with methods based on anatomical localization of cerebral and extra-cerebral tissue such as superficial signal regression.

#### **CONCLUSION**

In the current study we investigated the impact of global systemic and local regulatory physiological processes on physiological noise in fNIRS measurements and developed a method for physiological de-noising of fNIRS data. Global systemic processes were quantified by measuring MAP, HR, and respiration. Local regulatory processes in the skin were measured by means of SBF recordings. WCA was employed to characterize the contribution of these physiological parameters on fNIRS noise, at different time scales. Time-domain measurements in combination with signals, based on different moments of DTOF, enabled us to obtain information on the depth localization of different physiological noise sources.

With the help of these methods we were able to identify three main mechanisms contributing to physiological noise in fNIRS signals at three time scales. Two of these processes were induced by global systemic physiology: Mayer waves and respiration. The third slow process was induced by local blood flow changes in the skin tissue. We show that HbO signals are more strongly affected by global processes in both, extra- and intra-cerebral compartments, and local SBF regulation, while HbR signals are less contaminated by extra-cerebral processes.

By means of GLM analysis and auxiliary physiological regressors we quantified the relative impact of each process. Moreover, we propose a de-noising algorithm and demonstrate its performance on a functional experiment on the forehead. The proposed method was shown to significantly improve the sensitivity of fNIRS to cerebral activation.

#### **ACKNOWLEDGMENTS**

The research leading to these results has received funding from the European Community's Seventh Framework Programme [FP7/2007-2013] under grant agreement n FP7-HEALTH-F5- 2008-201076, and funding from German Research Organization (DFG) in the framework of Cluster of Excellence "Languages of Emotion" EXC302. IT and NY were supported by the Wellcome Trust (088429/Z/09/Z). The authors are thankful to Dr. Angela Heine for development of the task used in this study.

#### **REFERENCES**


*Sitzungsberichte Akad. Wiss. Wien Math.-Naturwissenschaftliche Cl. Anat*. 74, 281–307.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 29 October 2013; paper pending published: 08 November 2013; accepted: 26 November 2013; published online: 17 December 2013.*

*Citation: Kirilina E, Yu N, Jelzow A, Wabnitz H, Jacobs AM and Tachtsidis I (2013) Identifying and quantifying main components of physiological noise in functional near infrared spectroscopy on the prefrontal cortex. Front. Hum. Neurosci. 7:864. doi: 10.3389/fnhum.2013.00864*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Kirilina, Yu, Jelzow, Wabnitz, Jacobs and Tachtsidis. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

#### **APPENDIX**

The concentration changes of HbO and HbR were determined based on three moments of DTOFs (*m*0, *m*1, and *V*) separately. The time traces of HbO and HbR related to the different moments exhibit different levels of non-physiological noise, due to the different influence of instrumental noise. In order to estimate the amplitude of this non-physiological noise in HbO and HbR time traces, in the first step we estimated the instrumental and photon noise induced standard deviation of the corresponding moments. In the second step, we investigated how this uncertainty is propagated through the calculation of HbO and HbR.

**Step 1.** It is assumed that the uncertainty of measurement is dominated by photon noise following a Poisson distribution. For this case the variances of the changes in moments are determined by (Liebert et al., 2003, 2012):

$$
\sigma^2 \left(\Delta M\_0 \left(\lambda\_i\right)\right) \approx \sigma^2 \left(\Delta m\_0\right) / m\_0^2 \approx 1/m\_0 \left(\lambda\_i\right) \tag{A1}
$$

$$
\sigma^2 \left( \Delta M\_1 \left( \lambda\_i \right) \right) \approx M\_{2, \text{raw}} \left( \lambda\_i \right) / m\_0 \left( \lambda\_i \right) \tag{A2}
$$

$$
\sigma^2 \left( \Delta M\_2 \left( \lambda\_i \right) \right) \approx \left( M\_{4,\text{ raw}} \left( \lambda\_i \right) - M\_{2,\text{ raw}}^2 \left( \lambda\_i \right) \right) / m\_0 \left( \lambda\_i \right) \text{(A3)}
$$

where *M*<sup>0</sup> = −ln(1 − *m*0/*m*0) is the attenuation change, *M*<sup>1</sup> = *m*1, *M*<sup>2</sup> = *V*. *M*2,raw = *V*raw and *M*4, raw are the variance (second central moment) and the fourth central moment of the DTOF, respectively, as directly obtained in the experiment, without eliminating the influence of the instrumental response function. *m*0, *M*2,raw and *M*4, raw were estimated from the experimentally measured DTOFs for each subject and each wavelength. Based on the group-averaged values of these parameters we estimated the standard deviations of each moment according to Equations (A1)–(A3) (see **Table A1**).

**Step 2.** In order to obtain the concentration changes of oxyand deoxyhaemoglobin, *HbOn* and *HbRn*, based on the *n*th moment we need to solve the over-determined set of linear equations:

$$
\begin{pmatrix}
\Delta M\_{\text{\tiny{\tiny{\text{\tiny{\text{\tiny{\text{\tiny{\text{\tiny{\text{\tiny{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\text{\cdot}}\cdot}}}}}}}}}}}}
}}
}}
}}
}}
}}
}}
}}
}}
}}
}}
}}
}}
}}
}}
}}
$$

where *Mn*(λ*i*) represents the changes in the *n*-th moment (*n* = 0, 1, and 2) (definitions see above) for the *i*-th wavelength. *Sn*(λ*i*) is a moment-specific sensitivity factor, HbO(λ*i*) and HbR(λ*i*) are the molar absorption coefficients of HbO and HbR, The sensitivity factors (for the homogeneous case) can be derived from moments of the measured DTOF by (Arridge et al., 1992; Steinbrink, 2000; Liebert et al., 2004):

**Table A1 | Group-averaged values of sensitivity factors** *Sn* **and of standard deviations due to photon noise σ -** *-Mn* **of the changes in moments for all three wavelengths.**


$$S\_0\left(\lambda\_i\right) = c\_m M\_1\left(\lambda\_i\right) \tag{A5}$$

$$\mathcal{S}\_1\left(\lambda\_i\right) = -\mathfrak{c}\_m M\_2\left(\lambda\_i\right) \tag{A6}$$

$$S\_2\left(\lambda\_i\right) = -c\_m M\_3\left(\lambda\_i\right) \tag{A7}$$

where *M*<sup>1</sup> = *m*1, *M*<sup>2</sup> = *V*, *M*3—third central moment of the DTOF and *cm*—speed of light in the medium. Note that *S*<sup>0</sup> is related to the differential pathlength factor (*DPF*) by *S*<sup>0</sup> = *DPFrsd* with *rsd* being the source-detector separation.

Equation (A4) can be rewritten as −−→*Mn* <sup>=</sup> *An* −−→*Cn* where *Cn* = HbO*<sup>n</sup>* HbR*<sup>n</sup>* is the vector of concentration changes, −−→*Mn* the vector of changes of the *n*-th moment for the three wavelengths and *An* the coefficient matrix on the right-hand side of Equation (A4). The solution of this system of equations in a

least-squares sense is given by:

$$
\overrightarrow{\Delta C\_n} = \left(A\_n^T A\_n\right)^{-1} A\_n^T \overrightarrow{\Delta M\_n} \tag{A8}
$$

The variances (i.e., squares of standard deviations) of the concentration changes HbO*<sup>n</sup>* and HbR*<sup>n</sup>* are the diagonal elements of the covariance matrix:

$$\text{cov}\left(\overrightarrow{\Delta C\_n}, \overrightarrow{\Delta C\_n}\right) = \left(A\_n^T Z\_n^{-1} A\_n\right)^{-1} \tag{A9}$$

Herein *Zn* is the covariance matrix of the moment changes −−→*Mn*. Since photon noise is uncorrelated for the measurements at different wavelengths, *Zn* is a diagonal matrix:

$$\begin{aligned} Z\_n &= \text{cov}\left(\overrightarrow{\Delta M}\_n, \overrightarrow{\Delta M}\_n\right) \\ &= \begin{bmatrix} \sigma^2 \left(\Delta M\_n \left(\lambda\_1\right)\right) & 0 & 0 \\ 0 & \sigma^2 \left(\Delta M\_n \left(\lambda\_2\right)\right) & 0 \\ 0 & 0 & \sigma^2 \left(\Delta M\_n \left(\lambda\_3\right)\right) \end{bmatrix} \text{(A10)} \end{aligned} $$

with σ(*Mn*(λ*i*)) being the standard deviation of the *n*-th moment at wavelength λ*i*.

For the estimation of the standard deviations of the concentration changes we used the group-averaged values of the sensitivity factors together with the group-averaged standard deviations of moments obtained in Step 1, both listed in **Table A1**.

## Prefrontal cortex and executive function in young children: a review of NIRS studies

### *Yusuke Moriguchi 1,2\* and Kazuo Hiraki 3,4*

*<sup>1</sup> Department of School Education, Joetsu University of Education, Joetsu, Japan*

*<sup>2</sup> Japan Science and Technology Agency, PRESTO, Tokyo, Japan*

*<sup>3</sup> Department of Systems Science, University of Tokyo, Tokyo, Japan*

*<sup>4</sup> Japan Science and Technology Agency, CRESTO, Tokyo, Japan*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Masatoshi Katagiri, University of Toyama, Japan Toru Yamada, National Institute of Advance Industrial Science and Technology, Japan*

#### *\*Correspondence:*

*Yusuke Moriguchi, Department of School Education, Joestsu University of Education, 1 Yamayashiki-machi, Joetsu 943-8512, Japan e-mail: moriguchi@juen.ac.jp*

Executive function (EF) refers to the higher-order cognitive control process for the attainment of a specific goal. There are several subcomponents of EF, such as inhibition, cognitive shifting, and working memory. Extensive neuroimaging research in adults has revealed that the lateral prefrontal cortex plays an important role in EF. Developmental studies have reported behavioral evidence showing that EF changes significantly during preschool years. However, the neural mechanism of EF in young children is still unclear. This article reviews recent near-infrared spectroscopy (NIRS) research that examined the relationship between the development of EF and the lateral prefrontal cortex. Specifically, this review focuses on inhibitory control, cognitive shifting, and working memory in young children. Research has consistently shown significant prefrontal activation during tasks in typically developed children, but this activation may be abnormal in children with developmental disorders. Finally, methodological issues and future directions are discussed.

**Keywords: prefrontal cortex, executive function, young children, NIRS, developmental disorders**

Executive function (EF) refers to the higher-order cognitive control process for the attainment of a specific goal. EF plays an important role in multiple areas of child development such as social cognition, communicative behavior, and moral behavior (Kochanska et al., 1997; Carlson and Moses, 2001; Moriguchi et al., 2008, 2010a). It has been shown that EF has several subcomponents in adults and older children (Miyake et al., 2000; Lehto et al., 2003; Huizinga et al., 2006). Miyake et al. (2000) have shown that the three major components of EF, namely, inhibition, shifting, and updating (working memory), are separable even though they are moderately correlated. However, there are still controversies regarding the data for preschool-aged children. Theoretically, there might be three components of EF in young children (Garon et al., 2008), but, empirically, a single-factor model (general EF) has been sufficient to account for the data in preschool-aged children (Wiebe et al., 2008).

#### **BEHAVIORAL AND ANATOMICAL EVIDENCE**

Extensive research has shown that cognitive shifting rapidly develops during preschool years. One task widely used in such research is the Dimensional Change Card Sort (DCCS) task (Zelazo et al., 1996; Kirkham et al., 2003; Moriguchi et al., 2010b). In this task, children are asked to sort cards that have two dimensions, such as color and shape (e.g., red boats, blue rabbits). There are two phases to the task. During the preswitch phase, children are asked to sort cards according to one dimension (e.g., color) for several trials. During the postswitch phase, children are asked to sort the cards according to the other dimension (e.g., shape) for several trials. It has been repeatedly shown that most 3-year-olds correctly perform the preswitch phase, but show difficulty with the postswitch phase (Zelazo et al., 1996; Moriguchi et al., 2012). Four- and five-year-old children correctly sort the cards according to the second dimension. The DCCS is used to index cognitive shifting as well as EF in general (Garon et al., 2008).

Researchers have used the Stroop-like Day-Night task and the Black-White task to examine the development of inhibitory control in young children (Gerstadt et al., 1994; Simpson and Riggs, 2005; Moriguchi, 2012). In the Day-Night task, children are instructed to say "day" in response to a picture of a moon with some stars and "night" in response to a picture of a sun. In order to perform the task correctly, children have to inhibit the dominant response (e.g., children have to inhibit day responses when presented with a sun card). For this task, response accuracy and latency has been shown to improve between 3 and 5 years of age. Although it has been suggested that both inhibition skills and working memory are needed to pass the task, this task is often used as an index of inhibition skills (Carlson and Moses, 2001).

The development of working memory is measured by the selfordered searching task (Luciana and Nelson, 1998). In this task, several colored squares are presented on a computer screen and each square contains a token. The children's task is to touch and open the squares and find as many tokens as possible. Each square has only one token; therefore, children must keep the previously selected squares in mind and use this information to inform subsequent responses. Luciana and Nelson (1998) gave 4- to 8-yearold children the task and found that the performance improved during this period. Specifically, 4-year-old children performed worse in three- and four-item searches than older children. In the six-item search, 7- and 8-year-old children outperformed younger children. Previous research has consistently shown that children develop visuospatial working memory during their preschool years (Garon et al., 2008).

Although behavioral evidence is accumulating, the neural basis of EF in young children is still unknown. There is some anatomical evidence that the prefrontal cortex develops during preschool years. Recent structural magnetic resonance imaging (MRI) studies have shown changes in brain structure over time within an individual. Studies by Giedd and colleagues (Giedd et al., 1999; Gogtay et al., 2004) have indicated that gray matter in the prefrontal cortex shows inverted U-shape changes during childhood. The volume in the prefrontal regions increase with age until adolescence (Tanaka et al., 2012). Additionally, Giedd et al. (1999) have shown that white matter volume in the frontal area increases linearly during the ages of 4–20 years. This evidence suggests the likelihood of structural changes within the prefrontal cortex during preschool years.

#### **NEURAL BASIS OF COGNITIVE SHIFTING AND INHIBITORY CONTROL IN YOUNG CHILDREN**

Little neuroimaging data have demonstrated the functional development of the prefrontal cortex during preschool years. However, recently, studies using near-infrared spectroscopy (NIRS) have shown that prefrontal activation is developmentally correlated with EF in young children. With NIRS, it is possible to monitor cerebral hemodynamics by measuring changes in the attenuation of near-infrared light passing through the tissue. Because NIRS is noninvasive and does not require fixing of the body as in functional MRI (fMRI), it is often used for brain imaging studies in infants and children. There are three NIRS parameters: oxygenated hemoglobin level, deoxygenated hemoglobin level, and total hemoglobin level; however, this review mainly concentrate on oxygenated hemoglobin findings because most of the previous research on young children consistently reported the oxygenated hemoglobin as an index of brain activation. The change in oxygenated hemoglobin level is considered to be a good indicator of brain activity (Strangman et al., 2002; but see also Huppert et al., 2006).

The spatial and depth sensitivity to activations in a region in NIRS system is limited compared to other neuroimaging technique such as fMRI (Strangman et al., 2013). Nevertheless, some NIRS researches are based on the previous other neuroimaging research such as fMRI to decide region of interest. Brain imaging studies using fMRI have shown that adult participants recruit inferior and dorsolateral prefrontal regions during cognitive shifting tasks, such as the Wisconsin Card Sorting Test (WCST; Konishi et al., 1998; Monchi et al., 2001). In the WCST, participants are asked to sort cards depicting geometric features, such as shape, color, and number according to rules that are detected through feedback given by an experimenter. After the participants figure out the rule and sort the cards for several trials, the rule suddenly changes and participants must adjust to the rule change depending on the feedback. In this task, participants recruit prefrontal areas, as well as the parietal cortex, when they have to switch from one rule to another (Konishi et al., 1998; Monchi et al., 2001).

Recently, Moriguchi and Hiraki (2009) examined the neural basis of cognitive shifting in young children (**Figure 1**). In this cross-sectional study, 3-year-old children, 5-year-old children, and adults were asked to perform the DCCS task while their brain activation was examined with a multichannel NIRS system that covered the inferior prefrontal regions corresponding to F7/8 in the International 10/20 system. Brain activation during the preswitch and postswitch was separately analyzed, and compared to the activation during the control phases.

At the behavioral level, 5-year-old children and adult participants easily performed both the preswitch and postswitch phases (Moriguchi and Hiraki, 2009). Some 3-year-old children performed the DCCS tasks perfectly, but others committed perseverative errors during the postswitch phases. At the neural level, adults and 5-year-old children showed significant activation in the right and left inferior prefrontal areas during the preswitch and postswitch phases compared to the control phase. The researchers

**FIGURE 1 | Experimental settings**. **(A)** A child with NIRS probe. **(B)** The NIRS probe was attached to the inferior prefrontal area. Each channel consisted of one emitter optode and one detector optode. The region of interest was located near F7/8, which corresponds to ch 15, 17, 18 and 6, 7, 9, respectively. **(C)** An example of preswitch and postswitch phases in the DCCS task. Figure from Moriguchi and Hiraki (2009). Neural origin of cognitive shifting in young children. *Proceedings of the National Academy of Sciences of the United States of America*, 106, 6017–6021.

analyzed the 3-year-old children separately according to whether they committed perseverative errors during the tasks. In the children who performed perfectly (pass group), the right inferior prefrontal areas were significantly activated during the preswitch and postswitch phases. In contrast, children who perseverated (perseverate group) exhibited no significant activation in the inferior prefrontal areas during both the preswitch and postswitch phases.

The results suggest that the development of cognitive shifting was correlated with the activations in the prefrontal regions. However, it has been shown that the NIRS signal is the product of the optical path length and the hemoglobin changes, and the optical path length differs across participants and head positions (Zhao et al., 2002). Thus, the comparison or integration of data between different subjects may be difficult. Rather, research of withinsubject designs can be appropriate to compare across different conditions within the subjects. Thus, longitudinal method may be useful to address the age-related changes in the activations in specific brain regions.

Moriguchi and Hiraki (2011) longitudinally examined the development of prefrontal activation in children. Children were given the DCCS task, and developmental changes in prefrontal activation were examined at 3 (Time 1) and 4 years of age (Time 2). Behavioral results indicated that children in the perseverate group (i.e., the children who committed errors at Time 1) improved their performances significantly. Children in the pass group (i.e., the children who did not commit errors at Time 1) performed correctly at Time 2. Thus, there were no significant behavioral differences between the children in the pass and perseverate groups at Time 2.

At the neural level, children in the pass group showed significant right inferior prefrontal activation during the preswitch and postswitch phases at Time 1 and Time 2, and had significant activity in the left inferior prefrontal areas during the preswitch and postswitch phases at Time 2 compared to Time 1 (**Figure 2**). The activation pattern at Time 2 was similar to that found in 5 year-old children in the study by Moriguchi and Hiraki (2009), with the children exhibiting bilateral inferior prefrontal activation during the DCCS task. On the other hand, the results for the perseverate group showed a different pattern (**Figure 3**; Moriguchi and Hiraki, 2011). At Time 1, children in the perseverate group exhibited no significant activation in the inferior prefrontal areas during the preswitch and postswitch phases whereas at Time 2, they showed significant activation in the left (but not right) inferior prefrontal regions during both phases.

Results of two studies showed that sustained unilateral (either right or left) inferior prefrontal activation across the preswitch and postswitch phases may be important for successful performance in the DCCS task. Similar results were obtained in an event-related potential study of DCCS (Espinet et al., 2012). Furthermore, there might be individual differences in the development of prefrontal function during preschool ages (Moriguchi and Hiraki, 2011). Children in the pass group showed activation of the right prefrontal regions at Time 1 and then recruited bilateral inferior prefrontal regions at Time 2. Children in the perseverate group showed no significant activation in the prefrontal regions at Time 1, but recruited the left inferior prefrontal

regions at Time 2 when they passed the DCCS tasks. It should be noted that 3-year-old children in the pass group (who successfully performed the DCCS earlier) recruited the right inferior prefrontal areas, whereas children in the perseverate group (who successfully performed the task 1 year later than those in the pass group) recruited the left prefrontal regions. These results suggest that the right inferior prefrontal areas may be relatively dominant in DCCS tasks, while the left inferior prefrontal areas may support or compensate for right inferior prefrontal activations (Moriguchi and Hiraki, in press).

Few NIRS studies have been conducted on the development of inhibitory control in young children. Recently, Mehnert et al. (2013) gave 4- to 6-year-old children and adults a Go/NoGo task, where participants were asked to respond to targets by pressing a button (Go trials) and to avoid making a response to non-targets (NoGo trials). The researchers measured activity in several brain regions including the prefrontal, parietal, and temporal regions using NIRS. Behavioral results showed that adults performed the tasks more accurately and faster than children did. NIRS results showed that adults activated right frontal and parietal regions during NoGo trials compared to Go trials, whereas children's right frontal and parietal activation was high in both Go and NoGo trials. Moreover, functional connectivity analyses revealed a stronger partial coherence in short-range connectivity in the right frontal and right parietal cortices in children compared to adults. In contrast, adults showed long-range functional connectivity between bilateral frontal and parietal areas. Although the research

**at Time 1 (AB) and Time 2 (CD)**. The grand averaged data during the task phases are shown. The numbers (1–10) indicate the channel of the NIRS probe. **(A)** and **(C)** show the brain activations during the preswitch phases, and **(B)** and **(D)** show the brain activations during the postswitch phases. The red areas highlight areas where stronger activations were observed during the task, and the blue areas highlight areas where deactivations were observed during the task. Figure from Moriguchi and Hiraki (2011). Longitudinal development of prefrontal function during early childhood. *Developmental Cognitive Neuroscience*, 1, 153–162., with permission from Elsevier.

relied on concentration changes in deoxygenated hemoglobin, their results revealed that children activated the right frontal and parietal areas in the Go/NoGo task.

### **NEURAL BASIS OF WORKING MEMORY IN YOUNG CHILDREN**

It has been repeatedly shown that regions in the prefrontal cortex, such as the dorsolateral prefrontal cortex, play an important role in visuospatial working memory in older children and adults as well as in non-human primates (Goldman-Rakic, 1995; Braver et al., 1997; Casey et al., 2005). However, little is known about the neural basis of working memory in young children.

Using NIRS, Tsujimoto et al. (2004) reported that the neural basis of working memory in young children covers the lateral prefrontal regions corresponding to Brodmann areas 9/46. In this study, 5- and 6-year-old children and adults performed a visuospatial working memory task. In this task, participants had to keep the locations of a sample cue array in mind during a delay period, after which they were asked to report whether a test cue location was identical to any of the sample cue locations. In adult participants, two (LOW condition) or four (HIGH condition) location cues were given as sample cue arrays, whereas children were given only two location cues. The authors examined prefrontal activity after presentation of the sample cues.

At the behavioral level, the adults' performances in the HIGH condition were significantly worse than their performances in the LOW condition. Children's performances were worse than adults' performances both in the LOW and HIGH conditions. At the neural level, adult participants showed significant activation of the bilateral lateral prefrontal regions in the HIGH condition, whereas relatively weaker activation was observed in the LOW condition. These results suggest that memory load affected activity in the lateral prefrontal cortex. The brain regions and time course of activity in children were similar to those in adults, as children also exhibited sustained lateral prefrontal activation after onset of the sample cues. Tsujimoto et al. (2004) concluded that the lateral prefrontal cortex was activated in young children during a working memory process.

Tsujii et al. (2009) examined the longitudinal development of prefrontal function using the same visuospatial working memory task. Children participated in the study at 5 (Study 1) and 7 years of age (Study 2). A multichannel NIRS shell was placed on the prefrontal regions corresponding to Fp1/2 in the International 10/20 system. Behavioral results showed that children significantly improved their performance on the working memory task between Study 1 and Study 2. At the neural level, children exhibited bilateral prefrontal activation during the working memory task in both Study 1 and Study 2, although activity at age 7 (Study 2) was weaker than that at age 5 (Study 1). Importantly, in Study 2, the children exhibited laterality effects, with activation in the right prefrontal regions being stronger than that in the left prefrontal regions. Such laterality effects were not observed in Study 1. Tsujii et al.(2009) interpreted these findings as suggesting that visuospatial working memory induces right-lateralization, whereas verbal working memory induces left-lateralization.

Other researchers focused on the limits of working memory capacities (Buss et al., in press). It is well known that visual working memory can hold 3–4 items at any given moment (Vogel and Machizawa, 2004). These capacity limits are often indexed by the change detection task. The basic procedure is similar to that of the working memory tasks cited above. In this task, participants are shown a cue array and instructed to keep the array in mind during a delay phase. Then, participants are shown a test array where either all items are the same as the cue array, or some of the features are changed. Then, they are asked to report whether there were changes in the test array or not. Task difficulty depends on the number of items presented in the cue array. In research using children, the number of items is 1, 2, or 3.

Buss et al. (in press) gave 3- and 4-year-old children this task, and examined the neural activation after presentation of the cue array with an NIRS system that covered the prefrontal regions corresponding to F3–5/4–6 and the parietal regions corresponding to P3–5/4–6 in the International 10/20 system. Their results showed that children's behavioral performances were a function of the number of items presented in the cue array. They performed worse when three items were presented than when one item was presented. The age effects were also significant, showing that 4 year-old children overall outperformed 3-year-old children. At the neural level, children exhibited significant activation in the frontal and parietal regions after presentation of the cue array. In addition, the activation was affected by the number of items. That is, activity in the left frontal areas and bilateral parietal areas was significantly stronger when three items were presented than when one or two items were presented. The age differences were evident in the parietal cortex, since stronger activation in this area was observed in 4-year-old children compared to 3-year-old children. In the right frontal cortex, activity in 3-year-old children first increased and then decreased. No such activation pattern was found in 4-year-old children. Interpretation of these results was difficult, but given that the activation patterns in the right prefrontal cortex were significantly correlated with behavioral performance, the right frontal regions may play an important role in visuospatial working memory.

Overall, research has consistently shown that children activate the right prefrontal regions during visuospatial working memory tasks. The developmental changes, however, were less clear. Some studies showed stronger activation in older children whereas other research showed weaker activation in older children. The mixed results may be due to the variation in task demands across studies. Generally, greater demand may induce stronger neural activation in the prefrontal cortex. Thus, it is possible that children exhibited age-related improvement in prefrontal activation as long as the task demands were appropriate. But in tasks that were too easy for older children, the older children might have had weaker prefrontal activation compared to younger children.

#### **EF IN CHILDREN WITH AUTISM**

Neuroimaging research with NIRS might also contribute to our understanding of developmental disorders, such as autism spectrum disorder (ASD). This disorder is characterized by deficits in social interaction and communication, as well as restricted, repeated, and stereotyped interests and behaviors (American Psychiatric Association, 2000). There are several cognitive theories for explaining the deficits of ASD, but one theory that might be related to stereotyped behavior involves EF (Hill, 2004). Some previous studies have revealed that patients with ASD have difficulties with cognitive shifting during the WCST (Ozonoff et al., 1994), but other research did not support the results (Nydén et al., 1999). The mixed results may be due to that WCST task includes many complex cognitive processes, such as planning, cognitive shifting, response inhibition, and error detection. Moreover, few brain imaging studies have investigated young children with ASD. Thus, it was unclear whether children with ASD may have functional and anatomical deficits in the prefrontal cortex.

Recently, some studies using NIRS have indicated behavioral and neural differences in cognitive shifting between typically developing (TD) children and children with ASD (Yasumura et al., 2012). Seven- to 12-year-old ASD children and agedmatched TD children were asked to perform an advanced version of DCCS tasks (ADCCS), while neural activity in the prefrontal cortex including F7/F8 was examined with NIRS. In the ADCCS, children need to switch flexibly between two incompatible rules within the same set. Half of the test cards have a border around them, while the other half do not. Children are asked to sort the cards according to one rule if the card has a border and according to another rule if the card has no border. Children typically have more difficulty performing the ADCCS task than the standard DCCS task (Hongwanishkul et al., 2005; Moriguchi and Hiraki, in press). The behavioral results revealed that children with ASD performed the ADCCS significantly worse than TD children did. The NIRS results demonstrated significant differences in prefrontal activation between the groups. TD children exhibited significant bilateral prefrontal activation during the ADCCS. In contrast, children with ASD showed significant left prefrontal activation, but the right prefrontal regions were not significantly activated. A direct comparison between the groups revealed significant differences in right prefrontal regions.

Xiao et al. (2012) examined the neural basis of inhibitory control in children with autism and attention-deficit/hyperactivity disorder (ADHD) using NIRS. In this study, TD children, children with high functioning autism (HFA) and children with ADHD were given color-word Stroop and Go/NoGo tasks. The research examined the activations in the forehead. As the results, there were no significant behavioral differences and the prefrontal activations between groups during the Stroop task. On the other hand, in the Go/NoGo task, children with autism and ADHD made more commission error during the NoGo blocks than typically developed children, and children with autism and ADHD did not differ in the errors. Moreover, the NIRS results showed that children with ASD and children with ADHD exhibited weaker right prefrontal activations during the Nogo block than TD children. The different results in the Stroop and Go/NoGo tasks may be due to that different brain regions may be recruited in each task. Indeed, it has been shown that Go/NoGo tasks mainly activate the right prefrontal regions (Aron et al., 2004) whereas the Stoop tasks recruit anterior cingulate regions (Pardo et al., 1990). The results in Stroop tasks were discussed later.

Taken together, although the evidence is not enough, these results show that children with ASD may have some difficulty with cognitive shifting and inhibitory control at both the behavioral and neural level.

### **EF IN CHILDREN WITH ATTENTION-DEFICIT/HYPERACTIVITY DISORDER**

Executive dysfunction may be related to ADHD (Barkley, 1997). This developmental disorder is characterized by inattention, hyperactivity, and impulsivity (American Psychiatric Association, 2000). Recent neuroanatomical research suggests that children with ADHD exhibit a marked delay in maturation of the prefrontal areas (Shaw et al., 2007). Moreover, it has been shown that patients with ADHD exhibit weaker prefrontal activation in EF tasks (Pliszka et al., 2006). These studies indicate that patients with ADHD may have functional and anatomical deficits in the prefrontal cortex (Bush et al., 2005). However, few brain imaging studies have investigated young children with ADHD.

Yasumura et al. (in press) asked school-aged children with ADHD, children with ASD and TD children to perform coroword Stroop and reverse Stroop tasks. In the Stroop task, participants had to select a color word (e.g., red) from among four color words shown in each corner of the screen that matched the word displayed in the center of the screen. The central word was the name of a color displayed in an incongruent font color (e.g., the word "green" displayed in red font). Participants had to choose the word that matched the font color of the central word and inhibit the tendency to select the word that matched the meaning of the central word. In the reverse Stroop task, participants had to select a color from among four colored patches shown at each corner of the screen that matched the meaning of the central word. The font color was incongruent with the central word meaning and interfered with the choice of matching the word with the correct colored patch. Participants had to inhibit the tendency to select the color that matched the font color of the central word. It has been shown that the reverse Stroop task may induce participants' conflict although the conflict may be smaller than that in the standard Stroop task (Ruff et al., 2001).

Behavioral results revealed that there were no significant differences between the groups on the Stroop task. In the reverse Stroop task, children with ADHD committed more errors than those with ASD and TD children. During the tasks, NIRS was used to examine activity in the prefrontal regions corresponding to F7/8 and FpZ in the International 10/20 system. Consistent with the behavioral data, no significant differences between groups were found for the Stroop task. On the other hand, in the reverse Stroop task, children with ADHD exhibited significantly lower activation in the right prefrontal regions than TD children did. The results revealed that children with ADHD exhibited abnormal behavioral performance and neural activation in the right prefrontal regions. Interestingly, children with ASD did not show such abnormalities.

The results of Stroop tasks were mixed. As noted above, Xiao et al. (2012) reported that there were no significant behavioral differences and the prefrontal activations between TD children and children with ADHD during the Stroop task. On the other hand, Negoro et al. (2010) reported that TD children and children with ADHD differed in changes of the oxygenated hemoglobin in the bilateral prefrontal regions during the color-word Stroop task. The different results may be due to the differences in task design. Yasumura et al. (in press) asked children to choose the correct words. Xial et al. (2012) also asked children to manually respond to the stimuli. On the other hand, in Negoro et al.'s (2010) study, children verbally responded the words during the Stroop task. It has been shown that NIRS signals on forehead were strongly affected by skin blood flow in a verbal fluency task (Takahashi et al., 2011). Given that, it is possible that the activations in the prefrontal regions were strongly affected by skin blood flow in the verbal Stroop task. The signal changes in the verbal task were relatively large compared to the manual task, by which researchers may easily detect the differences between groups.

In another study, Tsujimoto et al. (2013) examined whether children with ADHD showed deficits in visuospatial working memory. The basic procedure of the task was the same as the procedure mentioned above for Tsujimoto et al. (2004). Briefly, participants had to keep the locations of a sample cue array in mind during a delay period. However, in this study, there were two conditions after the delay period. In the distractor condition, participants were given a distractor task, where three purple dots and three yellow dots appeared at random locations on a screen, and the participants had to touch only the yellow dots. In the no-distractor condition, there were no distractor tasks. After that, white dots appeared on the screen, and participants had to touch the positions where sample cues had been presented.

Using NIRS, activity in the prefrontal regions corresponding to F7/8 and FpZ in the International 10/20 system of school-aged ADHD children and TD children was examined while they performed the working memory task. Children with ADHD showed poorer performance on the working memory tasks than TD children. Specifically, children with ADHD made more errors in the distractor condition than in the no-distractor condition, but no such differences were found in TD children. NIRS results revealed significant differences in prefrontal activation between children with and without ADHD in the distractor condition. Specifically, stronger activation in the right and middle prefrontal regions was observed for children with ADHD than for those without ADHD. No such differences were found in the no-distractor condition. Moreover, a significant correlation between error rates and right prefrontal activation was found in children with ADHD. Thus, hyperactivity in the right prefrontal regions may play a role in error-making during working memory tasks (Tsujimoto et al., 2013).

The research shows that children with ADHD may have some difficulty with inhibitory control indexed by Go/NoGo and working memory at both the behavioral and neural level. With regards to the Stroop tasks, the results were mixed. It should be noted that the activation in the prefrontal regions was sometimes stronger and sometimes weaker in children with ADHD that in TD children. Tsujimoto et al. (2013) suggested that hyperactivity observed in ADHD children may be due to the compensation of the inefficient neural processing in the prefrontal regions. Thus, the children may recruit the prefrontal regions in the EF tasks, but the activations may not be efficient enough to perform the tasks successfully. On the other hand, the hypoactivations observed children with ADHD may be due to that they fail to recruit the prefrontal regions in the EF tasks. As young children generally failed to activate the prefrontal regions, children with ADHD may fail to activate the prefrontal regions. Nevertheless, there is little data regarding the issue. Future research should be done to address it.

#### **METHODOLOGICAL ISSUES**

We had to note that there were several methodological issues in NIRS research for young children. First, we reviewed previous research focusing on oxygenated hemoglobin because most of the previous research on young children consistently reported the oxygenated hemoglobin as an index of brain activation. It has been shown that BOLD signal showed significant correlations with the oxygenated hemoglobin measured by NIRS (Strangman et al., 2002). A recent study using a working memory task and a finger tapping task revealed that oxygenated (and deoxygenated) hemoglobin were significantly correlated with BOLD signals in the prefrontal and the sensorimotor regions (Sato et al., 2013). The results indicate that the change in oxygenated hemoglobin is good indicator of brain activity in EF studies.

However, there are reasons why researchers should report both oxygenated and deoxygenated hemoglobin changes. The NIRS research is partly based on evidence of other neuroimaging technique such as fMRI, and it is well known that BOLD signal in fMRI tightly correlates, and have common physiological origins, with the deoxygenated hemoglobin (Huppert et al., 2006). There were mixed evidence regarding whether either oxygenated or deoxygenated hemoglobin showed the better correlation with BOLD signals. Huppert et al. (2006) argued that the mixed results may be due to low signal-to-noise ratio in NIRS measurements, and showed that, with high signal-to-noise ratio in the measurements, deoxygenated hemoglobin was better correlated with BOLD signals than oxygenated hemoglobin in a short-duration motor task. Further, it has been shown that body motions can cause baseline fluctuation in the NIRS signal (Yamada et al., 2009). Importantly, oxygenated hemoglobin is more highly influenced by the motions than deoxygenated hemoglobin. Such fluctuations can be removed by task designs and separating signal components, but some of the research did not consider such removals. Given the evidence, we suggest that both oxygenated and deoxygenated hemoglobin changes should be analyzed and reported.

Second, the spatial and depth sensitivity in NIRS measurement should be considered. The sensitivities to the brain tissues are dependent on the placement of emitters and detectors. Here, we focused on the depth sensitivity. NIRS system examines brain activation in the upper areas of the cerebral cortex. This is because of the fact that the near-infrared light is predominantly absorbed by the brain tissue hemoglobin at approximately 10–30 mm below the scalp. Simultaneous measurement studies have shown that the best correlation between the NIRS signal and parameters in other imaging method, such as PET (Hock et al., 1997) and fMRI (Schroeter et al., 2006), was at a depth of approximately 1–1.5 cm from the skin. Nevertheless, more recent research has shown that the depth sensitivity at a depth of around 1 cm into the intracranial space is quite low by the thickness of several overlying tissue layers, such as scalp (Strangman et al., 2013). Although there is little evidence regarding this issue in young children, we had to note that the NIRS system cannot measure the activations of the deeper areas in the brain.

Third, some of the previous study used between subject designs, which may have a problem. As noted above, the direct comparison or integration of data between subjects is difficult because the optical path length differs cross participants and head positions (Zhao et al., 2002). Thus, longitudinal or microgenetic method may be appropriate to address the developmental changes in activations of specific brain regions. Moreover, even in the longitudinal study, researchers had to care about the comparisons between several different time points. Brain as well as superficial tissues such as scalp and skull are developing during childhood. A MRI study showed that there were developmental changes in the distance between scalp and brain in children. The distance depends on children's age and the brain regions (Beauchamp et al., 2011), but the frontal regions showed age-related changes in the distance until middle childhood. Coupled with the issue of the depth above, we have to consider such data to analyze and interpret the results in the NIRS signals.

Despite the limitations, NIRS has several advantages for research on infants and young children. For example, as noted above, NIRS is noninvasive and does not require very exact fixations of body and head such that other neuroimaging methods require. Children can sit in a chair during an experiment. Also, a NIRS experiment can be conducted silently compared to an fMRI experiment. The facts make research on infants and young children easier. Nevertheless, there would be motion artifacts in young children's research, which may benefit from analyses that separated functional brain activities from other components in NIRS signals (Scholkmann et al., in press). In addition, the NIRS is portable and less costly. We can measure children's brain activations in natural settings, such as children's home, kindergartens or nursery schools as well as an experimental room. Moreover, NIRS can apply to children with developmental disorders and behavioral difficulties, and have the potential for the use of their interventions.

### **CONCLUSION AND FUTURE DIRECTIONS**

Collectively, the results of these studies show that both children and adult participants show significant activation in the prefrontal regions when performing cognitive shifting, inhibitory control and working memory tasks. Importantly, the children with ADHD and those with ASD who had difficulties with the tasks exhibited abnormal activation in the prefrontal areas. Taken together, the studies discussed in this review suggests that activation in the prefrontal regions may be important for successful performance on EF tasks in young children.

The next step is to determine how the development of prefrontal function may be related to other aspects of cognitive and social development. It has been shown that the development of EF correlates with the development of socio-cognitive skills, such as theory of mind, communicative skills, and emotional regulation (Dempster, 1992; Eisenberg et al., 1997; Carlson and Moses, 2001; Moriguchi et al., 2008). Given the correlational evidence, researchers have suggested that EF may contribute to the emergence of such skills (Moses, 2005). However, the exact mechanisms of the relationship between socio-cognitive skills and EF are still unclear. NIRS may aid in understanding such relationships. In fact, recent research has examined the relationship between prefrontal activation and emotion regulation (Fekete et al., in press; Perlman et al., in press). However, further research should be conducted to examine the exact mechanism underlying this relationship.

Another issue is how prefrontal activation may change developmentally across different tasks (e.g., working memory and cognitive shifting tasks). On the behavioral level, a single-factor model (general EF) is sufficient to account for the data for preschool-aged children (Wiebe et al., 2008). The available results suggest the possibility that the prefrontal regions may be generally activated across different tasks in younger children, but may become localized to specific regions in older children. Johnson (2011) proposed that some regions in the cerebral cortex may start with broad functionality, and consequently are partially activated in different stimuli and tasks. Indeed, Durston et al. (2006) reported the developmental shift from diffuse to focal activations in the prefrontal regions when school-aged children were given a Go/NoGo task.

In addition, other brain regions may be activated during these tasks. Given the limitation of the NIRS system, this review focused on the role of the prefrontal regions in the development of EF. However, using fMRI, Morton et al. (2009) reported that schoolaged children showed significant activation of the superior parietal cortex, dorsolateral prefrontal cortex, and presupplementary motor regions during the DCCS task. Furthermore, Monchi et al. (2001) have revealed that the dorsolateral prefrontal cortex and parietal cortex are significantly activated in adults during the WCST. Thus, future studies should examine activation in other brain regions in young children during EF tasks. Some recent reports examined other brain regions as well as the prefrontal regions during EF tasks in young children (Buss et al., in press). Such examinations may lead to a better understanding of the brain mechanisms involved in the development of EF in young children.

#### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 27 September 2013; accepted: 27 November 2013; published online: 17 December 2013.*

*Citation: Moriguchi Y and Hiraki K (2013) Prefrontal cortex and executive function in young children: a review of NIRS studies. Front. Hum. Neurosci. 7:867. doi: 10.3389/fnhum.2013.00867*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2013 Moriguchi and Hiraki. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## Activation of the rostromedial prefrontal cortex during the experience of positive emotion in the context of esthetic experience. An fNIRS study

### *Ute Kreplin\* and Stephen H. Fairclough*

*School of Natural Science and Psychology, Liverpool John Moores University, Liverpool, UK*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Leonid Perlovsky, Harvard University and Air Force Research Laboratory, USA Hasan Ayaz, Drexel University, USA*

#### *\*Correspondence:*

*Ute Kreplin, School of Natural Science and Psychology, Liverpool John Moores University, Tom Reilly Building, Byrom Street, Liverpool, L3 3AF, UK e-mail: u.kreplin@2011.ljmu.ac.uk*

The contemplation of visual art requires attention to be directed to external stimulus properties and internally generated thoughts. It has been proposed that the medial rostral prefrontal cortex (rPFC; BA10) plays a role in the maintenance of attention on external stimuli whereas the lateral area of the rPFC is associated with the preservation of attention on internal cognitions. An alternative hypothesis associates activation of medial rPFC with internal cognitions related to the self during emotion regulation. The aim of the current study was to differentiate activation within rPFC using functional near infrared spectroscopy (fNIRS) during the viewing of visual art selected to induce positive and negative valence, which were viewed under two conditions: (1) emotional introspection and (2) external object identification. Thirty participants (15 female) were recruited. Sixteen pre-rated images that represented either positive or negative valence were selected from an existing database of visual art. In one condition, participants were directed to engage in emotional introspection during picture viewing. The second condition involved a spot-the-difference task where participants compared two almost identical images, a viewing strategy that directed attention to external properties of the stimuli. The analysis revealed a significant increase of oxygenated blood in the medial rPFC during viewing of positive images compared to negative images. *This finding suggests that the rPFC is involved during positive evaluations of visual art that may be related to judgment of pleasantness or attraction. The fNIRS data revealed no significant main effect between the two viewing conditions, which seemed to indicate that the emotional impact of the stimuli remained unaffected by the two viewing conditions.*

**Keywords: fNIRS, BA10, emotion, esthetics, prefrontal cortex**

#### **INTRODUCTION**

The experience of viewing art is influenced by a modulation of attentional focus between external features of the stimuli and internal feelings/thoughts. Internal cognitive processes such as object recognition, memory recall and mental imagery facilitate content recognition during the viewing of visual art (Fairhall and Ishai, 2008). Recognition of familiar content evokes a pattern of activation in multiple extrastriate ventral and dorsal regions, the hippocampus, intra parietal sulcus and inferior frontal gyrus (Ishai et al., 2007; Fairhall and Ishai, 2008; Nadal et al., 2008; Nadal and Pearce, 2011). According to a MEG time frequency analysis, a peak of acitivity around 170 ms has been related to the commencement of coding for object identity and transformation of sensory code to cognitive processing has been associated with a peak of activity at approx. 170 ms during the observation of visual art (Munar et al., 2011). This process of feature extraction from visual art and the generation of associated thought and feelings have a distinct temporal window.

The rostral prefrontal cortex (rPFC) may be an important site of activity during the processing of art; this region has been associated with higher order cognitive processes such as prospective memory (Volle et al., 2010; McDaniel et al., 2013), emotional regulation strategies (Amting et al., 2010; Viviani et al., 2010; Campbell-Sills et al., 2011) and sustained attention (Van Veen and Carter, 2006; Ernst et al., 2012). However, there is little consensus regarding the functional specificity and cytoarchitecture of the prefrontal cortex (PFC), particularly the rostral area of the PFC. Ramnani and Owen (2004) suggested that the rPFC is activated when the outcomes of two or more separate cognitive operations require integration in the pursuit of a higher behavioral goal. Other accounts emphasized the involvement of medial rPFC during the processing of self-related information (Seitz et al., 2009; Denny et al., 2012).

The gateway hypothesis (Burgess et al., 2003, 2007) was developed to connect activation in the rPFC to higher-order selfreferential processing and the evaluation of internally generated information. According to this model, rostromedial areas of the PFC (medial BA10) are implicated in the maintenance of attention towards external stimuli whereas activation of the rostrolateral areas (lateral BA10) are associated with the preservation of attention on internal cognitions; this functional differentiation is proposed to act as a gateway between the direction of attention towards external and internal stimuli (Burgess et al., 2007). The central proposal of the model is that the rPFC is part of a system that allows conflicts to be resolved during ambiguous situations (where information activating relevant schemata has low triggering input) or increases activation of schemata in accordance with higher-level goal representations (Cupchik et al., 2009; Volle et al., 2010; Henseler et al., 2011). Evidence for this functional differentiation has been observed during the comparison of shapes (Henseler et al., 2011), letters (Gilbert et al., 2005; Benoit et al., 2012) and the identification of features between two different stimuli such as texture or aspects of geometric shapes (Volle et al., 2010).

The general function of the rPFC as described by the gateway hypothesis is twofold. This area enables the activation of schemata in a situation where no schema is sufficiently triggered by incoming stimuli, e.g., if the stimulus is entirely novel and cannot be associated with existing information. Secondly, the rPFC enables attentional bias when many schema are simultaneously activated (e.g., if a situation is very difficult or complex) or if there are a multitude of possible established outcomes without an obvious advantage to one of them (Burgess et al., 2005). The rPFC plays a key role in the goal-directed co-ordination of stimulus-independent and stimulus-orientated cognitions in situations where established patterns of behavior are insufficient. Stimulus-independent cognitions include introspection or creative thoughts which are neither provoked by nor directed toward external stimuli. Stimulus-oriented cognitions represent the opposite category, being provoked and oriented towards sensory input. The rPFC would be typically activated in situations that are novel or where a specific demand for it has been determined (e.g., "I must pay special attention to...", "I must think about..."; Burgess et al., 2003, 2005, 2007; Gilbert et al., 2005; Volle et al., 2010; Benoit et al., 2012). The contemplation of visual art requires a shift from stimulus-dependent processing to those stimulus-independent processes that permits an assessment of stimuli as being esthetically pleasing or not. It may be hypothesized that attention is directed to external properties of the stimulus (identification of physical properties within the painting), the reinvestment of attention onto internally generated thoughts (what does the artist want to say with this painting), and the reinvestment of attention onto subjective self or personal entity (what does the painting mean to me). It is plausible that the rPFC is involved in object identification as well as directing attention to those self-referential states that are relevant to esthetic appreciation, but the roles of medial and lateral areas of the rPFC during the contemplation of visual art as defined by the gateway hypothesis remains unclear.

There is evidence connecting activity in the rPFC to aspects of cognition that are implicit within emotional processing, e.g., attention to emotion, emotion regulation, appraisal or interpretation of emotion. For instance, Phan et al. (2002) reported strong connections between the PFC (BA9/10) and the anterior cingulate cortex (ACC) and suggested that both areas of the PFC could serve as top-down modulators of intense emotional responses (see also Amting et al., 2010; Holroyd and Yeung, 2012). Evidence for this interpretation stems from human lesion studies where damage to the PFC leads to socially inappropriate expressions of emotions and impairment in making advantageous personally relevant decisions suggesting a lack of awareness/comprehension of emotionally "loaded" situations (Damasio, 1999; Leopold et al., 2012; Maier and di Pellegrino, 2012). Similarly, activation of the rPFC was related in a linear fashion with an emotion induction task that required different degrees of self-monitoring (identifying with the feelings/emotions depicted in a picture compared to just viewing a picture) (Herrmann et al., 2003; see Denny et al., 2012 for a review).

Studies relating activity in the rPFC to the process of emotional regulation present an alternative to the gateway hypothesis. The regulation of emotions has been associated with activation of the rPFC during the up- and down regulation of emotions (Mitchell, 2011). Furthermore medial BA10 has been linked to activity related to the subjective self or personal entity (Amting et al., 2010; Denny et al., 2012). Seitz et al. (2009) suggested that nodes in the medial PFC participate in early processing of sensory information and mediate the value judgment of the stimulus by assessing self-relevant meaning to the sensations. An esthetic experience, or more specifically the contemplation of visual art, is a highly subjective process. It is therefore important that the experiences are self-referential, particularly if the viewer is not trained in an art-related subject and bases his/her evaluation of visual art largely on personal experiences. Evidence supporting this interpretation and the involvement of the medial rPFC was provided by Vessel et al. (2012) who reported an increase of activation in the medial rPFC for those paintings subjectively judged as most esthetically moving. This activation of the medial rPFC was specific to the assessment of esthetic pleasure. This study suggested that a highly-subjective emotional connection to visual art is important to the esthetic experiences and activation in the rPFC was associated with this type of experience. Alternatively, the involvement of the lateral rPFC during esthetic experiences was reported by Cupchik et al. (2009) who investigated how cognitive control/perceptual facilitation and the experience of emotion contributed to esthetic perception. Participants in this study were instructed to view a painting from either a pragmatic everyday viewpoint or from an esthetic viewpoint. Pragmatic viewing was associated with activation of the fusiform gyrus and areas related to object recognition whereas the esthetic viewing condition activated the insula and left lateral rPFC (BA10). The authors interpreted activation of the latter in terms of stimulus-independent thought related to the contemplation of visual art.

The aim of the current study was to differentiate activation within the rPFC using functional near infrared spectroscopy (fNIRS) during the viewing of visual art selected to induce positive and negative mood. Both categories of image were viewed under two conditions (emotional introspection and external object identification) designed to draw attention to stimulus-independent and stimulus-dependent features respectively. Images were viewed for 60 s overall and split into three viewing periods (early, middle and late) for the analysis. The extended viewing time was chosen because temporal differences have been observed during activation of the PFC during emotional experiences measured by fNIRS, where an overshoot of activation into later periods was reported (León-Carrion et al., 2008), and during the experience of art (Munar et al., 2011). Our primary hypothesis was that emotional introspection would activate lateral areas of the rPFC whereas viewing the image with an emphasis on external object identification would selectively activate the medial rPFC. Our alternative hypothesis was that emotional introspection would activate the medial rPFC as predicted by the studies conducted by Seitz et al. (2009) and Vessel et al. (2012).

### **METHODS**

#### **THE SURVEY**

Standardized visual materials, such as the International Affective Picture System (IAPS; Lang et al., 1993) are often used to provoke emotional experience under controlled conditions. The purpose of the survey exercise was to create a database of esthetic images with known psychological properties for experimental purposes. Sixty-three paintings including both representational and abstract images were selected from online sources; all artists had sold or exhibited their work and agreed to allow their work to be used in the survey exercise. Participants were asked to rate each image on six components represented by 9-point Likert scales. Three components represented cognitive components, these scales were labeled: complexity (how complex was this image?), comprehension (do you understand this image?) and novelty (how novel or unusual was the image?). The remaining three components represented emotional factors including: valence (is the image positive or negative?), activation (do you find the image stimulating) and attractiveness (do you find the image attractive or repellent?). The cognitive components were derived from Silvia's(2008) analysis of interest whilst emotional components were based upon the Self-Assessment Mannikin (Lang et al., 1993).

The survey was made available online and 1043 participants (63% female) with a mean age of 33 years. (s.d. 14.24) provided ratings for the images. The resulting database was used to select images rated as positive/negative valence for the experimental study.

#### **EXPERIMENTAL STUDY**

#### *Participants*

Thirty right handed participants (15 female) were recruited from an undergraduate population. Participants had a mean age 22 years. (s.d. = 3.26 years) with no formal training in an artrelated subject and no history of neurological disorder. Participants were informed about the procedure and operating mode of the fNIRS prior to providing written consent. All procedures were approved by the University Research Ethics Committee prior to data collection.

#### *Experimental task*

The stimuli were projected onto a white wall in front of the participants using E-prime 2.0 (PST Inc.); the image dimensions were 2040 × 786 pixels. The viewing distance to the screen was approx. 1.90 m. Each stimulus was presented for 60 s and preceded by a 60 s baseline consisting of a light grey screen with a fixation \*. Following image presentation, participants were asked to provide ratings for valence and complexity on the scales used in the survey; ratings were recorded in e-prime via a keypad. The tasks were designed to induce different demands on attending to external properties of the stimulus (the spot-the-difference (SD) task) and to internal cognitions (the emotional introspection (EI) task).

We closely considered the prerequisites stated by Burgess et al. (2005) that a task requires to draw attention to external stimulus processing when designing the SD task. According to Burgess et al. (2005) an external attentional task requires: (1) that the information to be processed is currently available (i.e., present in the sensory environment); (2) that the attention is directed to external stimuli or stimulus features; and (3) that the operations involved prior to responding are relatively automatic or welllearnt. To elicit attending to external stimuli features during the SD task, all images were duplicated to form a pair and between three and six aspects of the image on the right were modified using PaintShopPro (**Figure 1A**). Instructions in the SD condition stated: "If you see "SD" on the screen you will be asked to spot the differences between two images. You will be able to see the answers after the image". Following the ratings in the SD task participants were provided with the same picture but with the differences highlighted with a red circle. All images were used in the SD task and the emotion induction task. Participants were prompted with SD or E before the image appeared as indicator which task to perform next during testing.

The EI condition was designed to initiate internal processes with respect to: (1) the information attended to is being processed internally; (2) that this information is self-generated or comes from a previously witnessed episode; and (3) that the response to be made are triggered by these internal representations. Instructions were as follows: "If you see an "E" on the screen you are asked to think about how the artwork makes you feel, i.e., what emotions does it trigger in you? Does it make you feel sad/happy/angry etc.? Does it remind you of an emotional event you have experienced in the past? Don't worry about the message the artist tried to bring across, just think about how the image makes YOU feel or if it makes you think of something that you found emotional". Participants were asked to write, using pen and paper, brief notes about these associations following the picture presentation. Participants were not asked to hand these notes to the experimenter and took them away upon completion of testing to assure confidentiality. The images were presented as identical pairs during the EI task to ensure consistency with the SD viewing condition (**Figure 1B**). A practice trial was completed for each condition, and the opportunity to ask questions was given to insure the instructions were fully understood before the start of the experiment. The presentation of positive/negative images and EI/SD was randomized.

*Stimuli.* Sixteen images (8 in a positive valence category and 8 in a negative valence category) were selected according to ratings of valence and complexity obtained during the survey (see section The Survey). Mean ratings for positive images were 2.57 (+/− 1.27) and negative images 7.17 (+/− 1.3), whilst subjective ratings of image complexity were constant between positive and negative images (mean 4.58 +/− 2.22 and 4.07 +/− 2.14 respectively).

**FIGURE 1 | (A)** Pictures used in the positive SD task ("Best Abstract" © Adrian Borda). The right image has slight differences to the left image, differences are circled in red. **(B)** Two identical images used during negative EI (Monika Weiss "Elytron" 2003,

#### self-shot photography, performance, installation, sculpture and video. Courtesy the artist and Chelsea Art Museum, New York). Participants were asked to think about how the images made them feel.

#### *Procedure*

Participants were informed about the nature of the study upon arrival and provided written consent before the fNIRS device was fitted. Participants were informed about the study tasks through written instructions projected onto the wall in front of them. Participants had the opportunity to ask questions before and during a practice trial for each task that was shown after the instructions. The experimental protocol began after the experimenter was satisfied that the participant understood the experimental tasks. Participants were thanked for their participation following the experiment and compensated with a £10 voucher for their time.

#### *fNIRS data collection*

fNIRS was recorded using fNIR Imager1000 and COBI data collection suit (Biopac System Inc.) and is described in detail elsewhere (Ayaz et al., 2010). The system has a temporal resolution of about 500 ms for one complete data acquisition cycle (about 2 Hz). The 16 channel probe was placed on the forehead aligned to Fp1 and Fp2 of the international 10–20 system, and rotated so that Fpz corresponded to the midpoint of the probe (Ayaz et al., 2006; **Figure 2**). Areas underlying the 16 voxels are right and left superior and inferior frontal gyrii (BA10 and BA46). Cognitive and emotional functions associated with BA10 and BA46 have been explored using fNIRS application similar to the device used

in this work (Plichta et al., 2006; Izzetoglu et al., 2007; León-Carrion et al., 2008).

fNIRS data were analyzed offline using fNIRS-Soft (Ayaz et al., 2010). Raw data was subjected to a Sliding-window Motion Artefact Rejection (SMAR) algorithm to remove motion artefacts and saturated channels (Ayaz et al., 2010). Oxygenated haemoglobin (HbO) and deoxygenated haemoglobin (HHB) were calculated using the modified Beer-Lambert Law. A finite impulse response linear phase low-pass filter, with order 20 and cut-off frequency of 0.1 Hz was applied to attenuate high frequency noise, respiration and cardiac effects (Izzetoglu et al., 2007; Ayaz et al., 2010). Sixteen segments with durations of 60 s were extracted using synchronization markers. Segments were averaged according to condition (Positive EI, Negative EI, Positive SD, Negative SD).

There is currently no strong consensus in the literature regarding the optimal feature of brain activation that can be derived from fNIRS data. Research investigating emotional processes using fNIRS has reported significant changes in the PFC for HbO alone (León-Carrion et al., 2008), for HHb alone (Ernst et al., 2012) or for both HbO and HHb (Glotzbach et al., 2011). It has been argued that HHb is sensitive to local haemodynamic changes, less prone to influences from psychophysiological noise, such as breathing or heart rate and has a close association with the blood oxygenation dependent (BOLD) signal obtained from

fMRI. However, HbO is the parameter which is less sensitive to variation in probe placement due to head size and shape because HbO activation is more global compared to HHb activation (Wobst et al., 2001; Hoshi, 2005; Plichta et al., 2006). We decided to calculate a compound score for oxygenation (Oxy = HbO – HHb) in order to capture both measures whilst controlling for changes in blood volume (Ayaz et al., 2010).

#### **RESULTS**

Individual differences in the subjective evaluation of visual art have been highlighted as a problem in previous studies (e.g., Cupchik et al., 2009; Vessel et al., 2012). Our participants were exposed to four groups of images (EI/positive, EI/negative, SD/positive, SD/negative), each of which contained four individual images. In order to control for individual differences, we selected only those three images from the sample of four that were most representative of their group designation in order to create the best representation of that image category for each participant, i.e., the three images in the EI/negative valence group with maximum scores for valence (rated on a 9-point Likert scale; 1 = positive, 9 = negative) were assumed to represent the best example of EI/negative valence for that particular person.

We conducted a manipulation check of the subjective valence ratings to assess differences between positive and negative images under the two viewing conditions. A 2 (condition) × 2 (valence) analysis of variance (ANOVA) was conducted on subjective ratings from three images in each valence/viewing condition. A significant main effect for valence (*F*(1,29) = 720.50, *p* < 0.01, η<sup>2</sup> = 0.96), and an interaction between condition and valence (*F*(1,29) = 5.06, *p* < 0.03, η<sup>2</sup> = .014) was found. Negative images (mean 7.17, +/− 1.07) were rated as significantly more negative than positive images (mean 2.57, +/− 0.85). Post-hoc analysis showed no significant difference between the EI and SD condition for negative images. For positive images a marginal trend was identified with images in the EI (mean 2.57, +/− 0.85) condition being rated as more positively than those viewed in the SD condition (mean 2.98, +/− 0.78, *p* < .06).

We separated 60 s of data into three time epochs, early, middle and late, each consisting of 20 s of data. Separate 3 (time) × 2 (condition) × 2 (valence) multivariate analysis of variance (MANOVA) was conducted for each voxel for the compound score oxygenation (Oxy), Greenhouse-Geisser corrections were applied to violations of sphericity. Outliers above and below three standard deviations were excluded. Results yielded a significant main effect for valence at voxel 3 (*F*(1,27) = 5.24, *p* < 0.03, η<sup>2</sup> = 0.16), voxel 5 (**Figure 2** for voxel location) (*F*(1,28) = 8.74, *p* < 0.001, η<sup>2</sup> = 0.23) and voxel 9 (*F*(1,26) = 4.32, *p* < 0.04, η<sup>2</sup> = 0.14), all showing greater Oxy for positive images. This effect is illustrated in **Figure 3** where heat maps were generated across the rPFC region based on the 16 voxels shown in **Figure 2**.

### **DISCUSSION**

The subjective self-report data supported the face validity of the valence manipulation, i.e., positive and negative images were rated appropriately by participants. Our analysis indicated that activation of medial BA10 (as defined by an increase of Oxy) was enhanced during the viewing of visual art that induced positive emotions compared to those paintings that provoked negative emotion (**Figure 3**). The fNIRS data showed no main effect for viewing condition, which indicated that activation of the medial rPFC was unaffected by the two different viewing conditions.

The increase of activation in response to positive stimuli was unexpected but is not without precedence in the literature (Ernst et al., 2012). It is suggested that positive images activated the medial rPFC because esthetic pleasantness was associated with this category of picture and this effect was unaffected by viewing condition. Previous studies have reported that activation in the medial rPFC is positively correlated with esthetic evaluation (Vartanian and Goel, 2004; Di Dio and Gallese, 2009; Ishizu and Zeki, 2011). When Vessel et al. (2012) found greater activation in the medial rPFC for esthetically pleasing images, they

speculated that intense esthetic experiences had high personal relevance, which created a heightened integration of external (sensory/somatic) sensations and internal (evaluative/emotional) states, as the individual experienced an emotional connection to the art. In support of this position, Seitz et al. (2009) suggested that the medial PFC is involved in the attribution of self-relevant, immediate and intuitive meaning. Therefore increased activation of the medial rPFC in the current study may have been the result of self-monitoring of positive emotions or pleasantness judgements during the contemplation of art.

The medial area of the PFC has been implicated in emotional processing in studies with a primary focus on emotion induction (Gray et al., 2002; Glotzbach et al., 2011; Euston et al., 2012; Lindquist and Feldman-Barrett, 2012). Increased prefrontal activation and medial areas in particular, have been associated with self-regulatory strategies designed to minimize negative affect in fMRI studies (for a review see Mitchell, 2011). However, studies investigating self-regulatory strategies during emotional experiences have a tendency to focus on negative emotions such as fear (Glotzbach et al., 2011), anger, sadness, or disgust (Lindquist and Feldman-Barrett, 2012). Herrmann et al. (2003) used positive and negative stimuli to investigate the change of Oxy in the PFC using fNIRS during two types of emotional induction, one with a higher and one with a lower self-monitoring component. The task with the high self-monitoring component resulted in higher levels of oxygenated blood in the medial rPFC regardless of valence. *This study suggests that the rPFC is highly sensitive to emotional induction tasks where an element of self-monitoring is implicated. However, we found activation only for positive images, regardless of viewing condition, and it is unclear why negative emotions would not have activated the rPFC unless the emotional response was specific to esthetic pleasure or the beauty of the picture.*

The gateway hypothesis proposed a functional differentiation between activation for stimulus-dependent and stimulusindependent cognitions in the rPFC (Burgess et al., 2003, 2007). However, the current study found no evidence to support this position because the two viewing conditions had no significant effect on activation in the rPFC. It could be argued that the SD task may not have been suitable to draw attention toward external properties of the stimulus. However the same task was successful in eliciting the functional differentiation in rPFC activation described by the gateway hypothesis in a previous study (Volle et al., 2010). Our stimuli consisted of complex visual art which contrasts with previous research that used simple geometric patterns (Volle et al., 2010) or shapes (Henseler et al., 2011) to demonstrate the effect described by the gateway hypothesis. The increased complexity of our stimuli may account for the absence of any effect on rPFC activation.

It could be argued that the EI task did not conform to the prerequisites set out by Burgess et al. (2005) for internally generated thoughts because the stimulus was present at all times. To invoke self-referential thoughts, participants were instructed to think about how the images made them feel, how they felt connected to them and what images or memories it provoked in them. The presence of the image throughout the introspective task may have confounded the result but we believe that the EI task used in the current study was ecologically valid with respect to real-life behavior in gallery spaces. Previous findings have reported activation of lateral areas during the contemplation of visual art, and attributed this to a focus on internally generated thought related to the artwork (Cupchik et al., 2009). However, these authors did not manipulate their viewing conditions to investigate the gateway hypothesis during the contemplation of art, but rather used the gateway hypothesis as a possible explanation of increased activity in the lateral rPFC. Nonetheless, future investigation may study the gateway hypothesis during the contemplation of visual art using a paradigm where the stimulus is not present during the EI task.

Future studies investigating the gateway hypothesis may benefit from an emotionally neutral condition during the contemplation visual art and the inclusion of a scale asking for subjective experiences of esthetic pleasantness or beauty. Questions remaining unanswered regarding the absence of rPFC activation during negative emotions and the interaction between a self-relevant and other-relevant focus during the contemplation of art. Future research may consider whether pictures with negative valence can be esthetically pleasing and how this is related to rPFC activation would be of benefit to neuroesthetics.

Our results suggested that emotional processing took precedence over differing viewing instruction during visual contemplation of art. Emotional salience may have been brought to the forefront because of participants search for personal meaning of the art images. Thus participants will have drawn onto their own evaluations and personal association of the art to form a subjective judgement about their value.

#### **REFERENCES**


to aesthetic experience. *Brain Cogn.* 70, 84–91. doi: 10.1016/j.bandc.2009. 01.003


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 26 September 2013; paper pending published: 23 October 2013; accepted: 02 December 2013; published online: 20 December 2013*.

*Citation: Kreplin U and Fairclough SH (2013) Activation of the rostromedial prefrontal cortex during the experience of positive emotion in the context of esthetic experience. An fNIRS study. Front. Hum. Neurosci. 7:879. doi: 10.3389/fnhum.2013.00879*

*This article was submitted to the journal Frontiers in Human Neuroscience*. *Copyright © 2013 Kreplin and Fairclough. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## Functional brain imaging using near-infrared spectroscopy during actual driving on an expressway

#### *Kayoko Yoshino1, Noriyuki Oka1, Kouji Yamamoto2, Hideki Takahashi <sup>3</sup> and Toshinori Kato1 \**

*<sup>1</sup> Department of Brain Environmental Research, KatoBrain Co. Ltd., Tokyo, Japan*

*<sup>2</sup> Department of Environment/Engineering, Tokyo Branch, Central Nippon Expressway Co. Ltd., Tokyo, Japan*

*<sup>3</sup> Department of Environment/Engineering, Central Nippon Expressway Co. Ltd., Nagoya, Japan*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Yukiori Goto, Kyoto University, Japan Anqi Zhang, Johns Hopkins University, USA*

#### *\*Correspondence:*

*Toshinori Kato, Department of Brain Environmental Research, KatoBrain Co., Ltd., 13-15-104, Shirokanedai 3, Minato-ku, Tokyo 108-0071, Japan e-mail: kato@katobrain.com*

The prefrontal cortex is considered to have a significant effect on driving behavior, but little is known about prefrontal cortex function in actual road driving. Driving simulation experiments are not the same, because the subject is in a stationary state, and the results may be different. Functional near-infrared spectroscopy (fNIRS) is advantageous in that it can measure cerebral hemodynamic responses in a person driving an actual vehicle. We mounted fNIRS equipment in a vehicle to evaluate brain functions related to various actual driving operations while the subjects drove on a section of an expressway that was not yet open to the public. Measurements were recorded while parked, and during acceleration, constant velocity driving (CVD), deceleration, and U-turns, in the daytime and at night. Changes in cerebral oxygen exchange (-COE) and cerebral blood volume were calculated and imaged for each part of the task. Responses from the prefrontal cortex and the parietal cortex were highly reproducible in the daytime and nighttime experiments. Significant increases in -COE were observed in the frontal eye field (FEF), which has not been mentioned much in previous simulation experiments. In particular, significant activation was detected during acceleration in the right FEF, and during deceleration in the left FEF. Weaker responses during CVD suggest that FEF function was increased during changes in vehicle speed. As the FEF contributes to control of eye movement in three-dimensional space, FEF activation may be important in actual road driving. fNIRS is a powerful technique for investigating brain activation outdoors, and it proved to be sufficiently robust for use in an actual highway driving experiment in the field of intelligent transport systems (ITS).

#### **Keywords: fNIRS, driving, frontal eye field, outdoor brain activation, acceleration, deceleration, constant velocity driving, U-turn**

#### **INTRODUCTION**

Driving a vehicle requires use of the higher brain functions such as planning, decision-making, and visual attention, for basic driving operations as well as driving safety. It has been suggested that white matter lacunar infarcts in the frontal lobe might be a predictor of traffic crashes (Park et al., 2013). The function of the prefrontal cortex is thus considered to be significant in driving behavior. The "Global status report on road safety" prepared by the WHO (2009) reported that the number of traffic fatalities has remained constant or declined slightly in the developed countries, but it has increased in most countries. According to a public report in Japan (National Public Safety Commission and National Police Agency, 2013), the number of traffic accidents and resulting injuries has declined in the past few years on ordinary roads, but it has increased on expressways.

Under these circumstances, from the point of view of organizations responsible for highway construction, lighting, signage, and the like, one goal of a study of this kind is the potential development of an evaluation system capable of examining physiologically the effects on the brain of highway design, and identifying ways to improve the ease of driving and highway safety. This requires a technique capable of obtaining information that is potentially useful for improving traffic safety by imaging brain activation while the subject is driving a vehicle. In the field of neuroscience, however, which has been developed through experiments indoors, there has been little development of techniques suited for outdoor activities, where the subjects engage in dynamic activities such as driving a vehicle. Driving simulation experiments have therefore been used to image brain activation related to car driving. The involvement of the prefrontal cortex in driving behavior has been reported in functional magnetic resonance imaging (fMRI) studies (Graydon et al., 2004; Horikawa et al., 2005; Calhoun and Pearlson, 2012). However, these experiments were performed in conditions that differed from actual driving, such as pushing buttons (Graydon et al., 2004) and operating a joystick (Horikawa et al., 2005). Technical problems remained in experiments where the simulation environment included a pedal and a steering wheel (Calhoun and Pearlson, 2012; Schweizer et al., 2013); namely, the human subjects were in a supine position, while the fields of view and depth from the driver's seat were smaller than during actual driving.

Functional near-infrared spectroscopy (fNIRS) has attracted attention in this field, and driving simulation experiments have been conducted using fNIRS with the subjects in a sitting position (Yanaginuma et al., 2007; Watanabe et al., 2011; Tsunashima et al., 2012). However, it is difficult to reproduce the various effects on the body such as gravity, and it is still unclear how the brain functions while driving on an actual road. Compared with an ordinary road, an expressway provides fewer changes in the background, because of the unobstructed view. However, driving speeds are much greater on an expressway and the speed differentials during acceleration or deceleration are higher than those on ordinary roads. In simulation experiments, the subjects are in a stationary state, and there have been no reports on brain activation when a subject is actually driving on an expressway. Accordingly, we conducted this first study to detect the brain activation of drivers as they drive on an actual expressway. Vehicle-mounted fNIRS equipment was used to obtain measurements, and brain activation was imaged during various driving operations, with the goal of acquiring basic data during actual highway driving. After recording the brain activation during operation by normal adult drivers in daytime and nighttime conditions, we investigated the sites of the brain involved in highway driving, that may be relevant in the development of traffic safety measures.

#### **METHODS SUBJECTS**

Twelve healthy adults participated in this study (eight males and four females, average age: 33.3 ± 4.5 years). Using the Edinburgh Handedness Inventory we confirmed that all the subjects were right-handed. The subjects had no history of mental illness or central nervous disorders, and took no medications on the day of the experiment. Written consent was obtained from the participants before enrollment in the study and the protocol was approved in advance by the ethics committee at KatoBrain Co., Ltd. To ensure the safety of the experiments, the subjects reported their physical condition and the amount of sleep they had the night before the experiment. There were no reports of extreme lack of sleep, and the experiments were conducted after the subjects confirmed that they felt fine. The subjects' average length of driving experience was 11.8 ± 5.8 years. Their frequency of driving was 6.1 ± 1.6 times/week, and their frequency of expressway driving was 4.5 ± 6.5 times/month. Only two subjects had experienced an accident (neither accident involved another vehicle or any personal injuries), and the average number of accidents was 0.2 ± 0.4. The average number of traffic violations was 1.3 ± 1.3 times, mostly for speeding. Since recruitment of the subjects was based on the conditions of age, right-handedness, and frequency of driving on a daily basis, the subjects' genders, their driving histories, and their histories of violations and accidents were completely random.

#### **LOCATION OF THE EXPERIMENT**

The experiment was performed in the Okitsu district, Shizuoka Prefecture, Japan, on a section of the Shin Tomei Expressway immediately before it entered service (**Figure 1**) (Yamamoto et al., 2012; Kato et al., 2013). The installation of wall parapets, pavement sections, and tunnel lighting had already been completed, so there were no problems with the safety of vehicle travel. The experimental course was restricted to the passage of one vehicle, so there were no vehicles present other than the test vehicle. Guard personnel were stationed at each point on the experimental course and they monitored the status of the road. They could immediately contact the test car using a transceiver if any danger arose on the course, such as an animal intrusion.

As **Figure 2A** shows, the full length of the test course was 2875 m, and it included a left curve (*R* = 5000) and a right curve (*R* = 5000). The curves were moderate, so the subjects did not need to engage in difficult handling operations while driving. The slope of the course included a downhill gradient of 2.0% and an uphill gradient of 2.0%, gradual slopes that the drivers hardly noticed. The road width was 18.25 m and there were two lanes (3.75 m wide) on each side. The test vehicle traveled in the left lane, because traffic moves on the left in Japan.

#### **EXPERIMENTAL PROCEDURES**

For each subject, the experiment took place over 2 days for preliminary trials and 2 days for the actual experiment. The daytime and nighttime trials were performed on different days for each subject, and the order of the experiments (day or night) was randomized. In the preliminary trials, performed within 10 days of the actual experiment, the subjects drove under the same conditions as in the actual experiment except for the actual course, in the test vehicle, wearing the fNIRS probe attachment. By the time of the actual experiments, the subjects were used to the experimental environment, and none of the subjects complained of any stress.

Diagrams of the different parts of the task are shown in **Figure 2A**. The task comprised 7 parts (A–G), and one trial consisted of this series A–G. The subjects performed three trials each, in the daytime and at night, on different days.

In A, the test car remained stationary for about 25–30 s in the parking mode of the automatic transmission, with the engine started and the parking brake set. The subject remained in a resting state in the driver's seat. In B, which measured 261 m, the subject accelerated to 100 km/h. The automatic transmission and the parking brake were operated by the experimenter, who rode in the passenger seat. A traffic cone was placed at the end point of B. In C, constant velocity driving (CVD) was maintained at 100 km/h inside a tunnel. After around 5 s at the beginning of C, while the subject adjusted to the feel of 100 km/h, the speedometer was hidden to prevent excessive attention to speed. To ensure the safety of the experiment, the speedometer could be seen by the experimenter in the passenger seat, who was to instruct the subject to abort the trial if the speed exceeded 125 km/h (no trials were actually aborted). CVD was continued outside the tunnel in D, and then in a tunnel again in E. F was the deceleration section of the course, indicated by a sign placed at the starting point of F, and G was a U-turn at the end of the test course. As **Figure 2B** shows, a tarpaulin was laid down in the area of the road designated for the U-turn, and the subjects were instructed to make a U-turn on the tarpaulin and then stop.

The study included daytime and nighttime trials, and the results in sections A, B, D, F, and G were analyzed under

conditions of natural light (i.e., no additional light). C and E included artificial light in the tunnel, so they were excluded from the analysis. The average vehicle speed in CVD (section D) was 99.9 ± 10.2 km/h in daytime and 98.0 ± 8.6 km/h at night, showing that the constant vehicle speed was maintained. The average times for driving each section of the course were: acceleration, 17.1 ± 0.3 s in daytime and 17.4 ± 0.3 s at night; CVD, 36.1 ± 0.6 s in daytime and 36.0 ± 0.6 s at night; deceleration, 41.6 ± 1.1 s in daytime and 40.1 ± 1.9 s at night; and U-turn, 28.9 ± 1.0 s in daytime and 31.9 ± 1.0 s at night. There were no significant differences between day and night in any part of the course. While the vehicle was parked, data for 21 s before starting were analyzed.

The subjects performed 1–3 practice drives for the daytime and nighttime trials before putting on the fNIRS probes. The number of practice drives was determined by the subjects themselves, who were asked after each practice drive whether they wanted any more practice. The experimenter accompanied the subject on each practice drive, and practice sessions were added if the CVD velocity was not stable. None of the subjects needed more than three practice drives.

#### **EXPERIMENTAL EQUIPMENT AND MEASUREMENT PROCEDURES** *Experimental vehicle*

A van ("Hiace," made by the Toyota Motor Corporation; ordinary vehicle classification, with super long, high roof specifications) was used in the experiment (**Figure 3A**). It is a two-wheel drive, gasoline-powered vehicle with four-speed automatic transmission. A global positioning system receiver and a vehicle speed pulse counter were installed for recording position, speed and acceleration.

#### *fNIRS apparatus*

A multi-channel fNIRS system (FOIRE-3000, Shimadzu Corporation) was mounted in the vehicle and used to measure hemodynamic responses (**Figure 3B**). The equipment irradiated three wavelengths of NIR light (780, 805, and 830 nm) to the cerebral cortex, and monitored changes in the hemoglobin (Hb) concentrations. Sampling intervals for measuring changes in the concentration of Hb were set to 70 ms. Conversions from absorbance to Hb concentration changes were performed in the apparatus using the method of Matcher et al. (1995), and measurements were performed in continuous mode. Location-related triggers were entered by an experimenter sitting in the passenger seat, at points indicating the beginning of each section of the course.

The fNIRS device was securely attached to the vehicle using two bars installed behind the driver's seat and a hook on the vehicle floor. The probe line was also attached to the bars behind the driver's seat. Power was supplied to the fNIRS equipment by an DC/AC inverter connected to the battery of the vehicle. To prevent noise due to sunlight, the front and back of the device, where the photomultiplier tubes are located, was covered with black cloth, and the subject also wore a black hood after the probes were attached to the head (**Figure 3C**).

and a graph of the driving task and vehicle speeds in each section

U-turn.

**FIGURE 3 | The experimental vehicle and the fNIRS apparatus. (A)** The experimental vehicle. **(B)** fNIRS equipment mounted in the vehicle. The equipment was secured to two bars attached to the vehicle, indicated by red arrows. **(C)** The driver's seat. A hood covering the subject's head was removed for the photograph. Probes were attached to the subject's head in a way that allowed for moderate changes in driving posture.

#### *Channel arrangement and registration*

The measurement areas were located on both sides of the prefrontal cortex, motor cortex, and parietal cortex. To ensure the safety of the experiment, we did not measure the occipital lobe. 48 channels were set up using 16 irradiation probes and 16 detection probes (**Figure 4A**). The distance between the irradiation and detection probes was 3 cm. The head attachment was placed so that the center of the frontmost row was 3.5 cm above the nasion.

When attaching the probes, first, the attachment was mounted on the subject's head, and then the hair was spread out carefully, to expose the skin under each probe. The probes were adjusted one by one so that they pressed the skin surface with a very slight force. The setup was tested to confirmed that there was no interference due to hair or improper contacts, and that the quantity of light received was suitable for data measurement. **Figure 4B** shows an example of raw data. As shown, the probe settings were carried out with the utmost care to provide the best possible signal-to-noise ratio over all 48 channels. No spiky motion artifacts or artifacts due to vehicle vibration were visually observed.

Positioning of the measurement points was confirmed by MRI with the subject wearing the probe attachment cap fitted with registration markers. MRI was conducted using 3D-T2 weighted

images with a 3 Tesla MRI (Philips Co., Achieva 3.0 Quasar Dual 3.0T-MRI). The sampling conditions used the spin-echo method, TE 247 ms, TR 2700 ms, image size 250 × 250 pixels, slice thickness 1.0 mm in the sagittal direction, with an inter-slice gap of 0 mm. The positions of the probes for all the sites measured were confirmed based on the locations of the registration markers (**Figure 4C**). **Figure 4D** shows the Brodmann's areas (BA) having the highest correspondence with each channel, from the MRIs of all the subjects. The area measured in the prefrontal cortex covered BA10, BA9, BA8, and BA46. Measurement of BA4 and BA6 in the motor-related areas, and BA3, 7, and 40 in the parietal lobe was also confirmed.

#### **ANALYSIS**

#### *Index*

We analyzed changes in oxyhemoglobin (-O) and deoxyhemoglobin (-D) as well as changes in cerebral blood volume (-CBV) and cerebral oxygen exchange (-COE), both of which were calculated from -O and -D. The relationship between -O, -D, -CBV, and -COE is described below based on a secondary square matrix (Yoshino and Kato, 2012).

$$
\begin{pmatrix}
\Delta O + \Delta D \\
\end{pmatrix} = \begin{pmatrix}
\text{l} & \text{l} \\
\end{pmatrix} \begin{pmatrix}
\Delta O \\
\Delta D
\end{pmatrix} = \begin{pmatrix}
\Delta \text{CBV} \\
\Delta \text{COE}
\end{pmatrix} \tag{1}
$$

$$
\begin{pmatrix} \Delta O\\ \Delta D \end{pmatrix} = \frac{1}{2} \begin{pmatrix} 1 \ -1\\ 1 \ 1 \end{pmatrix} \begin{pmatrix} \Delta \text{CBV} \\ \Delta \text{COE} \end{pmatrix} \tag{2}
$$


$$
\Delta \text{CBV} = \frac{(\Delta D + \Delta O)}{\sqrt{2}} \tag{3}
$$

$$
\Delta COE = \frac{(\Delta D - \Delta O)}{\sqrt{2}}\tag{4}
$$

#### *Data processing and statistics*

The -D and -O data were subjected to low-pass filtering at 0.1 Hz to remove any high frequency components. -COE and -CBV were calculated using these data. For each of these four indicators, average changes per second for each section of the course (excluding C and E) were determined for each channel. In this process, changes within each section were calculated by first setting the level to zero at the beginning of the section. A total of 35 daytime trials and 36 nighttime trials were analyzed. One daytime trial was excluded because the data did not record correctly.

Analysis of variance with *post-hoc* multiple comparison tests (Scheffe's) were performed to compare average change values for each index for the different sections of the course (at rest, acceleration, constant velocity speed, deceleration and U-turn), for all the sites. These tests were applied separately to the daytime and nighttime data. To investigate differences between the daytime and nighttime experiments, an independent *t*-test was performed for each part of the experiment. The level of significance was set at 5%.

#### **RESULTS**

#### **CHANGES IN** *-***O AND** *-***D DURING ACTUAL DRIVING**

**Figure 5** shows functional images from the prefrontal cortex for each section of the experiment. They show that the areas activated changed according to the driving operation. The functional imaging results showed high reproducibility in the daytime and nighttime experiments, despite the fact that they were performed on different days.

#### **ACCELERATION AND DECELERATION**

**Figure 6A** shows functional images of -CBV data during acceleration and deceleration. **Table A1** (Appendix) shows the results of the comparison of -CBV between the parts of the experiment. In the bilateral BA7 and BA40, the medial BA3, and part of the medial BA6 (Ch 33), -CBV increased significantly more during acceleration than during deceleration in the daytime. In the nighttime experiment, the significant differences in BA7 and BA40 were reproduced, but the significant differences in BA3 and BA6 disappeared. At night, -CBV increased significantly more in the right BA46 and the right BA9 (Ch 17, 23), compared to when the vehicle was parked. There were no sites where -CBV increased significantly more during deceleration than during acceleration, either in the daytime or at night.

**Figure 6B** shows functional images of -COE during acceleration and deceleration. **Table A2** (Appendix) shows the results of comparison of -COE between the parts of the experiment. In the daytime, the areas with significant -COE increases during acceleration were the left BA46 and part of the right BA6 (Ch 28). The increase in -COE in the left BA46 was reproduced in the night experiment. In addition, -COE increased significantly at night in part of the right BA6 (Ch 23), part of the right BA9 (Ch 18), and the right BA8. In these same sites, -COE increased in the daytime; these increases were not significant, but indicate reproducibility between the daytime and nighttime experiments during acceleration. In addition, both in the daytime and at night, -COE decreased significantly over a wide range in BA7 during acceleration, compared with other parts of the experiment (Appendix, **Tables A2**, **A3**).

In contrast, areas where -COE increased significantly more during deceleration than during acceleration were the medial and the left BA8, the medial and part of the left BA6 (Ch 29, 33), and part of the left BA9 (Ch 20). These significant differences were observed both in the daytime and at night. Also, in the medial BA3 in the daytime, -COE increased more during deceleration than during acceleration (Appendix, **Tables A2**, **A3**).

**FIGURE 5 | Functional neuroimages of** *-***O and** *-***D during actual driving.** Functional imaging of -O **(A)** and -D **(B)** from the prefrontal cortex (Ch 1-27), during all parts of the experimental task. Dotted lines on the images

indicate the boundaries of the Brodmann areas shown in **Figure 4D**. The images show that increased activation occurred in the same areas in the daytime and nighttime experiments.

**FIGURE 6 | Functional neuroimages of** *-***CBV and** *-***COE from actual driving, during acceleration and deceleration.** Functional imaging of -CBV **(A)** and -COE **(B)** from the prefrontal cortex (Ch 1-27), the motor-related

areas (Ch 28-36), and the parietal areas (Ch 29-48), during acceleration and deceleration. Note that the prefrontal images and the motor-related and parietal images are reversed on the left and right.

#### **U-TURNS AND CVD**

**Figure 7A** shows functional images of -CBV data during CVD and U-turns. **Table A1** (Appendix) shows the results of the comparison of -CBV between the parts of the experiment. In the daytime, -CBV increased significantly more during U-turns than during parking in the right BA10 and part of the medial BA7 (Ch 47). At night, the significant difference in the right BA10 was reproduced. -CBV also increased significantly more during Uturns than during parking in the right BA46 and part of the left BA10 (Ch 10) at night.

**Figure 7B** shows functional images of -COE. **Table A2** shows the results of comparison of -COE between the parts of the experiment. In the daytime, -COE increased significantly more during U-turns than during acceleration in the prefrontal cortex, in the medial and left BA8, part of the left BA6 (Ch 29), and part of the left BA9 (Ch 20). Among these areas, only the change in the left BA8 was reproduced at night. In the motor-related areas, -COE increased significantly in the medial BA3 in the daytime, and this was reproduced at night. In the nighttime experiment, there was a significant increase in -COE across both sides of BA4. In the parietal lobe, -COE increased significantly in the right BA7 and part of the left BA7 (Ch 48) in the daytime. This response was also spread widely across both sides and the medial BA7 at night (Appendix, **Tables A2**, **A3**).

During CVD, there were no areas where -CBV increased significantly more than in other parts of the experiment in either the daytime or nighttime experiments. -COE, however, increased significantly more during CVD than during acceleration in the medial, the left BA8, part of the left BA6 (Ch 29), part of BA9 (Ch 20), and part of the right BA7 (Ch 43). Among these areas, the changes in the left BA8, part of the left BA6 (Ch 29), and part of the right BA7 (Ch 43) were reproduced at night.

#### **CO-OCCURRENCE OF OXYGEN RESPONSES IN THE AREAS MEASURED**

The results described in ACCELERATION AND DECELERATION and U-TURNS AND CVD are summarized in **Table 1**. Increased -COE levels indicate hypoxic changes in blood vessels in the pixels measured. The co-occurrence of these increases in -COE indicate areas recruited during specific driving behaviors.

BA8 was involved in all 4 driving operations, and was detected with the highest levels of significance of all the areas measured, followed by BA7, BA6, and BA9. Among the 4 driving operations, deceleration and U-turns activated more brain areas than acceleration and CVD.

#### **FRONTAL EYE FIELD FUNCTION DURING DRIVING**

**Figure 8** shows changes in -COE in both sides of the frontal eye field (FEF, BA8), which showed the most involvement in the driving operations. Differences in laterality were observed during acceleration and deceleration. -COE increased in the right BA8 during acceleration, and in the left BA8 during deceleration (Appendix, **Table A4**). Reproducibility in BA8 between day and night was high, with no significant differences.

#### **DISCUSSION**

The results of the present study suggest that fNIRS may be an effective technique for evaluating brain activity for the purpose of obtaining objective feedback that can be useful in the construction or improvement of road structures. We were able to measure brain activation during actual driving on an expressway using a multichannel fNIRS system mounted in a vehicle, where we found that the activated areas varied according to the driving operation. A previous fNIRS study investigated actual driving at 30–50 km/h using two channels (NIRO-300) (Harada et al., 2007), but only two sites were measured in the frontal cortex, and localization in the frontal cortex was not clearly shown. The present experiment is the first to image brain functional localization associated with actual expressway driving.

We invested considerable time in preparing the subjects for the experiment and paid particular attention to the safety of the experiment, because we thought that driving while wearing fNIRS probes might induce changes in heart rate and global activation from the autonomic nervous system that would mask localized function related to driving. This did not happen, and the results we obtained showed statistically significant differences in brain functional localization while driving, and showed a high degree of reproducibility in the daytime and nighttime experiments. The ability to perform actual road experiments using fNIRS is potentially advantageous in field studies for intelligent transport systems (ITS). In the future, fNIRS may provide a means for re-evaluating conventional highway



*Numbers in parentheses are channel numbers. Areas without channel numbers recorded increases in more than one channel.*

design, and creating highways that apply less stress to the brain.

#### **CORTICAL ACTIVATION DURING ACTUAL EXPRESSWAY DRIVING**

U-turns and deceleration mobilized more of the brain sites we measured than did acceleration and CVD. During U-turns, there were -COE increases in BA3, BA7, and BA4, and these increases were more robust during nighttime driving. This result may reflect the association between motor function and visual processing required during the manipulation of the foot pedals and the steering wheel (Uchiyama et al., 2012). The significant differences observed in BA3 (the primary somatosensory area) and BA4 (the primary motor area) at night during U-turns were not found in the other parts of the experiment. This may be explained by poor visibility at night for the complex manipulations of the foot pedals and the steering wheel required to back and turn the vehicle when making a U-turn.

A localized increase in -COE and a slight increase in -CBV occurred around the medial and the left BA8 during deceleration. This is a situation in which increased blood supply accompanies oxygen metabolism, showing enhanced FEF function. The fact that activation in the parietal lobe increased more during deceleration also suggests that deceleration induces a wider range of brain activation than acceleration.

During acceleration, -COE increased markedly around the right BA8 and in the left BA46 in the prefrontal cortex. The dorsolateral prefrontal cortex, which includes BA46, is considered to be

related to attention control during executive functions (Shallice, 1982). In the present study, the dorsolateral prefrontal cortex may have been involved with internal monitoring to reach the target speed within the distance provided. In BA7 (the superior parietal lobule), -COE increased in sections of the course other than acceleration, whereas a global change characterized by a strong decrease in -COE and a strong increase in -CBV was observed during acceleration. This response during acceleration, in which only blood volume increased with almost no increase in oxygen metabolism due to neural activity (Yoshino and Kato, 2012), suggests a decline in activation in the superior parietal lobule. Thus, BA8 and BA46 were activated at the same time during acceleration, just as BA8 and BA7 were simultaneously activated in other parts of the experiment. BA46 is involved in the generation of eye movements, and BA7 is involved in visual attention processing that accompanies eye movement (Goldberg et al., 1991). Thus, it can be hypothesized that acceleration requires attention control and generation of eye movements under the motivation of the dorsolateral prefrontal cortex for navigation, while visual attention processing in the superior parietal lobule is reduced. Indeed, in order to reach the target speed of 100 km/h within the short distance of 261 m, the subjects had to depress the accelerator pedal almost to full throttle. In a merging situation on an expressway, possible new traffic safety measures may be based on the assumption that although drivers have high internal control with respect to their goal, their processing of external visual stimuli may be reduced. This suggests that in other parts of the experiment, activity of the superior parietal lobule is increased, by the requirement of greater spatial perception than that needed during acceleration, and by the motor skills required to manipulate the brake, the accelerator, or the steering wheel.

There was no strong activation area detected uniquely in the CVD section of the course, although significant activation around BA8 was detected, as in the rest of the course. There was little need for handling operations on the CVD section of the course, because it was a comfortable road with a brand new surface. We considered the level of brain activation to be low because only minimal movements were required to maintain the car's speed. Thus, when one is simply maintaining vehicle speed on an expressway, one is likely to be driving with low awareness. The above results may be part of the physiological explanation for low awareness driving.

#### **FRONTAL EYE FIELD FUNCTIONS DURING EXPRESSWAY DRIVING**

The FEF (BA8) is the area that was activated in all parts of the experiment. It is known to be involved in voluntary eye movements, unlike the primary visual cortex in the occipital lobe (Fukushima et al., 2000; Pierrot-Deseilligny et al., 2004). The FEF controls two movements: pursuit eye movements from side to side, and vergence eye movements responsible for depth perception (Gamlin and Yoon, 2000). Vergence is required more in expressway driving, because it is almost straight with unobstructed views, as was this experimental course. It has been reported that the FEF is particularly important during the formation of eye movement signals in three-dimensional space (Fukushima et al., 2002). The detection of increased oxygen metabolism in the FEF suggests that we were able to capture activation associated with voluntary eye movement control in three-dimensional space. Involvement of the occipital and parietal regions in simulation experiments has been reported in several studies (Walter et al., 2001; Horikawa et al., 2005; Spiers and Maguire, 2007; Uchiyama et al., 2012), but there have been few reports of localized activation in the FEF during driving. In a visual simulation experiment without driving operations using fMRI (Graydon et al., 2004), FEF activation was detected in a task that required pushing a button in response to visual cues displayed in a video of actual expressway driving as seen from the driver's seat. The present study resembled that study in that it involved an actual expressway and not artificial road images. We believe our results detected activation of the FEF during three-dimensional processing or high speed visual processing.

Indeed, based on the -COE increases we observed, activation of the FEF was stronger during deceleration and acceleration than during CVD or U-turns. We can hypothesize that the greater the change in speed, the more strongly the FEF is activated. The fact that the FEF was more strongly activated during the deceleration and acceleration parts of the course than during CVD, where there were rapid changes in the field of view at 100 km/h, suggest a relationship between the magnitude of acceleration and the activation of the FEF. In fact, MRI imaging has been reported to show that a professional driver participating in Super GT races at 300 km/h had a bulging FEF (Sumida, 2009). We plan to investigate the relationship between activation of the FEF and the speed of acceleration in a future driving study.

We do not know whether this FEF activity is a response specific to the driver of a vehicle, or whether similar activity occurs in the brain of a person in the passenger seat, and this is an interesting subject. It seems possible that when a person sits the passenger seat and pays the same kind of visual attention as the driver, the same brain activity in the FEF may be detected in the passenger. This kind of control experiment was not performed in this driving experiment, because of the difficulty of reproducing the exact same vehicle behavior, to control the field of vision, with the same subject in the driver's seat and the front passenger seat. This subject may be better approached in a driving simulation study.


#### **BRAIN FUNCTIONAL INDICES FOR OUTDOOR EXPERIMENTS**

There are many possible effects on the autonomic nervous system in dynamic outdoor experiments, so they require an index (or indices) that can accurately differentiate local activation from global change. When brain function is evaluated based on increases in -O, it is difficult to differentiate functional changes from effects of the autonomic nervous system and skin blood flow (Takahashi et al., 2011; Kirilina et al., 2012). Functional localization was poor in an earlier driving simulation experiment using only -O (Yanaginuma et al., 2007). In contrast, increases in -D can be used to precisely identify sites of neural activity (Ances, 2003). Small -D increases such as the initial dip have been difficult to detect, but -D increases are likely to occur with higher load tasks (Rupp and Perrey, 2008), and their reproducibility is also high, as we found in the present study.

In recent years, analytical approaches have been proposed that use concentration changes in more than one Hb index to facilitate more precise functional diagnosis (Wylie et al., 2009; Yoshino and Kato, 2012; Sano et al., 2013). -COE, which was used in this study, is similar the subtraction index of -O and -D that has frequently been used as an index of oxygenation in fNIRS muscle studies. The situation inside the blood vessels as it reflects neural activity has two components, oxygen metabolism and blood volume, and they are both determined by the balance between -D and -O (Kato, 2004). This means that the use of a single index is likely to give rise to misdiagnosis. In fact, in the present study, -CBV increased markedly in the prefrontal cortex during U-turns and in the parietal areas during acceleration, but these increases were not accompanied by increase in -COE. It is highly possible that responses of this kind are merely increases in blood volume, unaccompanied by oxygen metabolism from neural activity. Evaluation of brain functional activation using -COE can be expected to provide new insights in driving studies, which have been previously been performed using BOLD and -O, because dynamic signals can be obtained even outdoors. Research that has potential social application, such as in measures for highway safety, urgently requires a reconsideration of indices and analytical methods.

#### **FURTHER STUDY**

The present study clarified some brain regions of interest in the context of traffic safety measures. In particular, it highlighted the importance of the FEF during actual highway driving. There are many visual stimuli on an expressway, such as road signs and lights, and it is possible that devising visual stimuli that would promote or alleviate activation in the FEF could play a part in implementing road safety measures. The study also suggested that the brain functional activation was low during CVD, and this suggests a need for road safety measures for avoiding driving with low awareness. In addition, more areas were activated during deceleration than during acceleration, suggesting that the coordination of more parts of the brain is required for deceleration. In the future, evaluation systems such as that used in this study may provide new suggestions for building safer roads.

Our analysis was focused on the changes during each part of the course, and we did not consider baseline changes over the entire course. There may be differences in baseline levels between indoor simulation experiments and actual driving experiments, when there is likely to be more tension. In the future, it will probably be necessary to compare results of driving simulation tests and actual road tests using the same course and the same indices, to determine the physiological load attributable to actual road driving relative to the baseline level. This type of comparison should elucidate differences between simulation experiments and experiments performed outside the laboratory on actual roads, and help to advance our understanding of brain function in the field of ITS.

Other possible subjects for additional study with larger numbers of subjects include investigation of age and gender differences in brain activity related to driving. There were fewer females than males in this study, not enough to calculate parametric differences and withstand reproducibility, and gender differences were not compared. Also, for the purpose of detecting underlying fundamental data, the subjects were made comfortable with the experimental environment through test runs and practice, and so brain activity when driving in unfamiliar situations was not a part of this study. Because mental tension likely increases on an unfamiliar road, and accidents can be caused by a driver's inability to predict the shape of the road ahead, this is another important subject for investigation.

#### **AUTHOR CONTRIBUTIONS**

Kouji Yamamoto and Hideki Takahashi conceived the intelligent transport systems (ITS) part of the study, and Toshinori Kato designed the neuroimaging part of the study. Kayoko Yoshino performed the experiments and analyzed the data together with Noriyuki Oka; Kouji Yamamoto, and Hideki Takahashi provided valuable help with ITS; and Kayoko Yoshino and Toshinori Kato co-wrote the paper. All the authors discussed the results and commented on the manuscript.

#### **ACKNOWLEDGMENTS**

This study was supported by Nagoya Electric Works Co., Ltd., who provided the environmental maintenance of the experimental course and guarding personnel to ensure safety. We would like to give special thanks to Patricia Yonemura for English editing.

#### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

#### *Received: 29 September 2013; accepted: 03 December 2013; published online: 24 December 2013.*

*Citation: Yoshino K, Oka N, Yamamoto K, Takahashi H and Kato T (2013) Functional brain imaging using near-infrared spectroscopy during actual driving on an expressway. Front. Hum. Neurosci. 7:882. doi: 10.3389/fnhum.2013.00882*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Yoshino, Oka, Yamamoto, Takahashi and Kato. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### **APPENDIX**


**Table A1 | Channels with significant differences in** *-***CBV between parts of the task according to a** *post-hoc* **test.**

*Ch, channel.*


**Table A2 | Channels with significant differences in** *-***COE between parts of the task in the daytime experiment according to a** *post-hoc* **test.**

*Ch, channel.*


**Table A3 | Channels with significant differences in** *-***COE between parts of the task in the daytime experiment according to a** *post-hoc* **test.**

*(Continued)*

#### **Table A3 | Continued**


*Ch, channel.*

#### **Table A4 |** *-***COE in the left and right FEF.**


*Differences were significant during acceleration and deceleration.*

## Correlation of prefrontal cortical activation with changing vehicle speeds in actual driving: a vector-based functional near-infrared spectroscopy study

#### *Kayoko Yoshino1, Noriyuki Oka1, Kouji Yamamoto2, Hideki Takahashi <sup>3</sup> and Toshinori Kato1 \**

*<sup>1</sup> Department of Brain Environmental Research, KatoBrain Co., Ltd., Tokyo, Japan*

*<sup>2</sup> Department of Environment/Engineering, Tokyo Branch, Central Nippon Expressway Co., Ltd., Tokyo, Japan*

*<sup>3</sup> Department of Environment/Engineering, Central Nippon Expressway Co., Ltd., Nagoya, Japan*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Yukiori Goto, Kyoto University, Japan Hirokazu Doi, Nagasaki University, Japan*

#### *\*Correspondence:*

*Toshinori Kato, Department of Brain Environmental Research, KatoBrain Co., Ltd., 13-15-104, Shirokanedai 3, Minato-ku, Tokyo, 108-0071, Japan e-mail: kato@katobrain.com*

Traffic accidents occur more frequently during deceleration than during acceleration. However, little is known about the relationship between brain activation and vehicle acceleration because it has been difficult to measure the brain activation of drivers while they drive. In this study, we measured brain activation during actual driving using vector-based functional near-infrared spectroscopy. Subjects decelerated from 100 to 50 km/h (speed reduction task) and accelerated from 50 to 100 km/h (speed increase task) while driving on an expressway, in the daytime and at night. We examined correlations between average vehicle acceleration in each task and five hemodynamic indices: changes in oxygenated hemoglobin (oxyHb), deoxygenated hemoglobin (deoxyHb), cerebral blood volume (-CBV), and cerebral oxygen exchange (-COE); and the phase angle *k* (degrees) derived from the other hemoglobin (Hb) indices. oxyHb and -CBV reflect changes in cerebral blood flow, whereas deoxyHb, -COE, and *k* are related to variations in cerebral oxygen metabolism. Most of the resulting correlations with specific brain sites, for all the indices, appeared during deceleration rather than during acceleration. Faster deceleration resulted in greater increases in deoxyHb, -COE, and *k* in the prefrontal cortex (*r* < −0.5, *p* < 0.01), in particular, in the frontal eye field, and at night, it also resulted in greater decreases in oxyHb and -CBV in the prefrontal cortex and in the parietal lobe (*r* > 0.4, *p* < 0.01), suggesting oxygen metabolism associated with transient ischemic changes. Our results suggest that vehicle deceleration requires more brain activation, focused in the prefrontal cortex, than does acceleration. From the standpoint of the indices used, we found that simultaneous analysis of multiple hemodynamic indices was able to detect not only the blood flow components of hemodynamic responses, but also more localized frontal lobe activation involving oxygen metabolism.

**Keywords: actual driving, supplementary eye field, outdoor brain activation, acceleration, deceleration, interregional correlation, phase angle, vehicle acceleration**

#### **INTRODUCTION**

In recent years, neuroscience research related to vehicle driving has become popular. Brain function during driving encompasses responses to external stimuli such as visual stimulation and vehicle acceleration, in addition to internal processing involved in functions such as motor control and decision making. Functional neuroimaging during driving examines multiple intricate factors such as perception, cognition, thinking, and motor functions. It is important for road safety measures in the Intelligent Transport Systems (ITS) field to separate these factors and identify the regions of interest in the cerebral cortex for each of them.

Kato et al. (2013) and Yoshino et al. (2013) demonstrated that areas of the brain that are activated during acceleration were significantly different from those activated during deceleration in actual road experiments. That study suggested that brain activation during deceleration is higher than activation during acceleration; and this led us to investigate in more detail the relationship between brain activity and vehicle acceleration. The most common cause of traffic accidents between vehicles in Japan is the rear-end collision (approximately 38.5%) (Institute for Traffic Accident Research and Data Analysis, 2011). Rear-end collisions are caused by deceleration that occurs too late. It is possible that this is related to differences in brain function between acceleration and deceleration. However, no studies have yet examined brain responses during vehicle acceleration and deceleration in actual road driving.

Since there has been very little investigation of brain activity during actual road driving, the kind of responses that can be expected is still unclear. We thus decided to use fNIRS vector-based analysis, which is capable of explaining all the possible variations in the ratios of concentration changes in oxyHb and deoxyHb (Kato, 2006, 2007). This is a method that displays compositely the hemodynamic response dues to changes in both oxyHb and deoxyHb on the same vector coordinate plane. Oxygenation and blood volume changes reflecting neural activity cannot be properly evaluated using the conventional analysis of either oxyHb or deoxyHb alone. In vector-based analysis, a cerebral blood volume (CBV) axis and a cerebral oxygen exchange (COE) axis are generated from an oxyHb and deoxyHb orthogonal coordinate plane, and the phase of vectors on this vector plane are evaluated (Yoshino and Kato, 2012; Sano et al., 2013), increasing the possible indices of brain activity. We thought it would be possible to use this method to evaluate brain activity during actual expressway driving from various perspectives.

We therefore investigated the relationship between brain regions and hemodynamic indices that increased or decreased in relation to calculated vehicle acceleration (m/s2) during tasks, using vector-based functional near-infrared spectroscopy (fNIRS) in a vehicle in an actual road experiment. This study aimed to extract brain responses and indices that are related to vehicle acceleration and deceleration by recording changes in cortical hemodynamic responses during driving by normal adults both during daytime and at night.

### **METHODS**

#### **SUBJECTS**

Right-handed twelve healthy adults participated in this study (eight males and four females; average age, 33.3 ± 4.5 years). The subjects had operational experience on expressways and ordinary roads on a daily basis. The subjects had no history of mental or central nervous illnesses, and they took no medications on the day of the experiment. The subjects were comfortable in the experimental situation because they had previously participated in other actual expressway driving research (Yoshino et al., 2013) while wearing the fNIRS system probes. Written consent was obtained from the participants before enrollment in the study, and the protocol was approved in advance by the ethics committee at KatoBrain Co., Ltd. The subjects' average length of driving history was 11.8 ± 5.8 years. Their frequency of driving was 6.1 ± 1.6 times / week, and their frequency of expressway driving was 4.5 ± 6.5 times / month. Only two subjects had experienced an accident (neither accident involved another vehicle or any personal injuries), and the average number of accidents was 0.2 ± 0.4. The average number of traffic violations was 1.3 ± 1.3 times, mostly for speeding. Since recruitment of the subjects was based on the conditions of age, right-handedness, and frequency of driving on a daily basis, the subjects' genders, their driving histories, and their histories of violations and accidents were completely random.

#### **EXPERIMENTAL FIELD AND TEST VEHICLE**

The experiment was performed in the Okitsu district, Shizuoka Prefecture, Japan, on a section of the Shin Tomei Expressway immediately before it entered service (Yamamoto et al., 2012; Kato et al., 2013). Installation of signage, lighting and so on had already been completed, and so there were no problems with the safety of vehicle travel. For further safety, no vehicles were present other than the test vehicle on the experimental course. Guard personnel were located at each point on the experimental course, and they could immediately contact the test vehicle with a transceiver in any unexpected contingencies.

The experimental course was 2875 m long. It was almost straight, but included a gentle left and right curve (*R* = 5000, both). The slope of the course was almost flat, but there were uphill and downhill gradients of 2.0%. The test vehicle traveled in the left lane (3.75 m in width) in accordance with Japanese traffic regulations for lane use in two lanes. There was no artificial lighting provided on the experimental course, either in the daytime or at night.

An ordinary van, *Hiace*, which is made by the Toyota Motor Corporation (Japan) and is super-long with a high-roof specification, was used in this experiment. The vehicle was a twowheel-drive, gasoline-powered vehicle with a four-speed automatic transmission. A global positioning system receiver and a vehicle speed pulse counter were attached to the test vehicle, to record information on vehicle position, speed, and acceleration. Power was supplied to the fNIRS equipment by connecting a DC/AC inverter to the battery of the vehicle.

#### **EXPERIMENTAL PROCEDURE**

The tasks included a speed reduction task of deceleration from 100 to 50 km/h and a speed increase task of acceleration from 50 to 100 km/h. The speed increase task was performed immediately after the speed reduction task (**Figure 1**). One trial consisted of these two driving tasks, and six trials each were performed in the daytime and at night. Start and stop positions were provided on the course, and the subjects performed the two tasks at their own pace, with no cues on the course. After receiving their instructions and before putting on the fNIRS probes, the subjects performed 1–3 practice drives for the daytime and nighttime trials. The day and night experiments were performed on different days and the order of the experiments (day or night) was randomized between the subjects. The average duration of the speed increase task (acceleration) was 16.8 ± 3.3 s, and the average duration of the speed reduction task (deceleration) was 21.1 ± 8.0 s. The average ratio of acceleration time to deceleration time for all the subjects was 0.90 ± 0.30.

#### **fNIRS MEASUREMENTS AND REGISTRATION**

A multichannel fNIRS system (FOIRE-3000, Shimadzu Corporation, Japan) was mounted in the vehicle and used to measure hemodynamic responses. The equipment irradiated three wavelengths of NIR light (780, 805, and 830 nm) to the cerebral cortex, and monitored changes in the hemoglobin (Hb) concentrations. Sampling intervals for measuring changes in Hb concentration were set to 70 ms.

**FIGURE 1 | Experimental tasks.** Each subject performed 6 daytime trials and 6 nighttime trials on the experimental course. The tasks included a speed reduction task of deceleration from 100 to 50 km/h and a speed increase task of acceleration from 50 to 100 km/h.

The fNIRS device was tightly secured to the vehicle using two bars installed behind the driver's seat and a hook on the floor of the vehicle. The probe line was also attached to the bars behind the driver's seat. Probes were attached to the subject's head in a way that allowed for moderate changes in driving posture. To prevent noise due to sunlight, the front and rear of the device, and the subject's head were covered with black cloth after the probes were attached.

Measurement areas were located on both sides of the prefrontal cortex, and on the motor cortex and the parietal cortex; the occipital lobe was excluded for the safety of the subjects (**Figure 2A**). Forty-eight channels were set up using 16 irradiation and 16 detection probes. The distance between irradiation and detection probes was 3 cm.

Confirmation of the position of each measurement point was done by magnetic resonance imaging (MRI), using a 3-Tesla 3D-T2-weighted MRI system (Philips Co., Achieva 3.0 Quasar Dual 3.0T-MRI), in which the subject probe attachments were fitted with registration markers. The sampling conditions used the spin-echo method with an echo time of 247 ms, a repetition time of 2700 ms, image size of 250 × 250 pixels, a slice thickness of 1.0 mm in the sagittal direction, and an interslice gap of 0 mm. As **Figure 2B** shows, the positions of the probes were confirmed for all of the measured areas based on the locations of the registration markers. **Figure 2C** shows the Brodmann areas (BA) having the highest correspondence with each channel, from the MRIs of all the subjects.

#### **ANALYSIS**

#### *The vector approach*

In the blood vessels, changes in oxygenation and blood volume occur in response to neural activity, and changes in the concentrations of both oxyHb and deoxyHb are involved in this response. To detect changes in both oxygenation and blood volume, the following vector analysis method was used.

OxyHb and deoxyHb have different chemical properties (paramagnetic or diamagnetic) that are due to differences in the bonding of oxygen molecules (Pauling and Coryell, 1936). Taking this into consideration, an orthogonal vector coordinate plane is set up, defined by oxyHb (-*O*) and deoxyHb (-*D*) axes (Kato, 2006, 2007). As **Figure 3** shows, rotating this -*O*/-*D* vector plane 45 degrees counterclockwise results in an orthogonal vector coordinate plane comprising a (-*O* + -*D*) axis and a (-*D* − -*O*) axis. A vector (-*O* + -*D*) can be defined as a cerebral blood volume (-CBV) vector, and a vector (-*D* − -*O*) can be defined as a cerebral oxygen exchange vector (-COE). A positive value for -CBV indicates locally increasing -CBV and a negative value

**FIGURE 2 | Measurement sites and the locations of the probes. (A)** Head probe attachments and channel numbers. **(B)** Positioning of the probes was confirmed using MRI. **(C)** Dashed curves represent the

schematic boundaries of the Brodmann areas that were confirmed by MRI. Straight lines connecting the emitters and detectors indicate channel positions.

for -CBV indicates decreasing -CBV. A positive value of -COE indicates hypoxic change from -COE = 0, and a negative value of -COE indicates hyperoxic change. The relationship among the four axes of -*O*, -*D*, -CBV, and -COE is described by the following square matrix:

$$
\begin{pmatrix} \Delta O + \Delta D\\ -\Delta O + \Delta D \end{pmatrix} = \begin{pmatrix} \text{l} & \text{l} \\ -\text{l} \text{ l} \end{pmatrix} \begin{pmatrix} \Delta O\\ \Delta D \end{pmatrix} = \begin{pmatrix} \Delta \text{CBV} \\ \Delta \text{COE} \end{pmatrix} \tag{1}
$$

$$
\begin{pmatrix} \Delta O\\ \Delta D \end{pmatrix} = \frac{1}{2} \begin{pmatrix} 1 \ -1\\ 1 \ 1 \end{pmatrix} \begin{pmatrix} \Delta \text{CBV} \\ \Delta \text{COE} \end{pmatrix} \tag{2}
$$

These can be expanded to obtain -CBV and -COE as follows.

$$
\Delta \text{CBV} = \frac{(\Delta D + \Delta O)}{\sqrt{2}} \tag{3}
$$

$$
\Delta \text{COE} = \frac{(\Delta D - \Delta O)}{\sqrt{2}} \tag{4}
$$

The phases (octants) on the vector plane described above provide a quantitatively defined representation of the degree of oxygen exchange; the phase of a vector is defined by the angle *k* (degrees), which is a ratio of -D to -O, and it reflects the strength of oxygen metabolism. *k* is the angle between a vector and the positive -O axis, and it is determined as follows:

$$k = \operatorname{Arc} \tan \left(\frac{\Delta D}{\Delta O}\right) \tag{5}$$

$$= \operatorname{Arc} \tan \left(\frac{\Delta \text{COE}}{\Delta \text{CBV}}\right) + 4 \mathbf{5}^{\circ} \quad (-135^{\circ} \le k \le 225^{\circ})$$

*k* = 0◦ is on the positive -*O* axis, and coincides with the oxygen density of arterial blood. An increase in *k* is defined within the range of increase in -*D* or -COE (0◦ *k* - 235◦), and a decrease in *k* is defined within the range of decrease in -*D* and -COE (−135◦ *k* - 0◦). An increase or decrease in *k* thus indicates a change in oxygen demand.

#### *Data processing and statistics*

The -*D* and -*O* data were subjected to low-pass filtering at 0.1 Hz to remove any high frequency components. For each of these two indicators, the average changes per second among the tasks were determined for each channel. -COE, -CBV, and *k* were calculated using these data. In this process, the actual changes for each task were calculated by setting the levels to zero at the beginning of each task.

Average vehicle acceleration (m/s2) during each task was determined by Equation (6). Positive values indicate average vehicle acceleration during speed increase tasks, and negative values indicate average vehicle acceleration during speed reduction tasks.

$$\text{Average vehicle acceleration} \tag{6}$$

$$\mathbf{=} (\text{initial velocity} - \text{final velocity})/\text{time}$$

Analyses of correlations (Spearman's rank correlation coefficient) between average vehicle acceleration and each hemodynamic index were performed. Analyses of interregional correlations (Spearman's rank correlation coefficient) within each hemodynamic index were also performed. These tests were applied separately to each of the daytime and nighttime experiments. The data used for analysis was a total of 70 trials in the daytime and 70 trials in nighttime. Two trials each in the daytime and nighttime experiments were excluded because Hb monitoring was not successful. The significance level was set to 5%. Correlation coefficients that were higher than ±0.4 were evaluated.

#### **RESULTS**

#### **AVERAGE VEHICLE ACCELERATION**

**Table 1** shows average vehicle acceleration during the two tasks. Negative values indicate deceleration. There were no significant differences between daytime and nighttime for either of the tasks.

#### **CORRELATIONS IN THE VEHICLE SPEED REDUCTION TASK**

**Figure 4** shows mapping images of the correlations between average vehicle acceleration and each of the hemodynamic indices in the speed reduction task. The results divide into significant negative and positive correlations according to the indices used. Significant negative correlations were observed for -COE, -*D*, and *k* (*p* < 0.01). Significant positive correlations were observed for -CBV and -*O* (*p* < 0.01).

#### *Negative correlations in the vehicle speed reduction task*

**Table 2** shows correlations of −0.4 or lower in the speed reduction task. The greater the deceleration, the more -COE, -*D*, and *k* increased. These correlations were observed only in the frontal lobe (BA10, BA9, BA8 and BA6), and none were observed in the parietal lobe.

In the speed reduction task, high negative correlations were observed between -COE and vehicle acceleration in the medial BA8 (daytime: *r* = −0.545, *p* < 0.001; nighttime: *r* = −0.670, *p* < 0.001). In the nighttime results, in addition to the negative correlations observed in the medial BA8, negative correlations increased in the peripheral regions of the left BA8 (Ch. 26), the left BA9 (Ch. 20), and the left BA6 (Ch. 29) (−0.622 < *r* < −0.613, *p* < 0.001). These correlations during the nighttime tasks were also observed with -*D* and *k* (-*D* : −0.661 < *r* < −0.543, *p* < 0.001; *k* : −0.596 < *r* < −0.431, *p* < 0.001). Negative correlations with vehicle acceleration were also observed with -COE and *k* at night in the medial BA10 (Ch. 3) and in the left BA10 (Chs. 4 and 9) (-COE: −0.698 < *r* < −0.604, *p* < 0.001; *k* : −0.614 < *r* < −0.539, *p* < 0.001).

#### **Table 1 | Average vehicle acceleration.**


*n.s., not significant.*

#### *Positive correlations in the speed reduction task*

**Table 3** shows correlations 0.4 or more in the speed reduction task. The greater the deceleration, the more -CBV and -*O* decreased. Activation occurred during the daytime tasks mainly on the periphery of the medial BA8 (*r* = 0.464, *p* < 0.001); and during the nighttime tasks, in the medial and the left BA8 and the area surrounding them (0.436 < *r* < 0.654, *p* < 0.001) and in both the right and left BA10 (0.486 < *r* < 0.674, *p* < 0.001). Positive correlations were also observed for -CBV in both the right and the left BA10 at night (0.422 < *r* < 0.601, *p* < 0.001).

#### **CORRELATIONS IN THE SPEED INCREASE TASK**

**Figure 5** shows mapping images of the correlations between average vehicle acceleration and each of the hemodynamic indices in the speed increase task. **Table 4** shows the correlations of 0.4 or more in this task.

This task did not divide into positive and negative correlations depending on the index used. There were no correlation coefficients greater than ±0.5 in the speed increase task, except for -CBV in BA46. The only reproducible correlations found in either the daytime or nighttime tasks were in the left BA46. In the left BA46 (Ch. 5), -*O* demonstrated a negative correlation with vehicle acceleration in both the daytime and nighttime tasks (daytime: *r* = −0.465, *p* < 0.001; nighttime: *r* = −0.412, *p* < 0.001). The more rapid the acceleration, the more -*O* decreased in the left BA46.

#### **INTERREGIONAL CORRELATIONS WITH THE SUPPLEMENTARY EYE FIELD**

**Figure 6** shows mapping images of the correlations between responses in the supplementary eye field (medial BA8, where there was a high correlation with vehicle acceleration in both the


**Table 2 | Negative correlations in the speed reduction task between vehicle acceleration and the brain activation indices.**

**Table 3 | Positive correlations in the speed reduction task between vehicle acceleration and each of the brain activation indices.**


daytime and nighttime tasks) and responses in the other areas, during the speed reduction task. **Table 5** shows the number of channels with correlations of 0.4 or more with the medial BA8 (*r* > 0.4), by index.

Interregional correlations with *k* were identified in 4–6 channels (daytime and nighttime tasks). For the other Hb indices, the number of correlations with the medial BA8 ranged from 11 to 23 channels. Particularly for -*O* and -CBV, localization was poor.

For *k* in the daytime speed reduction task, positive correlations with the medial BA8 were identified in the left BA8 (Ch. 26; *r* = 0.415), the left BA6 (Ch. 29; *r* = 0.657), the medial BA10 (Ch. 3; *r* = 0.415), and the left BA10 (Ch. 4; *r* = 0.446) (*p* < 0.01). These correlations were higher at night (0.477 < *r* < 0.741). Correlations also occurred in the area surrounding the left BA10 (Ch. 4; *r* = 0.630) and in BA9 (Ch. 20; *r* = 0.533) (*p* < 0.01).

#### **DISCUSSION**

#### **RELATIONSHIP BETWEEN CHANGING VEHICLE SPEEDS AND BRAIN ACTIVATION**

Correlations between the hemodynamic indices and vehicle acceleration were found to be higher in the speed reduction task, and lower in the speed increase task. In the speed reduction task, there were negative correlations between vehicle acceleration and the indices -COE and *k* (that is, -COE and *k* showed greater increases during faster deceleration). Because -COE and *k* are indicators of change in oxygen metabolism, this suggests that oxygen metabolism increased during rapid deceleration of the vehicle.

Areas that typically exhibited greater increases in oxygen metabolism during vehicle speed reduction were BA8 and its surrounding area (including BA9 and BA6), which are involved in eye movement (Fukushima et al., 2000, 2002; Pierrot-Deseilligny et al., 2004). The field of view is narrower when the vehicle is driven at a high speed, while it spreads gradually, approaching a steady state when the vehicle speed decreases. BA8 controls sideto-side eye movements, and vergence eye movements responsible for depth perception (Gamlin and Yoon, 2000). The element of


*\*Correlations of r* < −*0.4 were all significant at p* < *0.01.*

vergence eye movements is particularly important in fast vehicle traveling, but a more balanced ratio between the two types of eye movements would be required in slow vehicle traveling. This suggests that a spread in the direction of eye movement control occurs along with the spread in the field of view that occurs during rapid deceleration, and the activation of BA8 possibly increases at that time. In contrast, a possible reason for the low correlation between vehicle acceleration (positive acceleration) and the activation in BA8 during the vehicle speed increase task is that the acceleration of the vehicle results in a narrowing of the field of view, and the direction of the control direction easily becomes fixed in one direction.

Changes in vision enter the driver's brain as movement of the optic flow. It can be hypothesized that the increase in oxygen metabolism in BA8 results from the optic flow that is derived from changes in vehicle speed. The relationship between BA8 and the optic flow may be a key point in future research, as one of the regions of interest in the neuroimaging research on the driver's brain.

During nighttime driving, there were more correlations in the area surrounding BA8 than during daytime driving, and there were also more correlations between vehicle deceleration and oxygen metabolism in BA10, which is involved in executive function. During rapid deceleration at night, enhanced attention, or increased awareness of one's visual field and adjustments to one's eye movements may be more necessary than in the daytime. BA10 is known to be activated more in dual tasks (Baddeley and Della Sala, 1996). Vehicle deceleration at night requires control of the vehicle speed on a dark road without artificial lighting, and

**FIGURE 5 | Correlation maps showing correlations between vehicle acceleration and results from each of the hemodynamic indices in the speed increase task.** There were no striking correlations.

**Table 4 | Correlations in the speed increase task between vehicle acceleration and each of the brain activation indices (***p <* **0***.***01).**


*\*Correlations of r* < −*0.4 were all significant at p* < *0.01.*

in addition, the useful field of view at night is narrower than in the daytime. It is thus likely that increased involvement of BA10 in deceleration at night occurs because the driver's attention must be allocated to a greater extent to both visual and motor control. Additional differences between daytime and nighttime driving may also exist in sites that were not included in our measurements.

#### **INTERREGIONAL CONNECTIVITY AND THE DIFFERENT BRAIN ACTIVATION INDICES**

Interregional correlations between brain activation in BA8 and BA10 in both the daytime and at night were shown by the angle *k*. Correlations with vehicle acceleration, however, were observed in BA8 only during the day and in both BA8 and 10 at night. This suggests the possibility of a functional connectivity between BA8 and 10 regardless of the presence or absence of a correlation with vehicle acceleration. Anatomically, in addition to BA8, the frontal eye field also includes BA6 and BA9 (Goldberg et al., 1991). This could explain why correlations were higher with the areas surrounding BA8: BA6 (Ch. 29) and BA9 (Ch. 20). BA10 has been considered to be a separate area from BA8, and the detection of functional connectivity between these areas is thus a new finding. Further investigation of this point will be necessary.

The present study demonstrated that the identification of relationships between local brain activation differs according to the hemodynamic index used. Localized interregional relationships were shown by the angle *k* (degrees), which reflects variation in oxygen metabolism. The relationships shown by -CBV and -*O*, however, covered most of the prefrontal cortex. Hemodynamic responses in the blood vessels contain an oxygen metabolic component and a blood flow component. Differences in -CBV reflect the blood flow factor, and differences in -COE reflects the oxygen metabolism factor. In other words, a hemodynamic index must include both indicators that reflect oxygen metabolic relationships and indicators that reflect vascular relationships. These factors are determined by the ratio of the variations in deoxyHb and oxyHb, and this is the ratio *k* on which the concept of phase is based. In the results of this study, the use of -CBV and -*O* reflected vascular relationships, and -COE and *k* were likely reflected oxygen metabolic relationships.

To date, fNIRS studies have frequently been conducted using time series correlations of oxyHb to detect functional connectivity (Homae et al., 2010). Increases in oxyHb, however, are related to the inflow of arterial blood, and thus do not necessarily reflect

**FIGURE 6 | Correlation maps showing interregional correlations between responses in the supplementary eye field (∗), where there were high correlations with vehicle acceleration during in both daytime and**

**nighttime trials, and the responses in other areas, in the speed reduction task, using** *k* **and** *-O***.** *k* showed localized areas of correlation, while -*O* showed broad correlations in the prefrontal cortex.

**Table 5 | Number of channels showing positive correlations with the medial BA8 (***r >* **0***.***4).**


increases in neural activity (Kato, 2004). Consequently, because these correlations based on oxyHb cannot distinguish actual activation from vascular networks in the scalp or the brain surface, the possibility that vascular relationships have been overestimated as functional connectivity cannot be denied. Skin blood flow are likely to be incorporated into measurements of oxyHb (Takahashi et al., 2011; Kirilina et al., 2012), and thus the use of phase, based on the angle *k*, has been proposed as a solution (Sano et al., 2013). In future research, the presence of differences in functional brain imaging based on these indices will need to be addressed as a technical and physiological issue.

Furthermore, in the present study we examined correlations using variation per second within the tasks, and this is another issue that it will be necessary to reconsider, with a view toward time series correlations in the future.

#### **THE SIGNIFICANCE OF NEUROSCIENCE FINDINGS IN ITS**

Since research on the actual expressway driving is still in its infancy, the brain scientific findings are obviously still insufficient in the field of ITS as it relates to expressway construction and management. In particular, there is little basic knowledge about influences on the brain from the behavior of the vehicle in actual highway driving. There has been a simulation experiment on expressway driving using fMRI (Graydon et al., 2004), but it does not include the actual physical driving operations, and the field of view is different from that in actual road conditions. From the viewpoint of actual safety measures and expressway construction and management, the authors believe that the observation of brain activity in actual road experiments is essential.

As indicated in the introduction, it is a well-known fact that more accidents occur during deceleration than during acceleration, but the physiological reasons for this are not known. Our findings show that brain activation involved in voluntary eye movement control and in executive function is likely to increase while the driver is decelerating rapidly, especially at night. As these kinds of findings accumulate, it is possible that traffic safety measures can be developed that would help to increase or decrease brain activation during deceleration, by such means as redesign of deceleration lanes, or warning systems in places where sudden deceleration is likely to occur.

The correlations we found between prefrontal cortex activity and vehicle acceleration, do not necessarily imply a causal relationship. Time-course analysis will be required to clarify many issues in the relationship between the human brain and vehicle operation, including causality.

### **CONCLUSION**

This study demonstrated that prefrontal cortical activation increased with faster deceleration during actual road driving. This means that strong brain activation is required in situations when a driver has to brake rapidly. If the driver's prefrontal cortex does not work well during vehicle deceleration, the risk of accident may be increased. We also found that localized prefrontal cortical activation can be detected with good reproducibility by the simultaneous analysis of multiple hemodynamic indices with vector-based fNIRS, which makes it possible to detect both oxygen metabolic relationships and vascular relationships in the evaluation of brain activation.

#### **ACKNOWLEDGMENTS**

We thank Nagoya Electric Works Co., Ltd. who provided environmental maintenance for the experimental course and guard personnel to ensure safety. We would like to give special thanks to Patricia Yonemura for English editing.

#### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 06 October 2013; accepted: 09 December 2013; published online: 25 December 2013.*

*Citation: Yoshino K, Oka N, Yamamoto K, Takahashi H and Kato T (2013) Correlation of prefrontal cortical activation with changing vehicle speeds in actual driving: a vector-based functional near-infrared spectroscopy study. Front. Hum. Neurosci. 7:895. doi: 10.3389/fnhum.2013.00895*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Yoshino, Oka, Yamamoto, Takahashi and Kato. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Prefrontal cortex activation during story encoding/ retrieval: a multi-channel functional near-infrared spectroscopy study

#### **Sara Basso Moro<sup>1</sup> , Simone Cutini <sup>2</sup> , Maria Laura Ursini <sup>1</sup> , Marco Ferrari <sup>1</sup>\* and Valentina Quaresima<sup>1</sup>**

<sup>1</sup> Department of Life, Health and Environmental Sciences, University of L'Aquila, L'Aquila, Italy

<sup>2</sup> Department of General Psychology, University of Padua, Padova, Italy

#### **Edited by:**

Nobuo Masataka, Kyoto University, Japan

#### **Reviewed by:**

Hidenao Fukuyama, Kyoto University, Japan Ursula Wolf, University of Bern, Switzerland

#### **\*Correspondence:**

Marco Ferrari, Department of Life, Health and Environmental Sciences, University of L'Aquila, Coppito 2, Via Vetoio, 67100, L'Aquila, Italy e-mail: marco.ferrari@univaq.it

Encoding, storage and retrieval constitute three fundamental stages in information processing and memory. They allow for the creation of new memory traces, the maintenance and the consolidation of these traces over time, and the access and recover of the stored information from short or long-term memory. Functional nearinfrared spectroscopy (fNIRS) is a non-invasive neuroimaging technique that measures concentration changes of oxygenated-hemoglobin (O2Hb) and deoxygenated-hemoglobin (HHb) in cortical microcirculation blood vessels by means of the characteristic absorption spectra of hemoglobin in the near-infrared range. In the present study, we monitored, using a 16-channel fNIRS system, the hemodynamic response during the encoding and retrieval processes (EP and RP, respectively) over the prefrontal cortex (PFC) of 13 healthy subjects (27.2 ± 2.6 years) while were performing the "Logical Memory Test" (LMT) of the Wechsler Memory Scale. A LMT-related PFC activation was expected; specifically, it was hypothesized a neural dissociation between EP and RP. The results showed a heterogeneous O2Hb/HHb response over the mapped area during the EP and the RP, with a O2Hb progressive and prominent increment in ventrolateral PFC (VLPFC) since the beginning of the EP. During the RP a broader activation, including the VLPFC, the dorsolateral PFC and the frontopolar cortex, was observed. This could be explained by the different contributions of the PFC regions in the EP and the RP. Considering the fNIRS applicability for the hemodynamic monitoring during the LMT performance, this study has demonstrated that fNIRS could be utilized as a valuable clinical diagnostic tool, and that it has the potential to be adopted in patients with cognitive disorders or slight working memory deficits.

**Keywords: verbal working memory, encoding, retrieval, cortical oxygenation, functional near-infrared spectroscopy, prefrontal cortex**

#### **INTRODUCTION**

Encoding, storage and retrieval constitute three fundamental stages in information processing and memory. Encoding is the initial elaboration of input data, that allows the creation of new memory traces; storage refers to the maintenance and the consolidation of new memory traces over time, while retrieval refers to the process of accessing and recovering stored information from short or long-term memory (Buckner and Koutstaal, 1998; for a review see Cabeza and Nyberg, 2000). Short-term memory allows the temporary storage of a limited quantity of information previously encoded (Atkinson and Shiffrin, 1968), and the processes of active maintenance and rehearsal allow to not forget the information and to hold them for a short period of time (from few seconds to hours). A wide range of complex cognitive activities as reasoning, language comprehension, planning, and spatial elaboration, requires the combination of the short-term storage and the manipulation of the information, processes that, taken together, are defined as working memory (WM; Baddeley, 2012). Indeed, WM is a core component of human cognition, being an essential part of the mnemonic processes, and fundamental for many cognitive activities (Baddeley and Hitch, 1974; Petrides, 1989; Fletcher and Henson, 2001). Among the several theories regarding the cognitive structure and functioning of WM, the Baddeley's model (Baddeley and Hitch, 1974) is likely to be the most influential one. Such model holds that WM is based on a supervisory system, the central executive, and three subsystems, each one specialized in the maintenance and manipulation of different types of information. The phonological loop deals with verbal information; the visuo-spatial sketchpad with spatial and visual information; the episodic buffer with episodic information (for a review see Baddeley, 2012). Other major theories of WM (e.g., Kane et al., 2001; Cowan, 2005; Oberauer, 2009) underline the importance of the attentional role in controlling the activation, maintenance, and manipulation of short-term internal representations. So, rather than a short-term store, WM is considered as a limited-capacity attentional system, that interacts both with perception and with long-term memory, allowing the construction of new representations (Oberauer, 2009). Indeed, attention and WM are increasingly viewed as overlapping constructs (Kane et al., 2001; Postle, 2006; Gazzaley and Nobre, 2012). Functional imaging has provided considerable evidence about the neural correlates of WM processes, showing that they reside in prefrontal cortex (PFC) (for review see D'Esposito et al., 2000; Fletcher and Henson, 2001). In particular, ventrolateral prefrontal cortex (VLPFC) is more often activated during tasks requiring maintenance (left VLPFC seems to be engaged in the maintenance of verbal information, and right VLPFC in the maintenance of spatial information), while dorsolateral prefrontal cortex (DLPFC) is more often activated during tasks requiring manipulation (for a review see Fletcher and Henson, 2001). This is consistent with Petrides' model (Petrides, 1989), which states that VLPFC initially receives and organizes the information, whereas DLPFC is additionally recruited only when monitoring and manipulation of information within WM is required. Imaging studies have also supported the dissociation between storage and rehearsal. The DLPFC and the anterior frontal regions would be associated with executive control of WM, as well as manipulation processes on the information already maintained in memory; the anterior frontal regions seem to be associated with maintaining the goals and products of one task while performing another (Fletcher and Henson, 2001). From a neural perspective, encoding and retrieval processes share some cortical structures (Rugg et al., 2008), even though it is still debated to what extent their neural circuitries overlap, and how much they depend on their own specific features, such as the type of encoded and retrieved material (Kelley et al., 1998). For instance, the encoding-related cortical activity seems to reflect the cognitive load elicited by the task, thus depending on the nature of the online processing demands (Otten and Rugg, 2001; Uncapher et al., 2006). Moreover, it seems that when the neural activity elicited during retrieval engages the same processing involved during encoding, a more successful retrieval is performed (Craik, 2002; Rugg et al., 2008). Encoding and retrieval can be experimentally investigated by presenting subjects different kinds of stimuli (such as words, pictures or narratives), that have to be learned and subsequently recalled (Cabeza and Nyberg, 2000). While words and pictures have been extensively used in memory research, much less studies have employed narratives. A narrative presentation is the description of a sequence of actions or events that follow one another over time, on the basis of causal principles (Graesser et al., 1980). Both narrative comprehension and production involve a number of identical cortical areas, including medial and dorsolateral regions of the frontal cortex. Moreover, the activation pattern observed in narrative processing seems to be different from the activation pattern elicited in word recognition and production or sentence-level operations (Crozier et al., 1999; Cabeza and Nyberg, 2000; Robertson et al., 2000). In narrative comprehension, as reviewed by Mar (2004), some regions appear to be critical: the frontopolar cortex (FPC), that supports the maintenance of the information; the DLPFC, associated with temporal ordering and integration, processing of sequential information, and monitoring/manipulating the contents of WM; the VLPFC, that seems to play a role in specification and/or maintenance of cues for long-term retrieval and encoding. Finally, it should also be noted that frontal areas modulate the attentional system by enhancing the recruitment of other specific cortical areas (Fuster, 2002). Pertaining to narrative production, neuroimaging studies showed a smaller body of evidence. Activations have been found in medial and dorsolateral frontal gyri, temporoparietal junction, and the posterior cingulate (Braun et al., 2001). Importantly, one critical aspect in narrative comprehension and production tasks is the normal functioning of encoding and retrieval processes, because they enable to encode incoming stimuli, rehearse received input, assess and retrieves stored knowledge, thus allowing all WM operations (Bayles, 2003). Recent researches have examined the potential clinical use of oral narratives to identify language impairments, in pediatric psychiatric population (Pearce et al., 2013) or in people suffering of mild cognitive impairments (Shankle et al., 2005; Roark et al., 2011). Being able to validly assess communicative and linguistic abilities through standardized tests is important for the clinical diagnostic evaluation of patients with cognitive disorders, memory impairments or slight WM deficits. Moreover, the possibility to monitor the neural activation elicited by the task provides an important contribution in cognitive functioning comprehension.

In the present study, we used the Logical Memory Test (LMT) of the Wechsler Memory Scale (Wechsler, 1997) as a spoken language derived measure to investigate the encoding and retrieval processes (EP and RP, respectively). So far, the LMT has been widely used as a clinical assessment measure of WM, because it represents an index of auditory-linguistic memory, requiring the immediate and delayed verbal recall of auditorally presented three-sentence verbal narrative containing 24 mnemonic units (Abikoff et al., 1987). Here, we recorded the hemodynamic activity of PFC using functional near-infrared spectroscopy (fNIRS), a functional brain imaging technique that, similarly to functional magnetic resonance imaging (fMRI), monitors hemodynamic changes in the cerebral cortex (see Cutini et al., 2012a; Ferrari and Quaresima, 2012, for reviews). However, unlike the blood-oxygen-level-dependent (BOLD) signal of fMRI, which is gathered from the paramagnetic properties of deoxygenatedhemoglobin (HHb), fNIRS is based on the intrinsic optical absorption of blood. As a result, fNIRS can simultaneously record the variations of HHb and oxygenated hemoglobin (O2Hb) concentrations. We investigated, with a 16-channel fNIRS system, the temporal and spatial characteristics of the hemodynamic PFC activity in healthy subjects performing the LMT. We expected a LMT-related PFC activation, given the major involvement of the PFC in WM processing. Specifically, we hypothesized a neural dissociation between EP and RP, given: (i) the contributions of the VLPFC in the semantic maintenance of verbal information, in the narrative comprehension, and in the maintaining memory cues for long-term encoding, and (ii) the DLPFC role in monitoring and manipulating the memory traces, in processing the sequential information giving them a temporal order and integration.

#### **MATERIALS AND METHODS**

#### **PARTICIPANTS**

Thirteen healthy subjects (27.2 ± 2.6 years; high level of education) participated in the study. Only men were recruited to avoid possible gender differences in emotional responses. Informed consent was obtained after a full explanation of the protocol and the non-invasiveness of the study. To exclude left-handed subjects, all participants completed the Edinburgh Handedness Inventory (Oldfield, 1971) assessing hand dominance.

#### **EXPERIMENTAL SETUP**

#### **Functional near-infrared spectroscopy (fNIRS) instrumentation and signal processing**

A 16-channel continuous wave fNIRS system (Oxymon Mk III, Artinis Medical Systems, Netherlands) was employed to map changes in O2Hb and HHb over bilateral PFC. This device measures changes in light attenuation at two wavelengths, 764 and 858 nm, and utilizes the modified Beer-Lambert law with an age-dependent differential pathlength factor to resolve changes in O2Hb and HHb concentrations within cortical brain tissue. Six optical fiber bundles (length: 3.15 m; diameter: 4.5 mm) were utilized to carry out the light to the left and the right PFC (three for each hemisphere), whereas eight optical fiber bundles of the same size (four for each hemisphere) were utilized to collect the light emerging from the same cortical areas. The illuminating and collecting bundles were assembled into a specifically designed flexible probe holder ensuring that the position of the 14 optodes, relative to each other, was firmly fixed. The probe holder consisted of two mirror-like units (9.7 × 8.9 cm each) held together, along the longest side, by three flexible junctions. The detector–illuminator distance was set at 3.5 cm. The optodes were inserted into apolyoxymethylene probe holder by connectors. The probe holder was appropriately placed over the head in order to include the underlying PFC. In particular, the two frontopolar fibers bundles collecting light at the bottom of the holder were centered (according to the International 10–20 systems for the EEG electrode placement) at the Fp1 and Fp2 for right and left side, respectively. The MNI (Montreal Neurological Institute) coordinates of the optodes and the relative 16 measurement points (channels) were calculated using a probe placement method (Cutini et al., 2011) based on a physical model of the head surface of ICBM152 (Mazziotta et al., 2001) (the standard brain template in neuroimaging studies) and a 3D digitizing system (BrainSight™, Rogue Research). As a final step, the MNI coordinates of each channel on the right hemisphere have been averaged with their symmetrical ones in the left hemisphere (1–9, 2–10, 3–11, 4–12, 5–13, 6–14, 7–15, 8–16). This procedure has allowed us to provide a broad estimate of the average error made during the probe placement. Notably, the mean difference in absolute values between left and right channels was 3.4 mm, well below the spatial resolution achievable with the present setting. Afterwards, in order to identify the corresponding Brodmann Areas (BAs), the measurement points were overlaid onto the Brodmann template using MRIcron software.<sup>1</sup> The investigated BAs were then: BA 9 (measurement points: 1, 3, 9, 11), BA 46 (measurement points: 2, 4, 5, 10, 12, 13), BA 10 (measurement points: 6, 8, 14, 16), and BA 45 (measurement points: 7, 15).

The probe holder was fixed over the head by a velcro brand fastener, adapting it to the individual size and shape of the different heads. This flexible probe holder and its position on the head provided a stable optical contact with the scalp for all optodes. The accuracy of the contact between the optodes and the scalp was verified at the end of the protocol. The pressure created by the velcro brand fastener was adequate to induce a partial transient blockage of the skin circulation during the fNIRS study, as witnessed by the presence of the well-defined 14 circles over the PFC skin (depressed cutaneous areas associated with the location of the 14 optodes).

The 14 circles over the forehead skin started to disappear 15–20 min after the end of the protocol. The adopted procedure would suggest that a consistent reduction of forehead skin blood flow was occurring as a result of this approach (Takahashi et al., 2011).

The O2Hb and HHb data from the sixteen measurement points, which are defined as the midpoint of the corresponding detector–illuminator pairs, were acquired at 10 Hz. During the data collection procedure, O2Hb and HHb concentration changes were displayed in real time, and the signal quality and the absence of movement artifacts were verified. The coming out concentration changes in O2Hb and HHb, calculated according to a modified Beer-Lambert Law, were transferred online from the fNIRS system to a personal computer (OxySoft DAQ 2.1.6, Artinis Medical Systems, Netherlands). The additional quantification of the concentration changes (expressed in 1µM) of O2Hb and HHb was obtained by including an age-dependent constant differential pathlength factor (4.99 + 0.067\*age0.814) (Duncan et al., 1996). In order to remove the drift introduced either by the system or by any possible spontaneous baseline fluctuations over the protocol (Scholkmann et al., 2014), the time series data of O2Hb and HHb concentration changes were first detrended utilizing the function "detrend" of the commercial numerical and statistical package MATLAB (MathWorks, Natick, MA).

A 2-s moving-average filter was applied to attenuate cardiac signal, respiration, and Mayer-wave systemic oscillations. The subject's heart rate (HR) was monitored by a pulse oximeter (N-600, Nellcor, Puritan Bennett, St. Louis, MO) with the sensor clipped to the index finger of the left hand.

#### **Experimental design**

Prior to the study, subjects were informed about the procedures and familiarized with the protocol. Subjects were asked to sit on a comfortable chair in front of a 17<sup>00</sup> PC monitor placed at a distance of 70 cm. The schematic illustration of the experimental protocol is reported in **Figure 1**. Specifically, the protocol started with a 30-s baseline period, in which participants were asked to relax, in order to get stable baseline fNIRS signals. Then, visual instructions informed the participants that the 27-s story of the LMT (Wechsler, 1997) would be acoustically presented (EP). Immediately after the end of the recording, a beep-tone and a visual instruction alerted the subjects to repeat the story aloud trying to recall as many details as possible (RP). The RP lasted

<sup>1</sup>http://www.mccauslandcenter.sc.edu/mricro/mricron/index.html

30-s after which participants were acoustically instructed to relax until the end of the 2-min recovery period. About 35 min later, a second RP, preceded by a beep-tone and a visual instruction, occurred. Once again the subjects were asked to repeat the story aloud trying to recall as many details as possible. During the 34.5-min period, subjects were distracted by performing a visual modified version of the digit span test (Wechsler, 2008). Twomin after the end of the second RP, two control tasks were administered to the subjects: (1) a 27-s backward story for the EP (EP control, encoding process control (EPC)), and (2) a 30-s week day aloud sentence repetition for the RP (RP control, retrieval process control (RPC)). The backward story consisted in presenting the previous story transformed backward (Audacity® 2.0, the Free, Cross-Platform Sound Editor), while the sentence repetition consisted in repeating self paced and aloud the week days. The baseline and the recovery periods lasted 30 and 120 s, respectively; the interval between EPC and RPC was set at 3 min. The verbal responses were recorded for each subject who vocalized as softly as possible to reduce movement related artifacts but loud enough to convey the response to the experimenter and the recorder. A commercially available software package (SuperLab Pro Edition 4.5 Executable, Cedrus Corporation, Canada) was used to present visual and auditory stimuli and instructions.

At the end of the protocol, subjects completed the DP-15 rating scale for the perceived difficulty (Delignières, 1993), attributing to the LMT a value from 1 to 15 (where 1 corresponds to "very, very easy"; 6 to "easy"; 10 to "difficult"; 15 to "very, very difficult"). In order to evaluate the "state anxiety", the subjects completed 20-items of the State-Trait Anxiety Inventory (STAI)-Form Y-1 before and after the protocol (Spielberger et al., 1983).

#### **DATA ANALYSIS AND STATISTICS**

#### **Performance evaluation and heart rate (HR) data analysis**

The LMT score (performance) was obtained matching the results of the two RP, calculating the mean number of the two summary scores (the raw number of elements recalled) (Wechsler, 1997). The mean values of the HR changes during the study were calculated every 5 s. A one-way repeated measures analysis of variance (ANOVA) was performed in order to evaluate the influence of the time on HR changes.

#### **Functional near-infrared spectroscopy (fNIRS) data analysis**

The maximum values of the concentration changes in O2Hb and HHb over the PFC were obtained from the mean values, calculated every 5 s, of the EP/RP data after subtracting the respective control tasks (EPC/RPC). These resulting maximum values were baseline-corrected. The baseline was calculated as follows: (1) the mean value over the last 10 s of the baseline period for the EP and for the EPC/RPC; and (2) the mean value over the last 10 s of the EP for the RP. In order to investigate the PFC activation in response to the EP and the RP, a repeated measures ANOVA was performed for O2Hb/HHb maximum values. The ANOVA analysis included two factors: channel (8 levels) and hemisphere (2 levels). A one-way repeated measures ANOVA was performed to evaluate the influence of the channel on O2Hb/HHb maximum values during the EP/RP. Student's *t*-tests were conducted in order to evaluate the presence of any difference over the 16 measurement points between the O2Hb/HHb maximum values of the EP vs. the EPC and the RP vs. the RPC. In order to check for the presence of detectable PFC activation during the control tasks (EPC/RPC), a repeated measures ANOVA was performed for O2Hb/HHb maximum values. The ANOVA analysis included two factors: channel (8 levels) and hemisphere (2 levels).

The Pearson's correlation coefficient was calculated to evaluate the relation between: (1) individual performance and O2Hb/HHb changes (mean of the 16 measurement point maximum values) during the RP; and (2) the subjects' HR (mean of the EP/RP maximum values) and O2Hb/HHb changes (mean of the 16 measurement point maximum values) during the EP/RP.

All statistical analyses were conducted with SPSS 20.0 (SPSS Inc., Chicago, IL). Data were expressed as mean ± SD. The criterion for significance was *p* < 0.05.

#### **RESULTS**

The behavioral data analysis revealed the following main results. The recalled elements of the LMT were 25 ± 10, falling within the equivalent range of 4, considered as a normal performance. The mean subjective rating of perceived difficulty during the LMT was 6.9 ± 2.5, suggesting that the task was considered by the participants as "somewhat difficult". The difference in the "state anxiety" before (31.0 ± 6.8) and after the protocol (28.0 ± 4.8) was not significant (*t* = 1.304, *p* = 0.205). HR started to increase 15 s after the beginning of the EP, reaching its maximum value (around 125% of baseline) 10 s after the beginning of the RP. Then, HR progressively declined, returning to the baseline value 5 s after the end of the RP (**Figure 2**). The ANOVA carried out on

panels refer to the cerebral projections of the measurement points superimposed on the ICBM152 template brain. The points have been created

to the previous value.

mean ± SD). \* p < 0.05 with respect to the baseline. <sup>N</sup> p < 0.05 with respect

**the subtraction of O2Hb and HHb changes occurring during the**

two vertical dotted lines limit the end of the EP and the beginning of the RP. (n = 13; mean ± SD).

HR changes, revealed a significant main effect of time (*F*(36,432) = 17.188, *p* < 0.001).

The fNIRS data evidenced a heterogeneous O2Hb/HHb response over the mapped area during both the EP and the RP (**Figure 2**). O2Hb progressively increased since the beginning of the EP in most of the measurement points, then progressively decreased in the second part of the EP; this O2Hb increase was more evident in the measurement points 7 and 15, corresponding to the VLPFC (**Figure 3**). Since the beginning of the RP, O2Hb increased consistently and progressively in most of the measurement points up to 10–15 s after the end of the RP. Then, O2Hb progressively decreased returning to the corresponding baseline values within the end of the considered recovery time. During the LMT, HHb changes were smaller than O2Hb changes.

The ANOVA analysis, carried out on O2Hb changes during the EP, revealed a significant main effect of the channel (*F*(3.16,37.87) = 3.177, *p* = 0.033). The ANOVA analysis carried out on HHb changes during the EP revealed no significant main effects nor interaction (all *ps* > 0.05). The series of channel-wise *t*-tests for O2Hb revealed a significant difference in channels 1, 7, and 15 in the comparison between EP and EPC. In particular, the channels 7 and 15 showed the highest *t*-value (*t* = 2.742, *p* = 0.018; *t* = 2.717, *p* = 0.019). The *t*-tests were limited to O2Hb because of the lack of significant effects in the ANOVA for HHb.

The ANOVA analysis, carried out on O2Hb changes during the RP, revealed a significant main effect of the channel (*F*(7,84) = 4.82, *p* < 0.001). The ANOVA analysis, carried out on HHb changes during the RP, revealed a significant main effect of the channel (*F*(3.43,41.20) = 2.858, *p* = 0.042). The series of channel-wise *t*-tests for O2Hb revealed a significant difference in channels 2, 3, 4, 5, 7, 8, 10, 12, 13, 14, 15, and 16 (*ps* < 0.05) in the comparison between RP and RPC. For HHb, the channel-wise *t*-tests revealed a significant difference in channel 9 only (*t* = 2.595, *p* = 0.023).

The ANOVA analysis carried out on O2Hb/HHb changes during the control tasks (EPC/RPC) revealed no significant main effects nor interaction (*ps* > 0.05).

No correlation was found between subjects' performance and the RP PFC changes in O2Hb/HHb (*r* = 0.543, *p* = 0.055; *r* = −0.249, *p* = 0.412, respectively). No correlation was found between subjects' HR and the corresponding EP/RP PFC changes in O2Hb (*r* = 0.336, *p* = 0.262; *r* = −0.403, *p* = 0.172, respectively) and in HHb (*r* = 0.045, *p* = 0.885; *r* = 0.312, *p* = 0.300, respectively).

#### **DISCUSSION**

To the best of our knowledge, this is the first time in which the LMT has been utilized in a fNIRS neuroimaging study. The hemodynamic results evidenced a moderate, but focused, activation in the bilateral VLPFC (measurement points 7 and 15) during the EP (**Figure 3**), and a broader activation in the bilateral VLPFC, DLPFC and FPC during the RP (**Figures 2**, **3**). These findings support our hypotheses and are consistent with the previous neuroimaging studies in which PFC is considered as the neural substrate of WM (D'Esposito et al., 2000; Fletcher and Henson, 2001). In particular, the prominent activation found in channels 7 and 15, corresponding to the right and left VLPFC, is in line with its role in the selection, comparison, or decision about the information held in memory (Petrides, 1989), and in the specification and/or maintenance of cues for long-term encoding (Mar, 2004). The VLPFC recruitment could be explained by the evidence that many cognitive processes are required in speech comprehension: both the semantic representations induced by the narrative stimulus, and the strategic/executive/control processes that are required to access, retrieve, compare and manipulate semantic information. Speech is a serial dynamic auditory signal that needs to be integrated through time, and this is especially true for narratives. Moreover, computing the correct meaning requires the selection from multiple competing representations of speech sounds that may have the same or similar sound. In the same way, the selection from the competing representation of speech sounds needs a constant monitoring on the contextual information (Price, 2012). Notably, speech production and speech comprehension share a considerable amount of cognitive operations (Papathanassiou et al., 2000; Price, 2012), consistently with the broader PFC activity found during RP, including VLPFC. The recruitment of the DLPFC and the FPC, other than the VLPFC, could be explained by the role of DLPFC in the monitoring and manipulation of the information, while the FPC activation could be explained by its role in constructing a global coherence, in particular to get the sense of the story from the encoded information (Mar, 2004). Interestingly, the correlation between the number of recalled elements during the first RP and the O2Hb/HHb changes was very close to reach significance, strongly suggesting the presence of a task-related activation. In fact, even though HR changes revealed a significant effect of the time, suggesting a possible effect of the task in the subjects' anxiety and hemodynamic perfusion, no difference was found in the "state anxiety" before and after the protocol. Moreover, no correlation was found between HR and O2Hb/HHb changes (for both EP and RP), showing that the HR and the hemodynamic changes where different in their time courses (**Figure 2**).

Notably, the PFC activation in response to WM tasks has been widely investigated with fNIRS, both in healthy subjects and in patients (for reviews see Cutini et al., 2012a; Ehlis et al., 2014). For instance, during the execution of a visual *n*-back task utilizing task-relevant and task-irrelevant faces, Schreppel et al. (2008) found that relevant stimuli activated the middle frontal/pre-central cortices and left post-central cortex bilaterally, while irrelevant stimuli activated superior, middle and inferior parts of the right PFC. Such pattern is consistent with the recruitment of a verbal rehearsal strategy to maintain the features of the relevant stimuli and with the selective inhibition needed to properly perform the task, respectively. This is in line also with other neuroimaging studies which demonstrated that PFC activity during WM tasks reflects processes of maintenance, selection and inhibition of information, as well as attentional monitoring (Fletcher and Henson, 2001). On top of that, it has been shown that the hemodynamic response in the PFC and in the temporal regions during an *n*-back WM task may act as a biological marker of social functioning in patients suffering of late onset depression (Pu et al., 2011). WM deficits and PFC dysfunction has also been found in patients with major depressive disorder (Pu et al., 2012). Moreover, Koike et al. (2013) have compared the cortical activation of patients that suffer of schizophrenia with healthy subjects activations during the execution of an *n*-back task, observing in the first a bilateral DLPFC and FPC activation, and in the second a bilateral VLPFC activation accompanied with a task-related deactivation in the DLPFC. Similarly, previous fNIRS studies have shown activation patterns in the VLPFC and DLPFC regions in healthy volunteers performing *n*-back tasks (Ehlis et al., 2008; Pu et al., 2011). To date, the only fNIRS study investigating PFC activity in specific relation to encoding and retrieval processes has been conducted by Okamoto et al. (2011). In particular, they investigated the cortical activity during encoding and retrieval in episodic memory processing of taste information. A cortical activation during taste retrieval was found significantly stronger than that observed during encoding in the bilateral FPC and right DLPFC regions, particularly in the right hemisphere.

The combination of the neuropsychological tests with fNIRS is particularly useful since it enables to monitor the hemodynamic task-related responses while the tests are simultaneously performed. Although the LMT has not been applied yet in the clinical diagnostic field in combination with fNIRS monitoring, it has the potential to be effectively adopted in patients with cognitive disorders and/or WM deficits. For example, Niu et al. (2013) have shown that WM and cognitive abilities, as well as functional deficits in frontal and temporal cortices, can be found in patients with mild cognitive impairment. The importance of identifying these cognitive deficits in the early stages of disease has been widely pointed out (e.g., Lecardeur et al., 2013; Pfeiffer et al., 2013). More specifically, narrative recall tasks, such as the LMT, are included in most standard neurological examinations, and provide a solid approach to elicit semi-structured spontaneous language data, as well as quantity (number of recalled elements) and quality (accuracy and coherence of the retellings) of information. Narrative recall ability is associated with a variety of neurodegenerative and developmental disorders, and it is considered a good predictor of a variety of cognitive and developmental problems as language impairments (Botting, 2002; Dodwell and Bavin, 2008; Duinmeijer et al., 2013), autisms (Tager-Flusberg, 1995; Diehl et al., 2006), dementia (Gomez and White, 2006; Roark et al., 2011), as well as a useful tool to improve the accuracy in the detection of, for example, mild cognitive impairment (Lehr et al., 2012).

Crucially, fNIRS provides neuroscientists with new possibilities for cortical investigations, given its very high experimental flexibility with respect to other neuroimaging methods. Compared with fMRI, fNIRS can simultaneously record the variations of HHb and O2Hb concentrations, with a higher temporal resolution, potentially providing a more detailed picture of cortical hemodynamics (e.g., Brigadoi et al., 2012; Cutini et al., 2012b, 2014; Szu˝cs et al., 2012). Furthermore, fNIRS is silent, more tolerant to subtle movement artifacts (for instance overt speech is allowed), it allows long-time continuous measurements and repeated measurements within short intervals, and offers the possibility to monitor the cortical activity in natural experimental settings.

Nevertheless, the fNIRS technique presents also some limitations that have been previously discussed (Dieler et al., 2012; Quaresima et al., 2012; Scholkmann et al., 2014). The task-evoked changes occurring in forehead skin perfusion could represent an overestimation of the cortical changes, as measured by fNIRS. Recent reports have raised a question against the assumption that PFC O2Hb/HHb changes originated only from the cortical hemodynamic response (Kohno et al., 2007; Gagnon et al., 2011, 2012; Takahashi et al., 2011; Kirilina et al., 2012). Such taskevoked changes could result either from systemic blood pressure changes or from skin-specific regulation mechanisms different from the HR autonomic control. In the present study, the subject's HR time course showed a different pattern in comparison with the time course of O2Hb/HHb changes (**Figure 2**). However, forehead skin perfusion changes would have occurred during the LMT. This potential confounder has been previously investigated by others measuring simultaneously fNIRS signals and forehead skin flow (by a laser Doppler meter) during cognitive tasks (Kohno et al., 2007; Takahashi et al., 2011; Kirilina et al., 2012; Funane et al., 2014). Unfortunately, the costly laser Doppler skin flow meter is not widely available in most of the laboratories, as in the case of the laboratory in which the present study has been carried out. Although several instrumental and/or analysis methods have been proposed to partly account for extracerebral hemodynamic trends in fNIRS signals (Kirilina et al., 2012, 2013; Hallacoglu et al., 2013; Funane et al., 2014), no consensus has been reached yet on the best strategy to be adopted in order to minimize this effect and/or separate superficial and cortical fNIRS responses. Very recently, Kirilina et al. (2013) have proposed a de-noising method that significantly improves the sensitivity of fNIRS to cerebral signals; in that study they combined concurrent time-domain fNIRS and peripheral physiology recordings (mean arterial blood pressure, HR, and skin blood flow) with wavelet coherence analysis. Depth selectivity was achieved by analyzing moments of photon time-of-flight distributions provided by time-domain fNIRS. In the future, the possibility to use time-domain fNIRS systems, combined with peripheral physiology recordings, would offer the advantage to eliminate the extracerebral hemodynamic trends in the fNIRS signals of neurocognitive studies (Torricelli et al., 2014). In terms of data analysis, two or more short-separation channels (as recently suggested by Gagnon et al., 2014) were not included in the layout of the probe holder commercially available for the continuous wave fNIRS instrument utilized in the present study. Therefore, the suppression of the potential superficial artifacts, using for example an additional systemic predictor in the general linear model analysis of the fNIRS data (Gagnon et al., 2011, 2012), was not possible. In addition, it has been reported that the extracerebral contribution is more pronounced in the O2Hb than in the HHb signal (Kirilina et al., 2012). In the present study, significant changes in O2Hb and HHb were found, and, as suggested by Takahashi et al. (2011), the superficial effect was minimized by an accurate "measurement setting" (see Section Materials and Methods). In particular, the flexible probe holder and its position on the head allowed the creation of a stable optical contact. Although the pressure under the probe was not measured by a membrane pressure sensor, as in the study by Takahashi et al. (2011), the pressure created by the velcro brand fastener was adequate to induce a partial transient blockage of the skin circulation during the present fNIRS study. Although no correction algorithm has been utilized, the higher amplitude of the cortical responses observed over the 16 measurement points during the EP/RP (**Figure 2**) in comparison with that one observed in the respective control tasks, the unrelated time course of the HR changes (**Figure 2**), the partial transient blockage of the skin circulation under the fNIRS optodes, and the adequate mean penetration depth of the utilized near-infrared light (sourcedetector distance was estimated about a half of the 3.5) could support the argument that the described PFC activation during the LMT is mainly a task-related change in PFC oxygenation. On top of that in the present study most of the potential confounders/contamination factors for the fNIRS signals have been consistently reduced by subtracting the PFC oxygenation changes associated with the EPC/RPC from those associated with the EP/RP, respectively. This subtraction procedure has removed also possible inter-individual anatomical differences (i.e., scalp depth) over the investigated PFC areas. Thus, even if a contribution of physiological noise on the broad cortical activity pattern observed during RP cannot be completely ruled out, its influence on hemodynamic activity should be negligible due to the aforementioned subtraction procedures. More importantly, the selective activation found in channels 7 and 15 (**Figure 3**) during the EP cannot be explained by extracerebral hemodynamic trends. In fact, during the EP the activation was significantly prominent in VLPFC with respect to the other PFC areas, and with respect to the RP. Taken together, we believe that the observed differences in the hemodynamic response over the investigated PFC areas are likely to be specifically bound to the encoding and retrieval processes.

For what concerns the protocol, some methodological considerations should be pointed out on the partial transient blockage of the skin circulation provoked by the 14 optodes. PFC is even involved in pain perception and modulation (Lorenz et al., 2003). Therefore, during fNIRS studies it is important that the probe holder, equipped with the optodes, does not cause any discomfort to the subjects (e.g., physical and/or psychological discomfort). In the present study, in order to avoid any PFC activation induced by discomfort and/or pain, the subjects were properly instructed to alert the researchers whenever they experienced any discomfort and/or pain during the fNIRS measurements. All subjects completed successfully the LMT recording. Moreover, several 60 min fNIRS measurements were carried out separately to evaluate any discomfort and/or pain induced by the probe holder and the optodes. At the end of those fNIRS measurements, no pain was recognized by all the subjects using a numeric rating scale (Turk and Melzack, 2010). Therefore, discomfort and/or pain did not interfere with the LMT-related cortical activation observed in the present study.

In order to suppress any potential interference due to the modalities of the EP and the RP, the EPC and RPC were subtracted to the corresponding tasks. During the EPC the subjects listened the LMT story presented backward; in this case neither memory nor language comprehension were required, and any PFC activity would be related only to the auditory processing of the information. During the RPC the subjects were asked to repeat aloud the week days, repetitively (automatic speech). In this case, the subjects did not have to draw at WM for retrieving the encoded information, but only the vocalization/articulator movements of speech production were required. Taking into account the biphasic time course of O2Hb/HHb changes during LMT (**Figure 2**), the selection of the maximum values of the O2Hb/HHb concentration changes during EP and RP was considered the most appropriate for data analysis. We also note some limitations of the present study: the number of participants was limited and the task order was not counterbalanced across subjects, so in future researches we are planning to extend the sample number, also counterbalancing the task order. Moreover, given the restricted PFC area that could be explored using the 16 channel fNIRS system, we note that it might be useful to extend the optical recordings to the whole PFC and to other cortical regions that could be involved in verbal WM processing, such as temporal regions.

In conclusion, this study has demonstrated that, in response to the LMT, PFC oxygenation increases in both hemispheres in healthy subjects. While the EP elicited a markedly selective activity of VLPFC, the activation during the RP was more widespread (including VLPFC, DLPFC, and FPC). This could be explained by the role of the VLPFC in maintaining the information in WM and in understanding the narrative, while the broader PFC activity observed during RP (including DLPFC), could be caused by the need of manipulating and retrieving the memory traces from the previous encoded information in order to perform the task. Thus, the FPC recruitment could be ascribed to the construction of a global coherence essential for the LMT story recall.

To conclude, the results of the present study confirm that fNIRS could be considered as a valuable diagnostic tool, because of its proved applicability for the hemodynamic monitoring during the LMT performance. Considering the necessity of expanding the existing types, quantity and quality of fNIRS paradigms that could induce a cortical activation (Ehlis et al., 2014), future studies should be focused on these purposes, adopting cortical activation tasks as the currently used LMT paradigm.

#### **ACKNOWLEDGMENTS**

The study has been performed in the framework of the "Research Centre for Molecular Diagnostics and Advanced Therapies". The authors wish to thank the "Abruzzo earthquake relief fund" (Toronto, ON) that supported in part this research with the purchase of the Artinis system.

#### **REFERENCES**


SNARC effect with functional near-infrared spectroscopy. *Cereb. Cortex* doi: 10. 1093/cercor/bhs321. [Epub ahead of print].


*Clin. Child Psychol. Psychiatry* doi: 10.1177/1359104513487001. [Epub ahead of print].


Wechsler, D. (1997). *Wechsler Memory Scale – Third Edition Manual*. San Antonio, TX: The Psychological Corporation.

Wechsler, D. (2008). *Wechsler Adult Intelligence Scale*. San Antonio, TX: The Psychological Corporation.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 October 2013; accepted: 17 December 2013; published online: 31 December 2013*.

*Citation: Basso Moro S, Cutini S, Ursini ML, Ferrari M and Quaresima V (2013) Prefrontal cortex activation during story encoding/retrieval: a multi-channel functional near-infrared spectroscopy study. Front. Hum. Neurosci. 7:925. doi: 10.3389/fnhum.2013.00925*

*This article was submitted to the journal Frontiers in Human Neuroscience*.

*Copyright © 2013 Basso Moro, Cutini, Ursini, Ferrari and Quaresima. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

## Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS

#### *Christian Herff\*, Dominic Heger , Ole Fortmann , Johannes Hennrich , Felix Putze and Tanja Schultz*

*Cognitive Systems Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Karlsruhe, Germany*

#### *Edited by:*

*Leonid Perlovsky, Harvard University and Air Force Research Laboratory, USA*

#### *Reviewed by:*

*Hasan Ayaz, Drexel University, USA Megan Strait, Tufts University, USA*

#### *\*Correspondence:*

*Christian Herff, Cognitive Systems Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology, Adenauerring 4, 76131 Karlsruhe, Germany e-mail: christian.herff@kit.edu*

When interacting with technical systems, users experience mental workload. Particularly in multitasking scenarios (e.g., interacting with the car navigation system while driving) it is desired to not distract the users from their primary task. For such purposes, human-machine interfaces (HCIs) are desirable which continuously monitor the users' workload and dynamically adapt the behavior of the interface to the measured workload. While memory tasks have been shown to elicit hemodynamic responses in the brain when averaging over multiple trials, a robust single trial classification is a crucial prerequisite for the purpose of dynamically adapting HCIs to the workload of its user. The prefrontal cortex (PFC) plays an important role in the processing of memory and the associated workload. In this study of 10 subjects, we used functional Near-Infrared Spectroscopy (fNIRS), a non-invasive imaging modality, to sample workload activity in the PFC. The results show up to 78% accuracy for single-trial discrimination of three levels of workload from each other. We use an *n*-back task (*n* ∈ {1*,* 2*,* 3}) to induce different levels of workload, forcing subjects to continuously remember the last one, two, or three of rapidly changing items. Our experimental results show that measuring hemodynamic responses in the PFC with fNIRS, can be used to robustly quantify and classify mental workload. Single trial analysis is still a young field that suffers from a general lack of standards. To increase comparability of fNIRS methods and results, the data corpus for this study is made available online.

**Keywords: fNIRS, near-infrared spectroscopy, prefrontal cortex, workload, mental states, user state monitoring, n-back, passive BCI**

#### **1. INTRODUCTION**

Functional Near-Infrared Spectroscopy (fNIRS) is an imaging modality measuring hemodynamic processes in the brain. It provides insights into the same activation patterns as functional Magnetic Resonance Imaging (fMRI), the de facto standard in neuroscience research, while not confining the subject in a small space. Thereby, it allows for measurements of large subject populations outside of clinical environments. Besides montages covering the whole head, fNIRS sources and detector optodes can also be placed on the subjects head to measure exactly the parts of the cortex that contain relevant activations for the investigated task. When the region of interest is known beforehand, this can be used to design optode holders that can be fixed in place in less than 1 min. Potentially, fNIRS could thus be used in real world scenarios, as well.

Most fNIRS studies investigate differences in average activation patterns for different conditions. Only very recently has fNIRS been used to classify single-trial activations for Brain-Computer Interfacing (Coyle et al., 2007). A Brain-Computer Interface is a communication channel between the brain and a computer through interpretation of neural activation pattern (Wolpaw et al., 2002). Nearly all existing single-trial studies differentiate fNIRS patterns of subjects performing a cognitive task from the rest state or no-control state. The most frequently used paradigm is motor-imagery (Sitaram et al., 2007).

Recently, neural signals have been used to adapt and complement traditional input sources, such as keyboard and mouse, by adapting the interface to the users' state instead of directly controlling the interface. These so called passive Brain-Computer Interfaces (Cutrell and Tan, 2008; Zander and Kothe, 2011) mostly use the Electroencephalogram (EEG). Passive Brain-Computer Interfaces (BCIs) often measure a user's state and adapt a user interface accordingly. In fNIRS, multiple studies investigate mental arithmetics (Ang et al., 2010a) to monitor users' engagement in arithmetic tasks. Power et al. (2012) investigate the consistency of mental arithmetic classification across different sessions. Instead of recognizing mental arithmetics, Power et al. (2010) show that mental arithmetic and music imagery lead to distinct activation patterns that can be classified in single trial analysis. Following up on this idea, Herff et al. (2013) differentiate three different mental tasks, namely mental arithmetics, mental rotation and word generation. Girouard et al. (2009) distinguish between two difficulty levels in the popular game Pac-Man, instead of discriminating from a rest state. Ang et al. (2010b) show robust classification for three difficulty levels in mental arithmetics using fNIRS to evaluate numerical cognition class-room settings. While Ang et al. focus on the differentiation of difficulty levels, our focus is on the classification of mental workload induced by a memory task. Recently, Hirshfield et al. (2011) evaluated the type of cognitive demand placed on a user by different types of tasks. The focus of their study is on the type of workload, while we are aiming at the quantification of workload in this study.

In a multi-modal study using blood volume pressure, respiration measures, electrodermal activity and EEG, Jarvis et al. (2011) measured workload in a driving simulator to adapt a driving assistant. Workload has been of interest in the fNIRS community, as well. Cognitive workload has been assessed for air-traffic controllers in several studies Ayaz et al. (2010, 2012). Izzetoglu et al. (2003) show that task load in the Warship Commander tasks yield distinct hemodynamic responses on average. Aiming at a usage for BCI, Ayaz et al. (2007) analyze workload induced by the *n*-back tasks, but limit their results to grand averages, as well. However, these studies look at average hemodynamic responses and do not attempt single trial analysis. To use these findings to adapt interfaces to the user's current workload, the hemodynamic responses have to be analyzed in single trial. Proving that a cognitive task yiels hemodynamic responses on average does not automatically mean that the activations can be robustly recognized in single trial, which is necessary if interfaces should be adapted. In this work, we provide evidence that different levels of workload yield hemodynamic responses that can be robustly classified without averaging.

Findings in EEG Brouwer et al. (2012); Berka et al. (2007) show that workload induced by the *n*-back task can be classified in single trial. Baldwin and Penaranda (2012) demonstrate how the models trained on one workload condition can be transferred to others in EEG. In this study, we show that the workload induced by different *n*-back conditions results in hemodynamic responses that are consistent enough to be classified on a single trial basis. We use an *n*-back task to induce different levels of workload, forcing subjects to continuously remember the last one, two, or three of rapidly changing items. To enable realistic passive BCIs, we not only evaluate whether a user is engaged in a task, but quantify the level of mental workload the user experiences during the *n*-back task (*n* ∈ {1*,* 2*,* 3}). Thereby, we quantify workload using fNIRS.

In functional imaging studies, the prefrontal cortex (PFC) has been identified to be among the relevant areas for memory related tasks (Smith and Jonides, 1997). The PFC has been found to be relevant both in PET (Smith and Jonides, 1997) and fMRI studies (Cohen et al., 1997). An in depth meta-analysis of *n*-back studies using fMRI (Owen et al., 2005) confirms the importance of the PFC for *n*-back. Hoshi et al. (2003) show spatio temporal changes for working memory tasks in the PFC using fNIRS. Their analysis is based on averages and does not include single trial analysis, but confirms that fNIRS is ideally suited for measurements of the PFC. An fNIRS headset can be quickly fixed to the forehead and enables measurements of the PFC within minutes, while guaranteeing high data quality. In an investigation using finger tapping and fNIRS, Cui et al. (2010b) show that the delay in fNIRS-based BCIs can be reduced to further improve the usability of fNIRS in real-life scenarios. Workload induced by a memory task and fNIRS-based measurement of the PFC are thus an ideal combination for a realistic passive BCI to monitor workload levels.

#### **2. MATERIALS AND METHODS**

#### **2.1.** *n***-BACK**

In the *n*-back task, users have to continuously remember the last *n* of a series of rapidly flashing letters. The *n*-back task requires subjects to react when a stimulus is the same as the *n*-th letter before the stimulus letter. We denote a (letter) stimulus, which is the same as the one *n* previously as a target. Subjects had to press the space key on a keyboard when they encountered a target. With increasing *n* the task difficulty increases, as the subjects have to remember more letters and continuously shift the remembered sequence. Performance in this task can be evaluated by measuring the amount of missed targets, when the subjects do not press the key for a target and through the amount of wrong reactions, when the subjects incorrectly identify a stimulus letter as a target.

#### **2.2. NIRS DATA RECORDING**

Like fMRI, fNIRS measures changes in blood oxygenation in brain areas triggered by neural activity. Using light in the nearinfrared range of the electromagnetic spectrum (620–1000 nm), which disperses through most biological tissue but is absorbed by hemoglobin, the level of oxygenated and deoxygenated hemoglobin (*HbO* and *HbR*) can be estimated using the modified Beer-Lambert law (Sassaroli and Fantini, 2004).

We used an Oxymon Mark III by Artinis Medical Systems to measure fNIRS signals. The system uses two wavelength of 765 and 856 nm and outputs concentration changes of *HbO* and *HbR*. To measure hemodynamic activity in the PFC, we attached four transmitter and four receiver optodes to the forehead. Each detector measures time-multiplexed from two sources, located at a distance of 3.5 cm, resulting in a total of 8 channels of *HbO* and *HbR*. Our signals were sampled at 25 Hz.

**Figure 1** shows the placement of our optodes on the subjects' forehead. The recording setup on the forehead is very simple and needs less than 3 min to be fixed in place and to assess data quality.

#### **2.3. EXPERIMENT DESIGN**

In our experiment, we investigated 10 trials each of 1-,2-, and 3 back tasks. Each trial contained 3 ± 1 targets. The experiment was presented to the subjects on a screen, which was placed in front of them in 50 cm distance.

A trial consisted of 5 s of instruction, informing the subject which task (1-,2- or 3-back) was about to start. The trial then presented a new letter every 2 s. Every letter was displayed for 500 ms. The screen was left blank for the remaining 1.5 s. A total of 22 letters was presented during every trial resulting in a trial length of 44 s. Subsequently, a cross was displayed for 15 s during which the subjects were asked to relax to ensure that hemoglobin levels returned to baseline. We excluded these periods from our analysis, as they are strongly influenced by the previous hemodynamic responses. After half of the trials, an additional 10 s of the resting cross were displayed to have data periods with no activity to be used as RELAX trials. We intentionally use periods with true relax signals for our analysis instead of periods in which *HbO* and *HbR* returned to baseline. **Figure 2** shows the experiment protocol. The order of the different *n*-back conditions was pseudo-randomized. A 150 s break during which the subjects could drink or chat was included after 15 trials. The entire experiment had a recording time of 37 min ( 30 trials of 64 s, 15 relax trials of 10 s and 150 s in the middle).

The fNIRS data was recorded continuously during the entire session. The trials were segmented afterwards based on the

time sequence induced by the described experimental setup. In addition to the recorded fNIRS data, subjects filled out a questionnaire regarding their age, occupation, handedness and a series of questions about the experiment on a 6-point Likert scale. The scale ranged from "no agreement" (1) to "complete agreement" (6) for a given statement. We asked our subjects how much they agreed with the statements "The *n*-back task was demanding," to evaluate subjective workload. Subjects were asked to judge their level of concentration during the first and second half of the experiment by indicating their agreement with the statement "I was very concentrated." Additionally, subjects indicated their agreement with the phrase "The system is comfortable to wear." Lastly, we evaluated whether our participants thought that the duration of the experiment was appropriate. Section 3.1 contains results of the questionnaire evaluation.

#### **2.4. PARTICIPANTS**

In this study, we recorded 10 subjects (4 females) with a mean age of 22 years. Using the Edinburgh handedness inventory Oldfield (1971), we evaluated the handedness of our subjects. In total, we had 8 right-handed and 2 left-handed participants. All subjects had normal or corrected to normal vision. The participants were informed prior to the experiment and gave written consent. None of the subjects had ever taken part in an *n*-back study before to ensure that no training effects are present.

To increase comparability between fNIRS methods and results, the complete data collected in this study will be shared with the community (see Section 4.1).

#### **2.5. SIGNAL PROCESSING AND ARTIFACT REMOVAL**

The signals measured by fNIRS are subject to biological and technical artifacts. Cardiovascular effects like heart-beat, respiration and slow waves (e.g., Mayer Waves) influence the recorded data. Movement artifacts which alter the position of the optodes and lift them off the scalp, causing spikes in the recordings, are present in most fNIRS datasets, as well. A general overview of fNIRS artifacts and artifact removal techniques can be found in Cooper et al. (2012).

To attenuate trends and Mayer Wave like effects, we used a moving average filter, which subtracted the mean of the 120 s before and after every sample from every *HbO* and *HbR* datapoint. Moving average filters have been used successfully before to remove slow trends in experiments with long trials (Heger et al., 2013). Heart-beat and faster frequency signals are attenuate using an elliptical IIR low-pass filter with cutoff frequency of 0.5 Hz and filter order of 6, which robustly reduces heart-beat influences in the data. Finally, we used a wavelet artifact removal method (Molavi and Dumont, 2010) to reduce the effect of movement artifacts.

The trials were then extracted based on the experiment timings and associated with a label according to the *n*-back condition or RELAX. Each trial of any of the *n*-back conditions is 44 s long, while the relax trials are 10 s long.

#### **2.6. FEATURE EXTRACTION AND SELECTION**

Typical hemodynamic responses increase for *HbO* with neural activity in a specific region and return to baseline afterward. In *HbR*, signals typically behave opposite and decrease upon stimulus onset and increase back to baseline after the end of the stimulus. This typical behavior is often used in the feature extraction. The mean value of the signal (Heger et al., 2013) in a specific window or the increase in mean value between different windows (Herff et al., 2012) is often used as a simple, but effective feature. In this study, we use the slope of a straight line fitted to the data in a window as the feature. The line was fitted using linear regression with a least-square approach. Window sizes were varied in the experiments. Even though *HbO* and *HbR* signals of every channel are strongly negatively correlated (Cui et al., 2010a), we extract the slope feature for *HbO* and *HbR* of every channel. Including both *HbO* and *HbR* signals often yields more robust classification results. This results in 16 features per window, as we extract one feature for *HbO* and one for *HbR* for each of the 8 channels.

To reduce the feature set size, we only include features with a high relevance for classification in the feature set. We calculate the Mutual Information between each continuous feature and the discrete labels on the training data using non-parametric probability density functions. These were estimated using kernel methods (Parzen windows). See Ang et al. (2008) for a more detailed description of feature selection methods using Mutual Information. In this study, we limit our feature set to the 8 features containing the highest Mutual Information with the labels, as the remaining half of the features only contained little to no relevance.

#### **2.7. EVALUATION**

To classify the data, we used a Linear Discriminant Analysis (LDA) classifier. For the multi-class experiments, we used a one-vs-one multi-class classifying approach (Duda et al., 2012). To evaluate classification accuracy in our experiment, we used a 10-fold cross-validation. For this, the data of one subject is divided into 10 equally sized parts and in a round-robin manner, 9 parts are used for feature selection and training, while the last part is used for evaluation. Presented accuracies are then averaged over all 10 folds. We only evaluate subject dependend systems in this paper. As we use a 10-fold approach and have 10 trials per class, we never use any data shortly before or after the testing data, which could be problematic given the high auto-correlation of fNIRS signals. To evaluate our data set, we first classified the three *n*-back classes from RELAX. The RELAX trials are only 10 s long, while the *n*-back trials last 44 s. We only extracted 10 s long windows from *n*-back classes for this task, as well. Therefore, we evaluated the effect on classification accuracy resulting from different offsets from the start of a trial.

To really quantify mental workload we evaluate classification between the three *n*-back classes. We evaluate classification accuracy depending on window length in which we extract the slope feature.

#### **3. RESULTS**

#### **3.1. USER PERFORMANCE AND SUBJECTIVE RATING**

To confirm that our subjects perceived the different *n*-back conditions as different, we analyzed the user performance. **Figure 3** shows user performance and subjective evaluation of the experiment.

We evaluated the amount of missed targets, when a subject failed to press the key when a target stimulus was presented. A One-Way ANOVA shows significant differences between the three *n*-back levels in the amount of missed targets (*F* = 16*.*3151; *p <* 0*.*001). The percentage of targets missed by the subjects increased from 5.7% on average for the 1-back condition to 16.7% for 2-back to 33.7% for the 3-back task. This clearly shows that the three tasks have significantly different difficulty levels (tested by one-sided t-tests, *p <* 0*.*01 after Bonferroni correction all three comparisons). Additionally, this clarifies that even in the 3-back tasks our subjects identified two thirds of the targets. Next, we evaluated the amount of wrong reactions, when subjects incorrectly identified a letter as a target and pressed the space key. The amount of wrong reactions is significantly influenced by the *n*back level (tested by ANOVA, *F* = 9*.*613; *p <* 0*.*001). Again, the number of wrong reactions increases from 1.4 on average to 1.9 to 4.5. The differences in wrong reactions between 1 and 3-back and 2 and 3-back are significant (tested by one-sided t-test *p <* 0*.*01 after Bonferrroni correction), while the difference between 1 and 2-back is not statistically significant. The subjective evaluation of the subjects agreeing with the phrase "The *n*-back task was demanding," clearly shows the different mental workload levels of the three conditions (statistically significant as tested by One-Way ANOVA, *F* = 25*.*8540; *p <* 0*.*001). While the average agreement was 1.6 (1 meaning no agreement) for 1-back, subjects answered 3.1 for 2-back and 5.1 on average for 3-back (6 being total agreement). All differences between the three classes are significant (tested by one-sided t-tests *p <* 0*.*01 after Bonferroni correction). This clearly shows the different levels of workload induced by the three *n*-back conditions.

Subjects stated that they were highly concentrated during the first half of the experiment, answering that they agreed with 4.9 with the phrase "I was concentrated during this half of the experiment." This decreased slightly to 4.0 for the second half. The fNIRS system was judged as being comfortable to wear (3.9 in agreement to a comfortable system) in the first half, which decreased to a medium 2.7 for the second half. Our subjects evaluated the duration of the experiment as appropriate (agreement of 4.7).

#### **3.2. HEMODYNAMIC RESPONSES**

To see whether the Hemodynamic responses for the three *n*-back conditions yield any differences, we first analyze the grand averages of all subjects. For this analysis, we baseline every trial by subtracting the mean of the 10 s prior to the trial for *HbO* and *HbR* of every channel. The trials are not baseline normalized for the remaining classification analyses. **Figure 4** shows grand averages for all channels and all *n*-back conditions.

Gray lines show grand averages for individual channels, while the black line shows the mean over all channels. In the *HbO* channels, there is little activity for 1- and 2-back, but a clear increase

**FIGURE 3 | User performance and subjective evaluation in the** *n***-back task (A) average number of missed targets (B) average number of wrong reactions (C) average subjective evaluation of task difficulty.** Whiskers

show standard deviations between subjects. All differences between the conditions are significant (tested by one-sided *t*-tests, *p <* 0*.*05 after Bonferroni correction), except for the difference between 1 and 2-back in **(B)**.

for most channels in the 3-back conditions. It is obvious that a feature derived from the slope of those grand averages could discriminate the 3-back trials from the others. In *HbR* the typical decrease can be seen for all three conditions. While the slope is negative for all three tasks, it is clearly steeper in the 2-back grand average than in the 1-back and steepest for the 3-back averages. These grand averages show that we have different activation patterns for the three conditions and visualize the basis of our classification.

#### **3.3.** *n***-BACK vs. RELAX**

To evaluate the data set we first classified our *n*-back trials from the RELAX trials collected after the signals returned to baseline. Since our relax trials are only 10 s long, while our *n*-back trials are 44 s in length, we evaluated the effect the offset from the beginning of the trial has on classification accuracies. **Figure 5** shows the classification accuracies depending on the offset from the beginning of the trial when extracting the 10 s long windows.

Extracting the 10 s long window directly after the beginning of the trial yields the worst results for all conditions. This can be explained by the fact that subjects are only beginning to memorize the stimuli and are not experiencing workload yet. After an offset of 10 s the results remain relatively stable. All results are significantly better than chance level (tested by Wilcoxon rank-sum). Even in the four-class classification task we could achieve accuracies up to 45% (chance 25%). As expected, classifying 3-back against RELAX yielded the best results of up to 81% accuracy. For 2-back, we could achieve 80% accuracy for classification against RELAX and 72% for 1-back, respectively. These results show that the single trial data can be robustly discriminated from a relax state.

**Table 1** summarizes classification accuracies of each of the conditions against relax and for the four class experiment with an offset of 10 s. These results can be used to compare with previous studies which focus on discriminating from the RELAX state.

#### **3.4. QUANTIFYING MENTAL WORKLOAD**

To quantify workload it is necessary to discriminate different levels of workload from each other and not only from a RELAX state. We investigate the three *n*-back conditions against each other in two class and three class scenarios. To evaluate the window length necessary for robust classification of mental workload, we show classification accuracies depending on window length in **Figure 6**.

Part **(A)** of **Figure 6**, shows accuracies for the two class discrimination between two levels of workload, while part **(B)** shows the three class accuracies of all three workload levels. Note that with increasing window size, the amount of instances reduces. While we can extract 80 instances for a window length of 5 s, this amount reduces to 10 for window lengths larger than 25 s. The little amount of training and testing data sets explains the unstable results for window lengths longer than 25 s.

Results increase for increasing window lengths and peak for the length of 25 s. The discrimination between 1- and 3-back works best, which can easily be explained as the degree of difficulty is most different in those two conditions. Classification between 1- and 2-back and 3- and 2-back yield comparable results as the difference in difficulty level across these conditions is similar. For longer window lengths, these results are significantly better than chance level. The three class experiment is above chance for all window lengths and peaks at 50% accuracy for 25 s window length. The detailed results for every subject for window length of 25 s can be found in **Figure 7**. It can be seen that all subjects yield good results for the discrimination between 1-3 back, while only roughly half of the subjects work well for the other two scenarios. The results across subjects are significantly better than chance level for all classification scenarios (tested by Wilcoxon rank-sum tests).

**Table 2** summarizes the mean results across all subjects for window lengths of 25 s and 15 s. We present the results for window length of 15 s as well, as this length has been used for workload evaluation with EEG before (Kothe and Makeig, 2011). The results for 25 s long windows clearly show that fNIRS signals can be used to robustly quantify different levels of workload. This is a large step toward passive BCIs using fNIRS for workload monitoring.

#### **4. DISCUSSION**

In this study of 10 subjects, we show that fNIRS signals measured from the PFC with an easy to setup montage can be used to robustly quantify users' workload. The analysis of user performance show significant differences in the amount of missed targets and wrong reactions depending of the *n*-back level. Additionally, the subjective evaluation of the users show big differences in perceived difficulty level between the *n*-back levels, as well.

Using 8 channels on the forehead, we were able to classify the different levels of workload induced by *n*-back tasks from a relax state with accuracies up to 81%. As expected, 3-back could be discriminated best from the relax state (81% accuracy), as the mental workload induced by this condition is the largest.



**FIGURE 6 | Classification accuracies depending on window length (A) two class problems between different workload levels (B) three class classification of all three workload levels.**

**Table 2 | Classification accuracies of the conditions against each other.**


However, classification of 2-back and 1-back against relax still yielded mean accuracies of 80 and 72%, respectively. These results show that even the workload induced by relatively simple tasks can be robustly discriminated from a resting state.

More importantly, the hemodynamic responses measured in the PFC are consistent enough to be used to discriminate between three levels of workload. While the classification of high vs. low workload (1 vs. 3-back) worked well for all 10 subjects and yielded an average of 78% accuracy, the discrimination between 1 and 2-back only resulted in usable results for half of the subjects (average of 58.5%). Classification between the workload induced by 2 and 3-back tasks resulted in an average of 61% accuracy. These results mirror the subjective and user performance evaluation, as the difference between 1 and 3-back is largest and the difference in workload induced by 1 and 2-back seems to be smallest (no significant difference in the amount of errors between those two conditions).

We thereby show the potential of fNIRS as a modality for passive BCI and user state monitoring, despite the fact that further investigation is necessary to differentiate between more levels of workload with higher accuracies. The simple optode montage and the robust results encourage fNIRS to be used in real-life scenarios like car navigation and class-room settings. In this study, the data was analyzed in an offline manner and especially the moving average filter needs to be adapted for usage in an online system. Instead of only classifying whether a subject was engaged in a task or not, we were able to reliably show the degree of workload a subject was experiencing. The presented results thus show the feasibility of using fNIRS to quantify workload in single trial.

#### **4.1. DATA SHARING**

Single-trial analysis of fNIRS data is still a very young field and to the best of our knowledge, there are only very few publicly available data sets of single trial fNIRS experiments. To increase comparability of single trial fNIRS methods and allow for benchmarking, the data corpus used in this study will be publicly available on the authors' website <sup>1</sup> . The fNIRS time courses for all 10 subjects and for all *n*-back conditions and RELAX can be downloaded in both MATLAB™and Comma-Separated-Value (CSV) file formats. The questionnaire and behavior results will be included, as well. Thereby, we hope to provide a common data set for evaluation and testing of fNIRS methods and algorithms.

#### **ACKNOWLEDGMENTS**

The fNIRS equipment used in this study is part of the DFG funded Karlsruhe Design and Decision Laboratory, a collaboration laboratory of Economic Sciences, Psychology and Computer Science to investigate decision processes in groups. We acknowledge support by Deutsche Forschungsgemeinschaft and Open Access Publishing Fund of Karlsruhe Institute of Technology.

### **REFERENCES**


<sup>1</sup>http://csl.anthropomatik.kit.edu/english/2506.php


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 September 2013; accepted: 25 December 2013; published online: 16 January 2014.*

*Citation: Herff C, Heger D, Fortmann O, Hennrich J, Putze F and Schultz T (2014) Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS. Front. Hum. Neurosci. 7:935. doi: 10.3389/fnhum.2013.00935*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Herff, Heger, Fortmann, Hennrich, Putze and Schultz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Broca's area processes the hierarchical organization of observed action

#### *Masumi Wakita\**

*Department of Behavioral and Brain Sciences, Primate Research Institute, Kyoto University, Inuyama, Japan*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Hirokazu Doi, Nagasaki University, Japan Shinichi Furuya, Hannover University of Music, Drama and Media, Germany*

#### *\*Correspondence:*

*Masumi Wakita, Department of Behavioral and Brain Sciences, Primate Research Institute, Kyoto University, Kanrin 41-2, Inuyama, Aichi 484-8506, Japan e-mail: wakita.masumi.2e@ kyoto-u.ac.jp*

Broca's area has been suggested as the area responsible for the domain-general hierarchical processing of language and music. Although meaningful action shares a common hierarchical structure with language and music, the role of Broca's area in this domain remains controversial. To address the involvement of Broca's area in the processing action hierarchy, the activation of Broca's area was measured using near-infrared spectroscopy. Measurements were taken while participants watched silent movies that featured hand movements playing familiar and unfamiliar melodies. The unfamiliar melodies were reversed versions of the familiar melodies. Additionally, to investigate the effect of a motor experience on the activation of Broca's area, the participants were divided into well-trained and less-trained groups. The results showed that Broca's area in the well-trained participants demonstrated a significantly larger activation in response to the hand motion when an unfamiliar melody was played than when a familiar melody was played. However, Broca's area in the less-trained participants did not show a contrast between conditions despite identical abilities of the two participant groups to identify the melodies by watching key pressing actions. These results are consistent with previous findings that Broca's area exhibits increased activation in response to grammatically violated sentences and musically deviated chord progressions as well as the finding that this region does not represent the processing of grammatical structure in less-proficient foreign language speakers. Thus, the current study suggests that Broca's area represents action hierarchy and that sufficiently long motor training is necessary for it to become sensitive to motor syntax. Therefore, the notion that hierarchical processing in Broca's area is a common function shared between language and music may help to explain the role of Broca's area in action perception.

**Keywords: near-infrared spectroscopy, Broca's area, language, music, action, hierarchy, syntax**

#### **INTRODUCTION**

Human language consists of a hierarchical structure in which phonemes are combined to form words, phrases and sentences up to the discourse level of speech structure according to several levels of rules. Uchiyama et al. (2008) studied brain regions involved in the structural analysis of language and revealed a posterior– anterior functional gradient with substantial overlap in the left inferior frontal region: phonological processes are localized in the Brodmann area (BA) 44, the processing of sentence structure is localized in BA 45 and semantic processing at the sentence level is related to BA 47 activity (also see Friederici and Kotz, 2003; Hagoort, 2005).

In the case of music, similarly to language, combinations of sequential and simultaneous notes make rhythms, melodies and harmonies to form an overall musical structure according to rules such as chord progressions. There is also a posterior-anterior functional gradient in the inferior frontal region in relation to the processing of musical hierarchy. For instance, the activation of BA 6/44 for rhythm discrimination and melody matching may reflect the processing of sequential sounds (Platel et al., 1997; Brown and Martinez, 2007). BA 44/45 is involved in harmonic evaluation (Tillmann et al., 2003; Brown and Martinez, 2007). Furthermore, BA 47 is sensitive to the meaning (or impression) of melody, reflecting the semantic processing of music (Platel et al., 1997). The similarities of linguistic and musical analysis raise the possibility that hierarchical processing in 2 different domains is subserved by common neural resources such as Broca's area (BAs 44 and 45) (Maess et al., 2001; Koelsch et al., 2002; Patel, 2003; Brown et al., 2006; Schön and François, 2011).

Action, as well as language and music, is organized hierarchically. A chain of individually meaningless body movements (such as extending an arm and opening a palm) can be combined into units of actions (such as reaching and grasping). These units of actions can be integrated into meaningful behavior (such as picking up a peanut). Despite evidence for the possible contribution of Broca's area in the hierarchical processing of observed skilled action (Fiebach and Schubotz, 2006; Tettamanti and Weniger, 2006; Fadiga et al., 2009; Higuchi et al., 2009; Wakita and Hiraishi, 2011), few studies have directly examined such involvement.

For instance, Higuchi et al. (2009) found an overlap of activity between tasks related to language and tool-use action in Broca's area. The authors suggested a domain-general role of Broca's area in hierarchical processing since language and tool-use action share computational principles for processing hierarchical structures common to these two domains. However, they did not directly test the involvement of Broca's area in the processing of action hierarchy. In order to confirm such an involvement, it is necessary to study whether Broca's area is sensitive to the degree of hierarchical complexity or to the sequential order of action.

Thus far, several patterns of musical action have been adopted to study the involvement of Broca's area in the hierarchical processing of observed action. Musical action consists of hierarchical structures in that the visual processing of structured key-press sequences and the integration of this information with the location of the fingers on the piano keyboard are essential to correctly identify harmony or melody by observing silent hand motion (Hasegawa et al., 2004).

For instance, Sammler et al. (2013) conducted an electroencephalography (EEG) study in which expert pianists watched silent videos of a hand playing either congruent or incongruent 5-chord sequences. The video corresponded to a linguistic task that tested the grammatical sensitivity of Broca's area. The results of the study revealed a higher activation of this area in response to sentences with non-canonical word order compared with sentences with standard word order (Friederici et al., 2006). The authors found positivity that possibly originated from a left inferior frontal source when participants observed hand trajectory toward syntactically incongruent chords. These findings indicate that the syntactical analysis of observed action is represented in Broca's area. However, such a silent harmony abstraction task is difficult to perform because theoretical knowledge of western harmony is necessary. Therefore, participation in this task is restricted to professional musicians.

In another study, using functional magnetic resonance imaging (fMRI), non-musicians watched silent videos of hands playing the piano while trying to identify the melodies (Hasegawa et al., 2004). This task may measure the sensitivity of Broca's area to tone or phoneme segmentation and reveal the activation of Broca's area to learned regularity (Abla and Okanoya, 2008; Karuza et al., 2013). Because less-trained participants could identify some melodies solely through the observation of silent keytouching sequences, this task may be adaptable to a larger number of participants than the silent harmony abstraction task. The authors' target region was the left planum temporale, but they observed higher levels of activation of BA 44 in well-trained participants compared with less-trained and naïve participants when they watched silent videos of hands playing melodies. However, in general, melody identification performance in this study was not good, even among the well-trained participants. Thus, the activation of Broca's area may have reflected an embodied simulation of piano playing rather than hierarchical processing.

In this study, to test whether Broca's area is involved in processing the hierarchical organization of observed action, well-trained and less-trained participants watched silent movies that featured hand movements playing familiar and unfamiliar melodies. The activation of Broca's area was measured using near-infrared spectroscopy (NIRS). The NIRS results for familiar and unfamiliar melodies were compared to reveal the involvement of Broca's area in the hierarchical analysis of observed action. Additionally, the comparison of NIRS results between well-trained and less-trained participants was examined to determine the influence of skill in the observed action on the activation of Broca's area.

All participants in the current study had sufficient knowledge about the familiar melodies preceding the experiments. Thus, tracking hand motions to identify the familiar melodies and even anticipating forthcoming action sequences may be easy. However, no participant had knowledge about the unfamiliar melodies. Thus, the participants had to pay extra attention to the hand motions to detect musical segments from stimulus movies and integrate them into whole melodies. Therefore, if a greater activation of Broca's area was found for unfamiliar than familiar melodies, it could be deduced that this area plays an essential role in the hierarchical processing of an observed action.

Although NIRS cannot detect deep brain activity, this technique offers several advantages. NIRS detects changes in hemoglobin concentration associated with brain activation occurring beneath the optodes. Therefore, this technique enables the estimation of the activation source using fewer sensors than are required for EEG. Additionally, NIRS allows the participants to maintain a comfortable posture in a quiet room during measurement, while body movements of participants are strictly restricted during an fMRI session. Thus, NIRS is a suitable technique to measure activity in Broca's area and is advantageous, especially for the participants.

#### **METHODS**

#### **SUBJECTS**

The subjects were 20 healthy, right-handed adults aged 23–40 years. Handedness scores were determined using the Edinburgh handedness inventory (Oldfield, 1971). The subjects were divided into 2 groups according to the criterion by Hasegawa et al. (2004). The subjects in the well-trained group (10 females; mean age, 28.7 years; range, 23–34 years) had received formal piano lessons for at least 8 years (mean, 11.4 years; range, 8–16 years). The subjects in the less-trained group (2 males and 8 females; mean age, 29.9 years; range, 24–40 years) had received formal piano lessons for less than 8 years (mean, 3.4 years; range, 1–7 years). All subjects were able to identify the familiar melodies by watching the keytouching hand movements. Prior to the experiment, the subjects were informed about the nature of the experimental procedures and gave their written informed consent for the study. This study was approved by the ethics committee of the Primate Research Institute, Kyoto University.

#### **STIMULUS**

First, hand movements playing the familiar melodies "Mary Had a Little Lamb" (M) and "London Bridge is Falling Down" (L) (key, C major; tempo, 120 beats per minute; duration, 8 s) on the keyboard were filmed (640 × 480 pixels) (**Figure 1**). There were fade-in and fade-out periods (1 s) in the movies to prevent the subjects from using the initial hand position as a cue to identify the melody. By making temporally reversed versions of these movies, "musically" unfamiliar stimuli were created. Consequently, four movies (familiar M and L movies and unfamiliar M and L movies) were generated. Two identical movies were combined to make the 16-s stimuli. No auditory signal was included in the stimuli (**supplementary videos**).Stimuliwerepresented ona17-inchliquid crystal display monitor placed ∼70 cm from the subjects' heads.

The unfamiliar stimuli contained spatial and temporal distributions of key-press durations of hand movement that were

**FIGURE 1 | Static example of a stimulus movie.**

identical to those of the familiar stimuli. However, by reversing the temporal order, the sequential structure of hand motions (corresponding to note transitions) was rendered largely different between familiar and unfamiliar stimuli.

### **PROCEDURE**

#### **ACTION OBSERVATION TASK**

An experimental session consisted of 12 trials. Each trial lasted 16 s, during which one of the four stimuli was presented. The length of the inter-trial interval was 25 s. All four movies were presented three times in a pseudorandom order, with each session beginning with a familiar stimulus. Every stimulus was presented once in four successive trials, and the same melody (M and L) and/or familiarity (familiar and unfamiliar) condition was repeated in no more than 3 consecutive trials. The participants were instructed to watch the movies carefully to identify the "melody" performed by the hand. The beginning of each stimulus movie was accompanied by a tone pip. The participants were asked to open their eyes gently when they heard the tone and close their eyes gently when the stimulus movie disappeared. Thus, the individuals' eyes were closed during the rest period. The entire experiment lasted for ∼30 min.

Prior to the beginning of the recording session, the participants practiced for 8 trials (4 stimuli × 2 trials) with experimental stimuli to confirm that they understood the instructions.

#### **NEAR-INFRARED SPECTROSCOPY MEASUREMENTS**

Cortical activity was continuously recorded throughout the experiment. Relative changes in oxy-hemoglobin (oxy-Hb) concentrations were measured using the NIRS system (ETG-100, Hitachi Medical Corporation, Tokyo, Japan). The sampling rate was set to 10 Hz. Nine optodes (in a lattice pattern forming 12 channels) were placed on both hemispheres. Because the main region of interest was Broca's area, a single recording channel was determined as a point between the optode nearest to F7 of the international 10–20 system and its posteriorly adjacent optode. Thus, NIRS results presumably reflected activity from Broca's area since F7 on the scalp has been reported to project onto the cortical surface of the anterior portion of the inferior frontal cortex (BA 45/47) (Okamoto et al., 2004, 2006; Koessler et al., 2009). Typically, the position of the target channel was located ∼3 cm above the level of the supraorbital process and ∼3 cm posterior to the lateral margin of the orbit.

It is true that cranio-cerebral correspondence using the international 10–20 system has been suggested (Okamoto et al., 2004, 2006; Koessler et al., 2009). However, there is substantial variation in the precise location and topographic extent of Broca's area among individuals (Amunts et al., 1999). Therefore, I acknowledge the difficulty in ensuring the degree of contribution of different subdivisions within the inferior frontal cortex to the NIRS signal.

The cortical responses for each trial were stimulus-locked and extracted from the raw oxy-Hb time series data. Pulsatile fluctuations were removed from the results by smoothing the oxy-Hb time series backward in time with a 5-s moving window. Baseline drift was corrected using linear interpolation between the time point of task onset and the time point of stimulus onset of the next trial. The oxy-Hb time series data from familiar and unfamiliar trials were individually averaged over trials, independent of the melody types. Finally, a mean oxy-Hb value per time point within a peristimulus period (a 16-s time window from 5 s after stimulus onset to 5 s after stimulus offset) was calculated for each condition. The averaged oxy-Hb values obtained during the familiar and unfamiliar conditions were then submitted to a paired *t*-test to assess the involvement of Broca's area in the hierarchical processing of observed action. The significance level was set to 5%.

One may assume that the degree of involvement of Broca's area to action hierarchy is influenced by the experience. Therefore, the relationship between this involvement of Broca's area and the duration of piano training was assessed. First, subtraction of the averaged oxy-Hb value in the familiar condition from that in the unfamiliar condition was individually calculated as an index that shows the sensitivity to action hierarchy. Then, the correlation between such contrast and the duration of piano training was evaluated using a Spearman's rank correlation test. The significance level was set to 5%.

#### **RESULTS**

**Figure 2** shows the average activity of Broca's area across time in the familiar and unfamiliar conditions. In the trained group (A), a difference in the patterns of signal change was evident between the two conditions. However, such a contrast was not observed in the less-trained group (B).

The activation of Broca's area within the peristimulus period is shown in **Figure 3**, where the mean signal intensities for individual participants are compared between conditions. In the well-trained group (left), statistical analysis revealed a significant difference in the signal intensities between conditions [familiar condition (mean ± SD): 0.018 ± 0.031 mM·mm, unfamiliar condition: 0.045 ± 0.035 mM·mm] [paired *t*-test, *t(*9*)* = 3*.*167, *p* = 0*.*011]. Contrary to the well-trained group, the less-trained group (right) showed no significant difference in signal intensities between conditions [familiar condition: 0.018 ± 0.038 mM·mm, unfamiliar condition: 0.018 ± 0.036 mM·mm] [paired *t*-test,

*t(*9*)* = 0*.*076, *p* = 0*.*941]. Thus, the involvement of Broca's area in the hierarchical processing of observed action was observed in the well-trained participants.

The relationship between the degree of such involvement of Broca's area and the duration of piano training were then assessed. A correlation analysis of the entire population revealed that the between-condition difference in mean signal intensity strongly correlated with the duration of piano training [**Figure 4**; Spearman's rank correlation test, *rho(*18*)* = 0*.*593, *p* = 0*.*006]. When a correlation analysis was applied to individual groups, however, no significant correlation was found both in the welltrained group (red dots; α = 0*.*05*/*2, *rho(*8*)* = 0*.*654, *p* = 0*.*040) or the less-trained group (blue dots; α = 0*.*05*/*2, *rho(*8*)* = 0*.*092, *p* = 0*.*800).

#### **DISCUSSION**

#### **GENERAL CONCLUSION**

The primary goal of the present study was to determine the role of Broca's area in the hierarchical processing of observed action.

**unfamiliar conditions and duration of piano training.** Red and blue dots represent the results of well-trained and less-trained participants, respectively.

To address this issue, well-trained and less-trained participants were exposed to silent hand motions that played familiar and unfamiliar melodies. A key finding was that well-trained participants showed increased activation in Broca's area in response to hand motions associated with unfamiliar rather than familiar melodies. Thus, the current data indicate that this region plays an essential role in the hierarchical processing of an observed action. Previous fMRI and magnetic stimulation studies that showed that Broca's area is involved in hierarchical sequence of action planning (Koechlin and Jubault, 2006; Clerget et al., 2011, 2013) may support the current findings.

Additionally, a correlation analysis of the entire population exhibited that the Broca's area of participants who had longer piano training experience exhibited a stronger sensitivity to the hierarchical structure of observed action. However, because there was no such correlation within each participant group, sensitivity to action hierarchy may not develop monotonically as a function of the duration of piano training. Rather, the current results imply that Broca's area becomes sensitive to perceived motor hierarchy only after sufficient motor training.

#### **WAS THE HAND MOTION MEANINGLESS TO THE LESS-TRAINED PARTICIPANTS?**

In the current study, the less-trained participants showed no significant difference in the activation of Broca's area between the familiar and unfamiliar melodies. It has been shown that the activation of Broca's area is associated with an action that is meaningful or possible to the observer (Decety et al., 1997; Grèzes et al., 1998; Costantini et al., 2005; Wakita and Hiraishi, 2011). Therefore, it may be assumed that the observed actions were meaningless to the less-trained participants. The debriefing after the recording session revealed that neither the well-trained nor less-trained participants could report what the unfamiliar melodies were like, but both well-trained and less-trained participants were able to identify the familiar melodies by the observation of key-touch. Therefore, the task performance was comparable between both groups of participants, and the hand movements featuring familiar and unfamiliar melodies were meaningful actions for both groups of participants. How can the difference in the activation of Broca's area between well-trained and less-trained participants explained?

#### **PROCESSING OF OBSERVED ACTION IN BROCA'S AREA OF WELL-TRAINED AND LESS-TRAINED PARTICIPANTS**

The current study implied that Broca's area is also responsible for the analysis of hierarchical structure in a wide range of functional domains. Therefore, it must be significant to survey how native and non-native languages are processed in Broca's area prior to considering the effects of motor expertise on the activity of this area.

Rüschemeyer et al. (2006) showed that BA 44 of a native speaker becomes active only when the processing of a structurally complex sentence is demanded; BA 44 is less active when processing a simple sentence. However, BA 44 of non-native speakers is consistently active, regardless of the degree of complexity of sentences (see also Rüschemeyer et al., 2005; Yokoyama et al., 2006). Moreover, Jeon and Friederici (2013) demonstrated that the syntactical analysis of complex non-native sentence recruited BA 47. Thus, only BA 44 of native speakers showed complexitydependent activation. However, the less-proficient syntactical processing of non-native speakers is compensated for by the semantic information in the sentences.

When well-trained native and less-trained foreign language processing in Broca's area is taken into account, the difference in the activation of this area between well-trained and less-trained participants may be plausibly explained. The current results indicated that Broca's area in well-trained participants was sensitive to the hierarchical organization of hand motion. This effect was likely due to the extensive piano experience of the participant and the concurrent acquisition of neuronal mechanisms in Broca's area that represent action hierarchy. Based on the findings by Rüschemeyer et al. (2006), as well as Jeon and Friederici (2013), it is conceivable that Broca's area is not necessarily activated by processing a simple hand motion action. Rather, Broca's area may come into play only when processing an unfamiliar hand motion sequence.

However, Broca's area in less-trained participants was not sensitive to the hierarchical organization of hand motion. Again, based on the aforementioned linguistic studies, less-trained participants most likely employed neuronal mechanisms that identified "melodies" by relying on the meaning of the melody [mediated by BA 47 (Platel et al., 1997)] rather than by relying on the hierarchical organization of hand motions. Consequently, the appearance of hand motion may induce an equivalent degree of activation in Broca's area regardless of the familiarity of the stimulus. Future studies may compare the activation from the posterior part of the inferior frontal region (BAs 44 and 45) and the anterior part of the inferior frontal region (BAs 45 and 47) to investigate the effects of expertise on the development of the representation of action hierarchy in Broca's area.

#### **ROLE OF BROCA'S AREA AS A DOMAIN-GENERAL HIERARCHICAL PROCESSOR**

Previous studies have reported that familiar body movements evoked increased activation of several brain regions of the mirror neuron system (MNS) than unfamiliar movements (see Rizzolatti and Craighero, 2004; Fadiga et al., 2005 for review). For instance, Calvo-Merino et al. (2005) found a stronger recruitment of MNS including left ventral premotor cortex (vPM) close to Broca's area and bilateral intraparietal sulcus in expert ballet and capoeira dancers for the observation of the dance style of a familiar genre when compared with the unfamiliar genre. The authors argued that the activation of MNS reflected an action resonance processes. However, as far as observation-related brain activation is explained by an uncertain notion, i.e., "action resonance," we do not currently understand the differential processing of information within and between brain regions among MNS.

Alternatively, considering the shared function of Broca's area in a wide range of functional domains must be helpful to understand the role of this area in MNS. Musso et al. (2003) indicated that Broca's area is centrally involved in the processing of grammatical rules. They trained a native speaker of German to learn the grammatical rules of a non-native language with different grammatical rules (e.g., Italian and Japanese). Consequently, they found a significant correlation between the degree of activation of Broca's area and the performance of the judgment of syntactical correctness of language. Thus, Broca's area became active only when regularity of hierarchical structures was successfully abstracted from the sentence.

Given that the acquisition of hierarchical structures is an essential factor for Broca's area to be active, the vPM activation in Calvo-Merino et al. (2005) may be explained in terms of the ability to process action hierarchy. Because classical ballet and capoeira do not share poses and motion sequences with each other, it is plausible to assume that ballet dancers could abstract the regularity of a ballet dance sequence but could not abstract action hierarchy from capoeira. Consequently, the inferior frontal region of ballet dancers may have been more thoroughly recruited for the observation of ballet compared with capoeira and vice versa. Thus, if the notion that Broca's area process hierarchical structures in a wide range of functional domains is correct, this may provide insights into the neural basis of action understanding.

In conclusion, Broca's area was shown to be necessary for the hierarchical processing of observed action. Therefore, the notion that hierarchical processing in Broca's area is a common function shared between the language and music domains may help to explain the role of Broca's area in action perception. In support of these findings, previous studies have reported that the effect of expertise in one domain leads to the improvements in tasks of untrained domains such as enhanced linguistic performance after musical training (Moreno et al., 2009; Strait et al., 2012; François et al., 2013). The effect of auditory rhythm on motor functioning has also been suggested (de Dreu et al., 2012; Hardy and LaGasse, 2013). In summary, the notion of Broca's area as a domain-general hierarchy processor may provide insight into the beneficial transfer across language, music and action abilities.

#### **ACKNOWLEDGMENTS**

This study was supported by grants from the Nakayama Foundation for Human Science. The author thanks Reiko Sawada for technical support.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnhum. 2013.00937/abstract

**Supplementary Video 1 | Example of a familiar stimulus.** This video shows a familiar stimulus generated from "Mary Had a Little Lamb."

#### **Supplementary Video 2 | Example of an unfamiliar stimulus.** This video

shows an unfamiliar stimulus generated from "Mary Had a Little Lamb."

#### **REFERENCES**


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 02 October 2013; accepted: 27 December 2013; published online: 17 January 2014.*

*Citation: Wakita M (2014) Broca's area processes the hierarchical organization of observed action. Front. Hum. Neurosci. 7:937. doi: 10.3389/fnhum.2013.00937 This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Wakita. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Replication of the correlation between natural mood states and working memory-related prefrontal activity measured by near-infrared spectroscopy in a German sample

#### *Hiroki Sato1,2\*, Thomas Dresler 1,3, Florian B. Haeussinger 1, Andreas J. Fallgatter 1,4 and Ann-Christine Ehlis <sup>1</sup> \**

*<sup>1</sup> Psychophysiology and Optical Imaging, Department of Psychiatry and Psychotherapy, University of Tuebingen, Tuebingen, Germany*

*<sup>2</sup> Hitachi, Ltd., Central Research Laboratory, Hatoyama, Japan*

*<sup>3</sup> LEAD Graduate School, University of Tuebingen, Tuebingen, Germany*

*<sup>4</sup> Center of Integrative Neuroscience, Excellence Cluster, University of Tuebingen, Tuebingen, Germany*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Yukiori Goto, Kyoto University, Japan Hirokazu Doi, Nagasaki University, Japan*

#### *\*Correspondence:*

*Hiroki Sato and Ann-Christine Ehlis, Psychophysiology and Optical Imaging, Department of Psychiatry and Psychotherapy, University of Tuebingen, Calwerstraße 14, Tuebingen 72076, Germany e-mail: hiroki.sato@ med.uni-tuebingen.de; hiroki.sato.ry@hitachi.com; ann-christine.ehlis@ med.uni-tuebingen.de*

Previous studies have suggested complex interactions of mood and cognition in the lateral prefrontal cortex (PFC). Although such interactions might be influenced by various factors such as personality and cultural background, their reproducibility and generalizability have hardly been explored. In the present study, we focused on a previously found correlation between negative mood states and PFC activity during a verbal working memory (WM) task, which had been demonstrated by using near-infrared spectroscopy (NIRS) in a Japanese sample. To confirm and extend the generalizability of this finding, we conducted a similar experiment in a German sample, i.e., participants with a different language background. Here, PFC activity during verbal and spatial WM tasks was measured by NIRS using a delayed match-to-sample paradigm after the participants' natural mood states had been evaluated by a mood questionnaire (Profiles of Mood States: POMS). We also included control tasks to consider the general effect of visual/auditory inputs and motor responses. For the verbal WM task, the POMS total mood disturbance (TMD) score was negatively correlated with baseline-corrected NIRS data mainly over the left dorsolateral PFC (i.e., higher TMD scores were associated with reduced activation), which is consistent with previous studies. Moreover, this relationship was also present when verbal WM activation was contrasted with the control task. These results suggest that the mood–cognition interaction within the PFC is reproducible in a sample with a different language background and represents a general phenomenon.

**Keywords: near-infrared spectroscopy (NIRS), fNIRS, prefrontal cortex, optical topography, mood, POMS, working memory, hemodynamics**

#### **INTRODUCTION**

While we feel that our mood can influence our everyday activities to some extent, researchers have revealed a wide range of cognitive functions to be modulated by mood (Mitchell and Phillips, 2007). An example of such a mood–cognition interaction has been suggested for working memory (WM), as behavioral studies have shown that WM performance is affected by some emotional states such as withdrawal-related negative states (Gray, 2001; Shackman et al., 2006). Furthermore, the prefrontal cortex (PFC), which plays a crucial role in WM function (Kubota and Niki, 1971; Petrides et al., 1993; D'Esposito et al., 1995), has been suggested to be a brain region where cognition and emotion interact (Pessoa, 2008). In fact, while the PFC has usually been thought to be the central region underlying cognition, Nauta already suggested in 1971 that it is also related to affective and motivational responses to the person's environment based on its close association with the limbic system (Nauta, 1971; Pessoa, 2008).

Recent neuroimaging studies have investigated the impact of mood states on WM-related cortical activity. For example, a functional magnetic resonance imaging (fMRI) study showed that activity in the dorsolateral PFC (DLPFC) during a verbal (number) WM task was reduced after the participants were exposed to negative mood-inducing acute psychological stress (viewing aversive movie clips) (Qin et al., 2009). On the other hand, another fMRI study revealed that an unpleasant emotional state (induced by video clips) resulted in enhanced DLPFC activity during a verbal (letter) WM task and reduced activity during a non-verbal (face) WM task, whereas a pleasant emotional state exhibited the completely opposite pattern (Gray et al., 2002). These contradictory findings suggest a complex mood–cognition interaction in the PFC; however, an integrated theory has not been established yet. Moreover, most of the previous studies used mood-induction methods, which differ in their respective effects (Martin, 1990) and which might be different from natural mood and its effect on cognition (Parrot and Sabini, 1990). To summarize, only few aspects about the relationship between "natural moods" <sup>1</sup> and WM-related PFC activity have been clarified.

To better understand the mood–cognition interaction in the PFC, we previously introduced a new approach which focused on the natural mood state in healthy participants without using a mood-induction method (Aoki et al., 2011, 2013; Sato et al., 2011). In these studies, a psychological questionnaire (Profiles of Mood States, POMS) (McNair et al., 1971; Yokoyama et al., 1990) was administered to assess the participants' natural moods during the past week, and near-infrared spectroscopy (NIRS) a non-invasive and less body-constraint neuroimaging technique (Maki et al., 1995; Koizumi et al., 2005; Ehlis et al., 2014)—was used to measure their PFC activity during an emotionally-neutral delayed match-to-sample verbal/spatial WM paradigm (Smith et al., 1996). A correlation analysis among 29 healthy participants revealed that subjects reporting higher levels of negative mood showed lower levels of PFC activity during the verbal, but not the spatial WM task (Aoki et al., 2011). In a next step, we tried to dissociate the state-dependent effect from trait-dependent factors. Using a within-subjects design (Sato et al., 2011), experimental sessions were repeated three times at 2-week-intervals to investigate time-to-time fluctuations. The results indicated that changes in the depressed-mood score between successive sessions were negatively correlated with changes in left PFC activation for the verbal WM task (Sato et al., 2011), which is well in line with the former study (Aoki et al., 2011). Another previous study examined the contribution of personality effects (measured with the NEO Five-Factor Inventory and the Behavioral Inhibition/Activation scales) and replicated the negative correlation between negative mood scores and PFC activity for the verbal WM task even after controlling for the participants' personality traits (Aoki et al., 2013). Together, these studies strongly suggest that PFC activity during verbal WM tasks reflects the participant's natural mood-state independent of various trait factors. Furthermore, the validity of NIRS for measuring PFC activation in response to a verbal WM task was demonstrated in a simultaneous NIRS-fMRI measurement (Sato et al., 2013): A significant correlation between the NIRS signals and blood oxygenation level-dependent (BOLD) signals for the WM-related PFC activity was shown—not only for the temporal changes, but also for the amplitude of the signal response (Sato et al., 2013).

In the present study, we aimed to replicate our previous findings on a mood–cognition interaction in the PFC for a verbal WM task (Aoki et al., 2011) in a German sample, i.e., participants with a different language background while also striving to improve previous methodical weaknesses: First, we find it necessary to confirm the reproducibility of the findings when the PFC activity is derived from a contrast with a suitable control condition in an improved task design. Because the previous studies only used activation values relative to baseline, possible confounds (input of visual stimuli, reaction by button press, etc.) could not be ruled out. Second, it is important to determine whether the phenomenon can be replicated in other samples with a different cultural and ethnical background. In particular, because a verbal function is the object of our study, the replication in a sample with a different language background is necessary to show the robustness of this phenomenon. Indeed, the Japanese letters used in our previous studies represent a minority in the world; besides, it is still difficult to determine the genetic affiliation of Japanese language to other languages or language families (Teruya, 2004). In this study, we conducted an experiment in a German sample whose language belongs to the major language family of "Indo-European languages" using Latin alphabet letters. Although we usually assume that there are no significant cultural/language-related differences in cognitive neuroimaging data, it is worth replicating the results in a sample with a different language background to make our knowledge more practical.

#### **MATERIALS AND METHODS PARTICIPANTS**

Thirty-one healthy German volunteers participated in this study (16 females and 15 males, mean age ±*SD* = 28*.*0 ± 7*.*1 years). All of them reported no history of neurological or psychiatric disease and gave their written informed consent to participate in this study, which was approved by the Ethics Committee of the University Hospital Tuebingen. All procedures were in line with the Declaration of Helsinki in its latest version. Both males and females were included in this study because our previous study did not reveal any gender differences for the main result of a correlation between negative mood and PFC activity (Aoki et al., 2011). After we excluded data from four participants with increased depression scores (BDI scores *>* 14) and one participant with poor task performance (i.e., accuracy *<*60%), data from 26 participants (13 females and 13 males, all right-handed except for one male) with a mean age of 25*.*9 ± 4*.*6 years (age range: 20–42) were further analyzed. All of them had a higher education (of at least 13 years; "Abitur" in German).

#### **WORKING MEMORY TASKS**

WM refers to the temporary retention of information to perform cognitive activities (Baddeley, 1983; D'Esposito, 2007). Here, we used verbal and spatial delayed match-to-sample WM tasks (Smith et al., 1996) as previously applied (Aoki et al., 2011, 2013; Sato et al., 2011) (**Figure 1A**).

In the verbal WM task, four lowercase letters were presented at four peripheral locations as a target stimulus (Target), and a capital letter was presented at one of the other four peripheral locations as a test stimulus (Test) (see Target and Test stimuli for verbal WM condition in **Figure 1A**). We pseudo-randomly selected and arranged four sets from eight letters (b, f, g, h, m, n, p, t) for the Target, not to make up any meaningful words. The participants were asked to judge whether the character presented as Test corresponded to any of the Target characters and to respond as quickly as possible by pressing a button ("yes"—right index finger, "no"—right middle finger). Since the characters in Target and Test were presented in different letter-types (lowercase/capital), participants had to decide by using the phonetic information (i.e., the phonological loop). In the verbal control (v-control) task, four "x"s were presented at the four fixed locations as Target, and either an "X" or "O" was presented at one of the other four

<sup>1&</sup>quot;Natural moods" are defined as relatively low-intensity and diffuse affective states without a striking preceding cause (Forgas, 1995), being different from emotion and experimentally induced moods.

locations as Test. The participant had to decide whether the Test character was "X" (Yes) or "O" (No). This v-control task was designed to match the verbal WM task in visual/auditory inputs and motor response with less WM load.

In the spatial WM task, the Target was given by the locations of four squares irregularly arranged at eight peripheral locations. The arrangement of squares was organized in a complex way (e.g., no more than three successive squares, no meaningful arrangement like large square or rhombus). After an arrangement was presented as Test, the participant was asked to judge whether the Test square location was identical to any of the Target square locations. In the spatial control (s-control) task, four circles arranged to make a regular square were presented as Target, and a circle was presented at a location among the eight peripheral locations as Test. The participant had to decide whether the location of the Test circle was identical to any of the Target circle locations. Responses were indicated as in the verbal task.

The software *Presentation* (Neurobehavioral Systems, Inc., U.S.A.) was used for stimuli presentation and reaction time (RT) recording. Each WM task trial was as follows (see **Figure 1**): First, a central fixation cross (30% black) turned white 500 ms before the Target onset with an auditory cue (1000 Hz sine wave of 100 ms duration) as a warning signal. Then the Target was presented for 1500 ms, followed by a delay period of 7000 ms with the white fixation cross. During the delay period for the spatial WM task, positions of the eight peripheral squares were also shown in dark-gray (75% black). In the same way, positions of the four peripheral circles were shown for the delay period in the s-control task. Test, indicated by an 800 Hz sine wave of 100 ms duration, was presented until the participant responded (maximum: 2000 ms). All stimuli in Target and Test were shown in light-gray (20% black) at a location among the eight peripheral locations, and the inter-trial-intervals with the central fixation cross (30% black) were randomly varied between 15 and 18 s. A dark-gray background (70% black) was shown throughout.

Participants performed the verbal and spatial WM tasks in different runs. In both verbal WM and spatial WM runs, 8 WM task trials and 8 control trials were presented in a pseudo-randomized order (16 task trials in total for each run). In addition, "yes" and "no" trials were pseudo-randomized to be the same number (i.e., 4 trials for "yes" and the other 4 for "no" in each condition). The order of runs (verbal WM run and spatial WM run) was counterbalanced across participants.

#### **PROCEDURE**

Before the NIRS measurement, the participants' natural mood was assessed by using a short form of the POMS (McNair et al., 1971) in its German version (Bullinger et al., 1990). The participants rated 35 mood-related adjectives on a 7-point scale ranging from 0 ("not at all") to 6 ("extremely") based on how they had been feeling in the past week including the measurement day. While the German version of the POMS consists of four identifiable mood states (depression/anxiety, vigor, fatigue, and discontent), we defined the total mood disturbance (TMD) score as a general negative mood score calculated as the sum of the negative mood scores (depression/anxiety, fatigue, and discontent) minus the positive mood score (vigor). As previous studies showed that PFC activity correlated negatively with negative moods while tending to correlate positively with positive moods (Aoki et al., 2011, 2013), TMD would be an appropriate mood index which is expected to correlate negatively with PFC activity.

Thereafter, PFC activity during the performance of the WM tasks was measured by using a multi-channel NIRS (optical topography) system (ETG-4000, Hitachi Medical Corporation, Japan). The system uses continuous wave laser diodes with two wavelengths (695 and 830 nm) as light sources. Optical fibers are used both for irradiation and for detection of near-infrared light. Seventeen irradiation positions and 16 detection positions were arrayed in a 3 × 11 lattice pattern with 30 mm separation, forming 52 measurement channels (**Figure 1B**). The average power of each wavelength light was 1.5 mW at the irradiation point, which was modulated at different frequencies to be separated for wavelengths and sources in the detection process. The reflected light at a position 30 mm apart from the irradiation position was detected by avalanche photodiodes and recorded in the computer with a sampling rate of 10 Hz.

To obtain the anatomical position of the NIRS channels, we measured the three-dimensional coordinates of the optode positions in four German volunteers by using a neuro-navigation system (LOCALITE GmbH, St. Augustin, Germany). The obtained optode coordinates were transferred from the volunteers' native MRI space to the standard Montreal Neurological Institute (MNI) space by applying normalization procedure of Statistic Parametric Mapping 8 (SPM8, http://www*.*fil*.*ion*.*ucl*.*ac*.*uk/spm/software/ spm8/). The normalized coordinates were used in the probabilistic registration method (Okamoto et al., 2004; Okamoto and Dan, 2005) to estimate the brain region for each measurement channel and to generate 3-D topographical maps (**Figures 1**–**3**). The averaged MNI coordinates on the brain are listed with the corresponding Brodmann Area (BA) number (Rorden and Brett, 2000) and the anatomical label determined by Automated Anatomical Labeling (AAL) (Tzourio-Mazoyer et al., 2002) in the Supplementary Material (**Table S1**).

#### **DATA ANALYSIS**

For NIRS data analysis, we used MATLAB (The MathWorks, Inc., U.S.A.) and the plug-in-based analysis software Platform for Optical Topography Analysis Tools (developed by Hitachi, CRL).

For each NIRS channel, the optical data for the two wavelengths were transformed into a time series of hemoglobin (Hb) signals (oxy-Hb and deoxy-Hb signals) on the basis of the modified Beer–Lambert law (Delpy et al., 1988; Maki et al., 1995). These signals were expressed as the product of the changes in hemoglobin concentration (mM) and optical path length (mm) in the activation region (effective optical path length). A butterworth band-pass filter (0.013–0.80 Hz) was then applied to the Hb signals to remove low-frequency drift/oscillation and high-frequency system noise. The time-continuous data of Hb signals were divided into 25.5-s task blocks, which consisted of a 1-s pre-task period (starting 1 s before Target onset), an 8.5 s task period (during the 1.5-s Target presentation and the 7-s delay period) and a 16-s post-task period (starting after Test onset). Each task block was baseline-corrected by subtracting the average value of the pre-task period. Here, we especially focused on the oxy-Hb signal in the analysis as previous studies underlying the present work found their main effects in this chromophore (Aoki et al., 2011, 2013; Sato et al., 2011). Note that we did not include data from error-response trials for the further analysis.

To evaluate WM-associated PFC activity, we calculated two kinds of activation values for each channel and participant. The first activation value (Act\_base) contrasts the mean oxy-Hb signal values during a 6-s "activation period" (starting 5 s after Target onset) with the baseline. The activation period was determined to investigate the signal change induced by the encoding and maintenance processes in WM function, avoiding confounding effects related to the motor response and proactive interference (D'Esposito et al., 2000). These oxy-Hb signal values were then averaged across correct-response trials and divided by their standard deviation across trials. The second activation value (Act\_ctl) contrasts WM activation with the control task: The differences between the mean values for a WM task minus those for the control task were averaged across correct-response trials and divided by the standard deviations across trials. These activation values, which take the within-participant reproducibility into consideration, should be more reliable than a mean value from a simply-averaged waveform which is easily influenced by an accidental large response-like change in a single trial.

In the subsequent group analysis, we performed channel-wise statistical tests on the activation values to determine the activation regions. First, significant PFC activation relative to the baseline was assessed by using one-sample *t*-tests (one-tailed) of Act\_base in all 52 channels. Next, significant PFC activation relative to the control task was assessed in a similar way using Act\_ctl for the

**FIGURE 2 | Hemodynamic responses to the verbal and spatial WM tasks. (A)** Activation t-maps of oxy-Hb signal increase during WM tasks compared to the baseline (zero level). **(B)** Activation t-maps of oxy-Hb signal increase during WM tasks compared to the signal during the control task. In both maps, the Student's *t*-value is indicated by a color scale for channels which

meet the *p*-value criteria described below each map. **(C)** Time courses of Hb-signal in a representative channel (Ch 25). These time courses represent the grand average across all participants (*N* = 26) with standard error (transparent color region). The green vertical lines show the time of (a) Target onset and (b) Test onset, respectively.

activated channels determined by the former analysis in Act\_base. In these channel-wise analyses, we used the false discovery rate (FDR) method to correct for multiple comparisons among channels (Benjamini and Hochberg, 1995; Singh and Dan, 2006). Based on the activation map for Act\_ctl, we selected significantlyactivated channels as regions of interests (ROIs). In each ROI, correlation coefficients between mood states (TMD scores) and activation values were calculated using Spearman rank correlation coefficients (*rho*) as the association may not be necessarily linear. Here, we tested significance of the negative correlation (onetailed) with the FDR correction for multiple comparisons for the number of ROIs. To further examine whether the correlations were influenced by confounding factors such as age, gender, and task performance (accuracy and reaction time), we performed partial correlation analyses. In addition, correlation coefficients between behavioral data (accuracy and reaction time) and the TMD score were also examined to consider mood effects on the behavioral level.

#### **RESULTS**

#### **BEHAVIORAL DATA**

The accuracy and RT data are shown in **Table 1**. Two-Way repeated-measures ANOVAs were used to examine the effects of WM load (WM task/control task) and content (verbal/spatial). For accuracy, a main effect of WM load was found [*F(*1*,* <sup>25</sup>*)* = 11*.*82, *p <* 0*.*01], which indicates that the control tasks were easier to solve, as expected. In contrast to that, no significant results were obtained for content [*F(*1*,* <sup>25</sup>*)* = 0*.*41, *p* = 0*.*53] or the interaction of WM load × content [*F(*1*,* <sup>25</sup>*)* = 0*.*37, *p* = 0*.*55]. For RT, a main effect of WM load was found [*F(*1*,* <sup>25</sup>*)* = 49*.*11, *p <* 0*.*001] without a main effect of content [*F(*1*,* <sup>25</sup>*)* = 0*.*38, *p* = 0*.*54], which indicates that both WM tasks were associated with longer reaction times than the control tasks. A significant interaction was also found [*F(*1*,* <sup>25</sup>*)* = 10*.*53, *p <* 0*.*01], indicating that the RT difference between the WM and control tasks was larger for the verbal WM condition (mean difference in the RTs: 161 ms) compared to that for the spatial WM condition (mean difference in the RTs: 65 ms).

The correlation coefficients between the accuracies and TMD scores were not significant (verbal WM: *rho* = 0*.*09, *p* = 0*.*65; spatial WM: *rho* = 0*.*25, *p* = 0*.*22). The correlation coefficients between the RTs and TMD scores were also not significant, but displayed a tendency toward a negative association (verbal WM task: *rho* = −0*.*33, *p* = 0*.*10; spatial WM task: *rho* = −0*.*34, *p* = 0*.*09), i.e., participants who had higher TMD scores showed lower RTs (faster responses) for both WM tasks.

#### **PFC ACTIVITY DURING WM TASKS**

Activation maps in the group analysis are shown in **Figure 2**. For the verbal WM task, the increase in oxy-Hb signal relative to baseline was significant in 11 channels in the bilateral PFC (*p <* 0*.*05, FDR corrected for 52 channels). Four of these channels also showed significant activation in contrast to the v-control task (*p <* 0*.*05, FDR corrected for 12 channels). The representative time courses (ch25) shown below the activation map (**Figure 2**) indicate clear responses in the oxy-Hb signal after the presentation of Target stimuli, where the responses for the WM task were larger than those for the control task.

This activation pattern was not evident for the spatial WM task, where no significant increases in oxy-Hb signal relative to the baseline were found when the threshold was corrected for the number of channels. Although the uncorrected activation maps indicated oxy-Hb signal increases in the PFC region relative to the baseline (similar to the result for the verbal WM task), no significant differences were found when contrasted with the s-control task (**Figure 2**). While the time courses (ch25) showed a similar response to the Target stimuli as shown in the verbal WM condition, the standard error (inter-participant variation) was larger and there was no clear difference from the response to the control task. As activation did not differ between WM and control task, we did not include the spatial WM data in the subsequent correlation analysis.

#### **CORRELATION OF MOOD STATES WITH PFC ACTIVITY FOR THE VERBAL WM TASK**

TMD scores measured by POMS ranged from −37 to 51 across participants (Mean = −3*.*65, *SD* = 19*.*36). To examine the relationship between the negative mood state and PFC activity for the verbal WM task, the correlation coefficients between TMD scores and activation values were calculated for the ROIs. Here, we defined the WM-related activation channels determined by contrasting to the control task (**Figure 2B**) as the ROIs (**Figure 3A**), and the mean activation values of the two channels (channels 18 and 29) were used in the correlation analysis for ROI-1.

The results of the correlation analysis are summarized in **Table 2**. In ROI-1 in the left PFC, which consists of channels 18 ("Frontal\_Mid\_L," BA45) and 29 ("Frontal\_Inf\_Tri\_L," BA45), both Act\_base (activation value relative to baseline) and Act\_ctl (activation value relative to the v-control task) showed significant negative correlations with TMD scores (*p <* 0*.*05, FDR corrected for 3 ROIs) (**Table 2** and **Figures 3B,C**). These correlations remained significant even when the control variables (gender, age, accuracy and RT for the verbal WM task) were entered into a partial correlation analysis (Act\_base: *rho* = −0*.*50, *p* =*<* 0*.*01; Act\_ctl: *rho* = −0*.*42, *p <* 0*.*05). Further, when analyzed separately by gender, females and males showed equivalent correlation coefficients for Act\_ctl


**Table 2 | Correlation coefficient (***rho***) between TMD scores and activation values in ROI.**


*Uncorrected p-values are shown for the correlation coefficients. Significant rho- and p-values (p < 0.05 with FDR correction) are indicated by bold letters.*

(female: *rho* = −0*.*45, *p* = 0*.*12; male: *rho* = −0*.*43, *p* = 0*.*15) although they did not reach statistical significance due to small sample sizes. On the other hand, ROI-2 in the right PFC (channel 25, "Frontal\_Mid\_L," BA46) also showed a significant negative correlation for Act\_base (*p <* 0*.*05, FDR corrected for 3 ROIs) but not for Act\_ctl (*p >* 0*.*05, FDR corrected for 3 ROIs). In addition, ROI-3 in the left temporal pole (channel 51, "Temporal\_Pole\_Sup\_L," BA38) did not show any significant correlation for either Act\_base or Act\_ctl (*p >* 0*.*05, FDR corrected for 3 ROIs).

As the waveform of oxy-Hb signals showed a bi-phasic pattern (**Figure 2C**), we additionally conducted the same correlation analysis using different activation values for the second peak (using activation period: 11–17 s after Target onset) (**Table S2**). As a result, the activation value relative to the baseline (Act2\_base) in the left PFC (ROI-1) still showed a significant negative correlation (*p <* 0*.*01, FDR corrected for 3 ROIs) while the activation value relative to the control task (Act2\_ctl) did not reach statistical significance (*p >* 0*.*05, FDR corrected for 3 ROIs). Activation values in ROI-2 and ROI-3 showed the same tendency of negative correlations with no statistical significance.

#### **DISCUSSION**

In the present NIRS study, we aimed to confirm and extend the generalizability of findings about an association between negative mood states and PFC activity for a verbal WM task as demonstrated in Japanese samples (Aoki et al., 2011, 2013). We could show that negative mood (TMD score evaluated by POMS) was negatively correlated with left-hemisphere PFC activity during a verbal WM task in a German sample. This was also evident when subtracting activation in the v-control task from the WM task. That is, participants with higher TMD scores showed less PFC activity during the verbal WM task, which is consistent with the above-mentioned findings. To our knowledge, this is the first study that replicates this relationship in a sample with a different language background, suggesting that the mood–cognition interaction in the PFC represents a general phenomenon that is reproducible across samples with different cultural and ethnical background.

#### **PFC ACTIVITY DURING WM TASKS**

The oxy-Hb signals increased during the verbal WM task in the bilateral PFC (**Figure 2**). The activated cortical region and the shape of Hb-signal changes do not only resemble those in the published studies (e.g., Aoki et al., 2011), but also those in corresponding fMRI studies (D'Esposito, 2007). The contrast with the control task showed more localized activation around the bilateral middle frontal gyrus (BA46) and the triangular part of left inferior frontal gyrus (BA45) consistent with available PET/fMRI studies (Petrides et al., 1993; D'Esposito et al., 1995; Swartz et al., 1995; Smith et al., 1996; Smith and Jonides, 1999; Walter et al., 2003). This result suggests that the control task was well-chosen to capture the WM relevant PFC regions. Although some studies suggested that extracranial hemodymanic changes such as skin blood flow considerably influence the forehead NIRS signals (Takahashi et al., 2011; Kirilina et al., 2012), a simultaneous NIRS-fMRI study demonstrated that in a comparable WM task Hb-signals were significantly correlated with gray matter BOLD signal rather than with soft tissue or skin blood flow signals (Sato et al., 2013).

In contrast to the verbal WM task, the increase in the oxy-Hb signal for the spatial WM task was not significant in a conservative analysis with FDR correction, although we observed a similar activation region when compared to the baseline without multichannel correction (**Figure 2**). This result was not consistent with our previous NIRS studies which showed the same activation for the spatial as in the verbal WM task (Aoki et al., 2011). As the behavioral accuracy and RT data did not reveal differences between tasks, a difference in difficulty cannot account for this discrepancy. However, the RT data indicates that the difference between WM and control task was less pronounced for the spatial WM task, which could explain the absence of a significant finding for spatial WM. In addition, as the four circle positions in the Target image were fixed for the s-control task to minimize WM load, it may have been seen as a totally different task by the participant. That is, both the WM task and the control task could activate regions involved in "set-shifting" (Konishi et al., 1999).

#### **CORRELATION OF MOOD STATES WITH PFC ACTIVITY FOR THE VERBAL WM TASK**

We found a significant negative correlation of the TMD score with both Act\_base (activation relative to the baseline) and Act\_ctl (activation relative to the v\_control task) for the verbal WM task in the left PFC (around BA45), as found in the aforementioned NIRS studies, demonstrating the reproducibility of the relationship between negative mood and WM-dependent PFC activity. This result extends our knowledge for the following reasons: First, though to a lesser extent, it was also found for Act\_ctl, which suggests that the main component of the correlated NIRS signal indeed stems from a verbal WM function rather than general task demands (e.g., visual/auditory inputs and execution of the motor response). Second, the significant correlation was demonstrated in a localized activation area within the left PFC, which overlaps with the main verbal WM-related regions identified by other modalities (Petrides et al., 1993; Smith et al., 1996; Walter et al., 2003). This can be interpreted as evidence against a critical distortion of our findings by confounding skin blood flow changes which are supposed to appear across large parts of the measurement array (Kohno et al., 2007). According to the "prefrontal asymmetry" hypothesis (Davidson, 1992), positive (approach-related) and negative (withdrawal-related) affect are predominantly associated with the left (verbal function) and right (spatial) hemisphere, respectively (Gray, 2001; Gray et al., 2002). Based on this idea, our results (i.e., decreased left PFC activity) might reflect a reduced positive mood state rather than increases in negative mood. Third, we could confirm the replication of our previous findings in a sample with a different language background. This suggests that the relationship between mood and verbal WM function is general regardless of language. Previous studies have shown cultural influences on the brain activation related to some social cognitive tasks (Chiao et al., 2010; Chiao and Immordino-Yang, 2013). However, up to now, no studies have shown such cultural influences on the lateral PFC during emotionally-neutral cognitive tasks such as our WM tasks, as far as we know. This might indicate that PFC activity for emotionally-neutral cognitive tasks is independent of cultural backgrounds. Although we could not directly compare the activation pattern between Japanese and German samples, cultural differences in PFC activity during emotionally-neutral cognitive tasks might be an important topic to determine the range of cultural effects on the brain. In addition, it should be emphasized that we could also replicate the significant correlation using partial correlation analysis with control variables of gender, age, and task performance. This indicates that the mood–cognition relationship in the PFC cannot be explained by any of the control variables.

Our results are basically in line with an fMRI study by Qin et al. (2009) who demonstrated that experimentally-induced acute stress was accompanied by a decrease in verbal WM-related DLPFC activity. In addition, both studies did not reveal correlational patterns with behavioral data, possibly due to ceiling effects of the task performance. This means that it is not clear whether the negative mood states hinder the PFC function or improve its efficiency. To increase the number of task trials or to make the task more difficult would be a possible solution to reveal the relationship between behavioral measures and mood states. Actually, the previous study showed that participants with the strongest stress response slowed down the most (supporting a hindrance of the PFC) (Qin et al., 2009), while our results indicated a tendency for participants with higher negative mood scores to show faster responses (supporting an improvement of PFC function). Nevertheless, these results suggest that decreased PFC activity can be a neural marker of heightened stress or increased negative mood states and seems to be more sensitive than behavioral performance.

#### **LIMITATION AND FUTURE PERSPECTIVES**

The first limitation of the present study concerns a failure to detect PFC activation for the spatial WM task using a corrected significance threshold. As the previous NIRS study showed a significant correlation between mood and PFC activity only for the verbal WM task but not for the spatial WM task while the two tasks induced almost equivalent PFC activities (Aoki et al., 2011), it would be worth investigating the difference of the two tasks in order to further identify the cognitive factors underlying the mood–cognition interactions.

Next, we could not directly compare the basic NIRS results between German and Japanese samples, because some differences in methods and theoretical considerations did not allow such a procedure. First, differences in the applied paradigms such as the control task and the repetition time restrict comparability and would make it difficult to attribute any potential group differences to a specific factor. Additionally, results from different NIRS machines should not be directly compared as general equivalence in technical parameters, etc. of the machines had not been tested. Second, a direct comparison was not intended as no hypotheses exist why and how these samples differ. Indeed, the main purpose of this study was to replicate the relationship between PFC activity and mood scores, which was successfully done regardless of some cultural/ethnical differences and task modifications. Only testing the samples using exactly the same paradigm parameters and the same NIRS machine would help to systematically explore potential differences between both samples.

Finally, the present study aimed at replicating previous NIRS findings and was not specifically designed to further explain the cognitive or physiological mechanisms underlying the observed mood–PFC relationship. Indeed, we found a similar correlation pattern for the second peak of PFC activation corresponding to the retrieval phase (**Table S2**), while focussing on the activity for encoding and maintenance phases. This might suggest the necessity to resolve the mood-related WM function into cognitive sub-processes in the future. Furthermore, it is necessary to investigate the effects of some other physiological parameters on the mood–PFC correlation to advance our knowledge about the causal relationship between natural mood and PFC activity.

#### **CONCLUSION**

The present study aimed at replicating a previous finding on an interaction between mood states and PFC activity for a verbal WM task, which was found by a NIRS study in a Japanese sample. Conducting a comparable experiment in a German sample, we could indeed confirm this finding; negative mood scores were negatively correlated with verbal WM-related PFC activity across participants, even when the activation values were contrasted with an adequate control task. Although we failed to observe reliable PFC activity for the spatial WM task, it is important to note that the relationship between negative mood and PFC activity for the verbal WM task could be confirmed in a sample with a different language background. Our results suggest that the mood–cognition interaction within the PFC is reproducible and represents a general phenomenon.

#### **ACKNOWLEDGMENTS**

We thank Betti Schopp and Ramona Taeglich for their proficient technical assistance. We also thank Agnes Kroczek, Beatrix Barth, Florian Metzger, Katja Hagen, Sabrina Schneider, and Saskia Deppermann for their helpful support. Thomas Dresler was partly supported by the LEAD graduate school [GSC1028], a project of the Excellence Initiative of the German federal and state governments.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www*.*frontiersin*.*org/journal/10*.*3389/fnhum*.* 2014*.*00037/abstract

**Table S1 | Estimated location of each NIRS channel on normalized brain image. Mean MNI coordinates across participants are shown with the corresponding Brodmann area (BA) number (Rorden and Brett, 2000) and the anatomical label determined by Automated Anatomical Labeling (AAL) (Tzourio-Mazoyer et al., 2002).**

**Table S2 | Correlation coefficient (rho) between TMD scores and activation values (Act2\_base: contrasted to baseline; Act2\_ctl: contrasted to the control task) for the second peak (activation period: 12–18 s after the Target onset).**

#### **REFERENCES**

Aoki, R., Sato, H., Katura, T., Matsuda, R., and Koizumi, H. (2013). Correlation between prefrontal cortex activity during working memory tasks and natural mood independent of personality effects: an optical topography study. *Psychiatry Res.* 212, 79–87. doi: 10.1016/j.pscychresns.2012.10.009


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 22 October 2013; accepted: 17 January 2014; published online: 06 February 2014.*

*Citation: Sato H, Dresler T, Haeussinger FB, Fallgatter AJ and Ehlis A-C (2014) Replication of the correlation between natural mood states and working memoryrelated prefrontal activity measured by near-infrared spectroscopy in a German sample. Front. Hum. Neurosci. 8:37. doi: 10.3389/fnhum.2014.00037*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Sato, Dresler, Haeussinger, Fallgatter and Ehlis. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Negative emotion modulates prefrontal cortex activity during a working memory task: a NIRS study

#### *Sachiyo Ozawa\*, Goh Matsuda and Kazuo Hiraki\**

*Graduate School of Arts and Science, The University of Tokyo, JST, CREST, Tokyo, Japan*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Satoru Otani, Ryotokuji University, Japan Shintaro Funahashi, Kyoto University, Japan*

#### *\*Correspondence:*

*Sachiyo Ozawa and Kazuo Hiraki, Graduate School of Arts and Science, The University of Tokyo, Komaba 3-8-1, Meguro-ku, Tokyo 153-8902, Japan e-mail: ozawa@ ardbeg.c.u-tokyo.ac.jp; khiraki@idea.c.u-tokyo.ac.jp*

This study investigated the neural processing underlying the cognitive control of emotions induced by the presentation of task-irrelevant emotional pictures before a working memory task. Previous studies have suggested that the cognitive control of emotion involves the prefrontal regions. Therefore, we measured the hemodynamic responses that occurred in the prefrontal region with a 16-channel near-infrared spectroscopy (NIRS) system. In our experiment, participants observed two negative or two neutral pictures in succession immediately before a 1-back or 3-back task. Pictures were selected from the International Affective Picture System (IAPS). We measured the changes in the concentration of oxygenated hemoglobin (oxyHb) during picture presentation and during the n-back task. The emotional valence of the picture affected the oxyHb changes in anterior parts of the medial prefrontal cortex (MPFC) (located in the left and right superior frontal gyrus) and left inferior frontal gyrus during the n-back task; the oxyHb changes during the task were significantly greater following negative rather than neutral stimulation. As indicated in a number of previous studies, and the time courses of the oxyHb changes in our study, activation in these locations is possibly led by cognitive control of emotion, though we cannot deny it may simply be emotional responses. There were no effects of emotion on oxyHb changes during picture presentation or on n-back task performance. Although further studies are necessary to confirm this interpretation, our findings suggest that NIRS can be used to investigate neural processing during emotional control.

**Keywords: NIRS, prefrontal activity, DLPFC, cognitive control, emotion regulation, working memory, n-back task, IAPS**

#### **INTRODUCTION**

A number of previous studies have suggested that emotion is regulated by cognitive processing (Dolcos et al., 2011; Cromheeke and Mueller, 2013). Studies of the cognitive control of emotion or emotion regulation have utilized neuroimaging methods, which are a useful tool that provide objective and precise indices of subjective emotional experiences. A large number of neuroimaging studies have recently been conducted in order to examine the neural functions underlying the cognitive control of emotion (Hare et al., 2005; Erk et al., 2006; Goldin et al., 2008; Adrian et al., 2011). According to these neuroimaging studies, the neural function of emotion regulation involve interactions between the *hot* emotional system in the limbic regions that involves the amygdala and *cold* higher order systems in prefrontal regions (Gray et al., 2002; Dolcos et al., 2011). More concretely, emotional control involves neural interactions that decrease the activity in the amygdala while increasing prefrontal activity. For example, Erk et al. (2006) have shown that working memory tasks downregulate amygdala activity induced by the anticipation of subsequent negative emotions. Ochsner et al. (2002) have indicated that amygdala activity is downregulated by reappraisals, when participants willfully no overrode their negative feelings toward the pictures. Van Dillen et al. (2009) have demonstrated that arithmetic tasks with a higher cognitive load downregulate the increased amygdala activity induced by a prepresented negative stimulus, and release negative emotions compared to arithmetic tasks with a lower cognitive load. Thus, cognitive processing associated with prefrontal enhancements has been shown to decrease activity in the limbic system. At the same time, the negative emotional state is thought to be relieved.

The cold, higher order, system includes brain regions typically associated with executive functions (Dolcos et al., 2011). One of these executive functions is working memory, which contributes to the maintenance of information relevant to current tasks over short periods of time (Baddeley and Della Sala, 1996). During the engagement of a working memory task, brain activity increases in frontal regions that include the dorsolateral prefrontal cortex (DLPFC) (Owen et al., 2005). Moreover, a number of studies have indicated that the cognitive control of emotion also activates various prefrontal regions, such as the orbitofrontal cortex (OFC; Banks et al., 2007), the medial prefrontal cortex (MPFC; Ochsner et al., 2002), and the ventrolateral prefrontal cortex (VLPFC; Lévesque et al., 2003). In particular, activity in the DLPFC has been consistently observed in a number of studies (Beauregard et al., 2001; Lévesque et al., 2003). DLPFC activity has been thought to be indirectly and reciprocally associated with activity in limbic systems through connections in the OFC (Cavada et al., 2000). The DLPFC appears to be the shared region for both working memory and emotional regulation.

In this study, we investigated whether task-irrelevant emotional stimuli elicited differential neural activity during a subsequent working memory task in order to investigate the prefrontal activity underlying the cognitive control of emotion. Previous findings (Dolcos et al., 2011) have suggested that task-irrelevant emotional stimuli can interfere with the maintenance of goalrelevant information and thus serve as a distracter in a cognitive task. A distracter can be presented during (Dolcos and McCarthy, 2006) or prior to (Deckersbach et al., 2008; Van Dillen et al., 2009; Hart et al., 2010) a working memory task. According to Dolcos and McCarthy (2006), when an emotional distracter is presented during a working memory task, DLPFC activity may be attenuated by the distracter and cognitive performance may be impaired. Only a few studies have been conducted with a prepresented distracter. For example, Van Dillen et al. (2009) have found greater activity in the right DLPFC in response to negative pictures than neutral pictures during an arithmetic task following 4 s of emotional picture presentation. A higher cognitive load also elicited greater activity in the right DLPFC. In addition, Hart et al. (2010) have examined the effects of emotional priming on a subsequent Stroop task. In that experiment, a negative or neutral picture was primed for 150 ms, and a Stroop task was then presented. Consequently, an attenuating effect was observed in response to the negative pictures during a congruent condition, but it was not observed during an incongruent condition. Hart et al. (2010) have explained that the need for cognitive processing overrode the deactivating effect of negative stimuli. As mentioned above, Van Dillen et al. (2009) have reported prefrontal activation during a cognitive task after emotional stimulation, and this was in contrast to Hart et al. (2010). The discrepancy of their results may partially depend on the duration of emotional stimulation. On the other hand, both of these studies have reported more activation in the prefrontal cortex during a task with a high cognitive load than that with low cognitive load. Therefore, emotional stimulation for at least 4 s, as in Van Dillen et al. (2009), and a task with a high cognitive load seem to be necessary to investigate the neural correlates of the cognitive control of emotion.

In order to measure prefrontal activity, we employed nearinfrared spectroscopy (NIRS). The attachment of the NIRS system is easy, and it imposes little strain on the participants. If the cognitive control of emotion can be captured by NIRS, emotion regulation studies will be easily conducted in a wide variety of participants, including psychiatric patients and children, without imposing a strain on them. Although NIRS is unable to measure deep activity in the brain, such as in the limbic system, several NIRS studies have recently attempted to capture emotion-related activity in prefrontal regions (Herrmann et al., 2008; Dieler et al., 2010; Hoshi et al., 2011). For example, Hoshi et al. (2011) have observed that strong unpleasant pictures activate the bilateral VLPFC, while strong pleasant pictures deactivate the left DLPFC. Dieler et al. (2010) have shown more activation in the right DLPFC and right VLPFC during attempts to suppress thoughts with negative content than those with neutral or positive contents.

In our experiment, negative and neutral pictures were selected from the International Affective Picture System (IAPS; Lang et al., 1998). In order to induce strong emotional response, a series of two pictures of the same emotional valence were presented in succession for approximately 10 s. For cognitive processing, an n-back working memory task was employed. The n-back task is one of the most prevalent paradigms used in the assessment of working memory. We employed two levels of cognitive load (a high cognitive load or a low cognitive load) because cognitive load can modulate the processing of negative emotion (Erk et al., 2007; Van Dillen et al., 2009). Thus, in the experiment, participants observed two negative or two neutral pictures in succession and then performed a 1-back or 3-back task. Analyses of hemodynamic responses were conducted independently for the picture presentation and n-back task.

We anticipated that, during picture presentation, prefrontal activation would be significantly greater in response to negative pictures than in response to neutral pictures in line with the results of Hoshi et al. (2011). For the n-back task period, the prefrontal region that includes the DLPFC would be activated after negative stimulation relative to neutral stimulation. As mentioned above, the DLPFC and other prefrontal regions have been shown to exhibit general activation during the cognitive control of emotion (Beauregard et al., 2001; Lévesque et al., 2003). In addition, because a previous study has reported that a high cognitive load downregulates amygdala activity significantly more than a low cognitive load (Van Dillen et al., 2009), we anticipated that cognitive processing with a high cognitive load (3-back) would increase prefrontal activity significantly more than that with a low cognitive load (1-back). The cognitive performance may be impaired in response to the distraction of negative emotional stimulation (Dolcos and McCarthy, 2006).

#### **MATERIALS AND METHODS**

#### **PARTICIPANTS**

Twenty healthy male undergraduate students (mean age: 19.38 ± 0.79 years) participated in this study. All participants were righthanded. None of them had with any history of psychiatric or neurological disorders. The study conformed to the Declaration of Helsinki and was approved by the Ethics Committee of the University of Tokyo. All participants were given a comprehensive explanation of the experimental procedures by an experimenter, and they provided written informed consents for participation.

#### **EXPERIMENTAL ENVIRONMENT**

The experiment was performed in a shielded room. Participants were instructed to sit on a chair in front of a 17-in Cathode Ray Tube monitor (32 × 24 cm). The distance between a participant and the monitor was set to around 80 cm. The NIRS equipment (Spectratech OEG-16; Spectratech Inc., Yokohama, Japan) was attached to their heads. The experimental task was implemented in dim light.

#### **EMOTIONAL STIMULI**

Thirty-two neutral and 32 negative pictures were selected from the IAPS (Lang et al., 1998) based on a 9-point rating scale of the IAPS normative data. The average emotional valence rating of the selected pictures was 5.10 for the neutral pictures and 2.00 for the negative pictures, and the average arousal rating was 3.18 for the neutral pictures and 6.00 for the negative pictures. The arousal ratings of all of the selected negative pictures were above 5.5. Examples of the neutral pictures selected were humans, plants, food, and materials, and examples of the negative pictures were victims, mutilations, insects, and dirty toilets. Pictures with obscure content were not selected.

#### **n-BACK TASK**

The 1-back and 3-back tasks were implemented with the digits 1–9. White-colored digits with a size of 12.70 mm were individually displayed in the middle of the black background, in sequence. The participants were instructed to press the Enter key as quickly as possible only when the current digit matched the one from one step or three steps earlier for the 1-back or 3-back, respectively. A total of 13 digits were presented in a sequence with 3 matches included.

#### **EXPERIMENTAL PROCEDURES**

At the beginning of the experiment, participants were instructed to avoid head and body movements and deep breathing as much as possible during the NIRS measurement. The state of the participants was video-monitored and no salient head and body movement or deep breathing was detected.

The experiment consisted of two successive sessions: a NIRS measurement session and a valence rating session. First, the NIRS equipment was attached to the participants' heads. Each trial in the NIRS measurement session included five periods (**Figure 1**). In the first resting period, a white fixed cross was displayed in the middle of the screen for 9.8 s. The participant was instructed to gaze at it calmly. In the picture presentation period, two IAPS pictures with the same emotional valence (neutral or negative) were sequentially presented for 5.2 s per picture without an interval. Soon after the picture presentation, white letters that read *1-back task* or *3-back task* were presented on the black background for 1.3 s (instruction period). In the n-back period, a sequence of digits was presented for 26.0 s. Each digit was presented on the screen for 300 ms, which was followed by a 1,700-ms interstimulus interval. After the n-back period, a fixed cross was presented for 5.2 s (the last resting period). There were six practice trials with only neutral pictures and 32 experimental trials with both neutral and negative pictures in the NIRS measurement session. At the end, the NIRS equipment was removed, and the participant rated the emotional valence of all of the pictures used in the experiment with a digital scale with nine grades from 1 (unpleasant) to 9 (pleasant).

#### **BEHAVIORAL ANALYSIS**

The accuracy and reaction time of the n-back task were used as parameters in the behavioral analysis. The accuracy was determined by the percentage of correct responses. The reaction time was the time from the disappearance of a digit to the participants pressing the Enter key.

In order to examine the effects of emotion on the performance of the cognitive task, a 2 (emotional valence: neutral or negative) × 2 (cognitive load: 1-back or 3-back) repeated measures ANOVA with within-subjects factors was performed.

#### **NEAR-INFRARED SPECTROSCOPY RECORDING AND ANALYSIS** *Near-infrared spectroscopy recording*

We used the Spectratech OEG-16, which consisted of six emission probes and six detection probes. This system is able to obtain three parameters of the concentration changes of hemoglobin: oxygenated (oxyHb), deoxygenated (deoxyHb), and total (totalHb) changes. The measures of the hemoglobin changes were obtained from the 16 channels. The emission probe and detection probe were 3 cm apart from each other, and all of the probes were set in a 15 × 3-cm matrix area. The Spectratech OEG-16 employs two wavelengths, which are approximately 770 and 880 nm, to record the absorption changes of hemoglobin at a depth approximately 2 cm below the scalp (Watanabe et al., 2000). The sampling interval was 655 ms. The center of the probe holder was placed on Fpz in the International 10/20 System (**Figure 2**). Our measurement area covers Fp1, Fpz, Fp2, F7, and F8, and regions a little inferior to F3 and F4. According to Okamoto et al. (2004), Fp1 and Fp2 are located on the left and right superior frontal gyrus, respectively. F7 and F8 are placed in the left and right inferior frontal gyrus, respectively. F3 and F4 are located in the inferior part of the DLPFC.

#### *Near-infrared spectroscopy analysis*

We examined the concentration changes of oxyHb in the analysis because it is the most sensitive measure to changes in regional cerebral blood flow (Hoshi et al., 2001), and it has the

strongest correlation with blood-oxygen-level-dependent signals (Strangman et al., 2002).

The raw oxyHb data were high-pass filtered (0.0076 Hz). Each channel was analyzed individually. All sampling data was segmented to each trial and converted to a z-score with the mean value and the standard deviation (*SD*) of oxyHb change during the last 4.59 s (7 samples) of the first resting period used as a baseline. Thus, the mean and *SD* of the baseline were changed into the z-scores of 0 and 1 (Matsuda and Hiraki, 2006). For artifact rejection, all difference values between the sampling data and their *SD*s were calculated. The trials that included at least one difference value over ±8 *SD* were excluded from the statistical analysis.

The prefrontal activities during the picture presentation and n-back task were separately analyzed by conducting both individual and group analyses. In the picture presentations, the oxyHb changes resulting from the neutral or negative picture presentation were analyzed. For the n-back task, the oxyHb changes of the 1-back and 3-back tasks after neutral picture presentation (neutral 1-back, neutral 3-back, respectively) and 1-back task and 3-back task after negative picture presentation (negative 1-back, negative 3-back, respectively) were analyzed independently.

In individual analysis, the means and *SD*s of the oxyHb change in all trials were calculated in each of the picture presentations and the n-back task. Then, the mean oxyHb values were compared with the baseline, in which the average value was 0, with a onesample *t*-test in order to determine any significant activation.

In the group analysis of picture presentation, the means and the *SD*s of the oxyHb changes of all participants were calculated. The mean oxyHb values were compared with baseline, in which the average value was 0, with a one-sample *t*-test. In addition, Student's *t*-test was performed between the mean oxyHb values of negative and neutral picture presentations to

examine the differences in prefrontal activity between emotional valences.

In the group analysis of the n-back task, the means and *SD*s of oxyHb change of all participants were calculated. A 2 (emotional valence: neutral or negative) × 2 (cognitive load: 1-back or 3-back) repeated measures ANOVA with within-subjects factors was performed.

#### **RESULTS**

#### **BEHAVIORAL RESULTS**

#### *Rating of the emotional valence*

In order to confirm whether the pictures used for the emotional stimulation in this study had significantly different emotional valence ratings, *t*-tests were performed. Neutral pictures were significantly different from negative pictures in their emotional valence ratings [mean valence scores: neutral, 4.21; negative, 1.96; *t(*19*)* = 10*.*07, *P <* 0*.*001].

#### *Behavioral performance*

A 2 (emotional valence: neutral or negative) × 2 (cognitive load: 1-back or 3-back) repeated measures ANOVA revealed no main or interaction effect for accuracy (**Table 1**) and a significant main effect of cognitive load for reaction times. A *post-hoc* analysis showed that reaction times in the 1-back task were significantly faster than those in the 3-back task (mean reaction time: 1-back, 246.32 ms; 3-back, 363.77 ms, *P <* 0*.*001). There was no other main or interaction effect for reaction times.

#### **NEAR-INFRARED SPECTROSCOPY**

#### *oxyHb changes during picture presentation*

The result of the individual analysis is shown in **Table 2**. For neutral picture presentations, significant oxyHb increases were observed in seven of 20 participants (35%) while significant oxyHb decreases was found in three participants (15%). For the negative picture presentations, there were nine of 20 participants (45%) who showed significant oxyHb increases while six participants (30%) showed significant oxyHb decreases. Channels that showed significant oxyHb changes were also inconsistent between participants.

**Table 3** displays the results of the group analysis. It represents the means and *SD*s of oxyHb changes of all participants in each channel. A one-sample *t*-test revealed significant activation in three channels during the negative picture presentation compared to baseline. In contrast, significant activation was found in 12 channels in response to the neutral picture presentation.

There were no significant differences found in any of the channels between the mean oxyHb changes during neutral and negative picture presentation.

#### **Table 1 | Means and standard deviations of accuracy and reaction time of response.**



#### **Table 2 | Channels that showed significant oxyHb changes during picture presentation (individual analysis).**

*Numerical numbers indicate channel numbers. The mark "-" indicates no significant oxyHb changes observed in any of the channels.*


*\*P < 0.05; \*\*P < 0.01; \*\*\*P < 0.001.*

#### *oxyHb changes during the n-back task*

The results of the individual analysis of the n-back task were as follows. In the negative 1-back, 11 of 20 participants (55%) exhibited a significant oxyHb increase while one participant (5%) showed a significant oxyHb decrease. In the negative 3-back, a significant oxyHb increase was also observed in 11 of 20 participants (55%) while significant oxyHb decreases occurred in two participants (10%). **Table 4** displays the results of the negative 1-back and negative 3-back tasks. In contrast, there were no participants who


**Table 4 | Channels that showed significant oxyHb changes during the n-back task (individual analysis).**

*Numerical numbers indicate channel numbers. The mark "-" indicates no significant oxyHb changes observed in any of the channels. The results of the neutral 1-back and neutral 3-back were omitted because there were no significant oxyHb changes observed.*

showed significant oxyHb increases or decreases in either neutral n-back task.

In the group analysis of the n-back task, a 2 (emotional valence: neutral or negative) × 2 (cognitive load: 1-back or 3-back) repeated measures ANOVA indicated significant main effects of emotional valence in channels 6 [*F(*1*,* <sup>19</sup>*)* = 6*.*31; *P* = 0*.*021], 9 [*F(*1*,* <sup>19</sup>*)* = 5*.*84; *P* = 0*.*026], 12 [*F(*1*,* <sup>19</sup>*)* = 6*.*35; *P* = 0*.*021], and 15 [*F(*1*,* <sup>19</sup>*)* = 12*.*05; *P* = 0*.*003] (**Table 5**; **Figure 3**). A Bonferroni *post-hoc* analysis revealed significantly more activation in the n-back task after negative picture presentations than in the n-back task after neutral picture presentation in all of the channels. No effects of cognitive load were found in any of these channels. In addition, no interaction effect of emotional valence or cognitive load was found.

**Figure 4** shows the time courses of oxyHb changes in channels 6, 9, 12, and 15 where the significant main effects of emotional valence were observed during the n-back task period. Each waveform represents a time course of one trial. It was constructed by calculating a grand average of oxyHb changes in the neutral 1-back, neutral 3-back, negative 1-back, and negative 3-back. A large difference in activation was observed between emotional valences around 6 s after the n-back task in all channels.

#### **DISCUSSION**

The present NIRS study examined the neural correlates of the cognitive control of emotion in prefrontal regions. In the experiment, task-irrelevant emotional stimuli were presented prior to a working memory task in order to produce an emotional distraction. Participants observed a series of two negative or two neutral pictures and then conducted a 1-back or 3-back task. The oxyHb changes during the picture presentation period and during the n-back period were individually analyzed in order to separate cognitive and emotional processing.

During picture presentation, there was no significant difference in brain activations between the neutral and negative stimulations though more channels showed significant activation in the neutral compared to the negative stimulation compared to baseline in group analysis (**Table 3**). This was probably because the variances of the oxyHb change of channels were relatively larger in negative compared to neutral stimulation though there was no large difference in the mean oxyHb values between the emotional valences (mean ± *SD*; neutral 0.82 ± 1.29, negative 0.83 ± 2.21). A large individual difference of prefrontal activation in response to negative (unpleasant) pictures is also observed in the findings of Hoshi et al. (2011). In their NIRS study, participants were exposed to a negative or positive (pleasant) IAPS picture for 6 s following a resting period of 14 s in order to examine the prefrontal processing of emotion. In response to negative pictures, seven of 14 participants (50%) indicated an oxyHb increase and six participants (42.9%) showed an oxyHb decrease. In a similar way, our findings showed that 45% of participants showed an oxyHb increase and 30% of the participants showed an oxyHb decrease in response to negative pictures (**Table 2**). In Hoshi et al. (2011), they then conducted group analysis by


**Table 5 | The results of 2 (emotional valence) × 2 (cognitive load) repeated measures ANOVAs of the oxyHb changes during the n-back task (group analysis).**

*\*P < 0.05; \*\*P < 0.01.*

using only the data that were rated as "1" (the most unpleasant) for the pictures in order to extract responses of strong negative emotion. Consequently, the group analysis revealed significant activation in the bilateral VLPFC. By adopting only the data rated as "1," individual differences in the oxyHb changes for the negative picture probably became smaller as the mean oxyHb values became larger. Doing this would mean that a statistically significant activation might be observed for negative pictures. In this way, by reducing individual differences in the responses for emotional pictures, prefrontal processing of emotion can be captured by NIRS. A large individual difference is presumably caused by differences of personality traits and brain structures and so on. Especially, in response to negative pictures, various personality traits seem to be related to a brain activity such as amygdala reactivity to negative picture presentation and amygdala-prefrontal connectivity. For example, it is reported that the right amygdala-DMPFC connectivity for angry and fearful compared to neutral faces was positively correlated with neuroticism scores (Cremers et al., 2010). Further research regarding individual differences in emotional processing caused by variations in personality traits is necessary.

Though no significant differences in oxyHb changes between neutral and negative stimulations were exhibited during the picture presentation, significant differences by emotional valence were observed during n-back task in channel 6, 9, 12, and 15 (**Table 5**). These channels were placed close to Fp1 (channel 12), Fpz (channel 9), Fp2 (channel 6), and F7 (channel 15) in the International 10/20 System. According to Okamoto et al. (2004), Fp1, Fpz, and Fp2 are located in the left and right superior frontal gyrus and F7 is in the left inferior frontal gyrus. Among these regions, the activation in the left inferior frontal gyrus

is frequently observed as the result of the cognitive control of emotion (Cromheeke and Mueller, 2013). Thus, activation in channel 15 was presumably led by emotion regulation. Greater activation in the left hemisphere, particularly the left inferior frontal gyrus, was observed in our study. However, we did not observe the right DLPFC activation which is also relatively frequently activated by emotion regulation (Lévesque et al., 2003; Dieler et al., 2010; Cromheeke and Mueller, 2013). For example, in Van Dillen et al. (2009), the right DLPFC activation was found after the onset of the complex arithmetic task following negative picture presentations. The reason why we observed greater activation in the left hemisphere is unclear. However, it may be partially because we employed a verbal n-back task, processing of which

is known to be associated mainly with the left hemisphere (Owen et al., 2005). The difference in the cognitive task may be one factor in determining which regions are susceptible to emotional effects.

The NIRS system is unable to measure the entire region of MPFC. However, channel 12 (Fp1), 9 (Fpz), and 6 (Fp2) possibly measure the oxyHb change in the anterior part of the MPFC. MPFC (Phan et al., 2002) are the regions commonly activated in emotional responses. Thereby, the additional activations during the n-back task after negative stimulation in these channels might simply represent emotional responses induced by picture presentation, considering the delay of hemodynamic responses. On the one hand, the MPFC is also known to be associated with emotional regulation (Davidson et al., 2000). Thereby, it is possible that the NIRS system captured the cognitive control processing occurred in the anterior part of MPFC in our study. Moreover, looking at the time courses of the oxyHb changes (**Figure 4**), during the picture presentation period, both the neutral and negative picture presentation increased the oxyHb change in these channels. Only when participants performed the n-back task, prefrontal responses in these channels began to differentiate: even more rapid oxyHb increase was occurred in negative 3-back and negative 1-back compared to neutral 3-back and neutral 1-back. This rapid oxyHb increase in negative 3-back and negative 1-back was reached approximately at 6 s after the n-back task onset. This rapid activation seems to represent the effect of the n-back task implementation and not a simple emotional response. This activation appears to include both emotional and cognitive processing and implies the possibility of the cognitive control processing of emotion.

We observed the effect of negative stimulation only in brain activations during the cognitive task while there was no emotional effect observed in behavioral performance. This is because brain activation might be caused by a compensatory effort of the brain according to the processing-efficiency hypothesis (Eysenck et al., 2007). The hypothesis suggests that highly-anxious people require greater cerebral activation, the compensatory effort, for cognitive control in order to maintain the same level of performance as non-anxious people. The impairment of behavioral performance is compensated for if resources in working memory are available (Fales et al., 2008; Basten et al., 2012). Thus, there seems to be a potential ability of a healthy brain to maintain behavioral performance at a high level under an anxious state. In our study, negative pictures yielded an unpleasant emotional state, which is presumably similar to anxiety. Even under such an unpleasant state, the behavioral performance was maintained probably because we employed participants who had sufficient working memory resources. However, it is important to examine if similar tendencies can be observed using other cognitive tasks.

Our findings revealed a significant effect of cognitive load on reaction times, with responses being significantly faster during the 1-back task than during the 3-back task (mean reaction times: 1-back, 246.32 ms; 3-back, 363.77 ms, *P <* 0*.*001). Thus, the two cognitive loads actually differed in difficulty level. However, cognitive load did not significantly affect accuracy on the n-back task, and although changes in oxyHb were greater in negative and neutral 3-back tasks than in negative and neutral 1-back tasks in some channels (**Figure 4**), group analysis did not reveal any effect of cognitive load on changes in oxyHb during the n-back task (**Table 5**). In contrast, Van Dillen et al. (2009) reported that accuracy was better and reaction times faster for a simple arithmetic task than for a complex one. At the same time, brain activity was greater for the complex arithmetic task than for the simple one after negative stimulation. Significant effects of cognitive load on reaction times and brain activity may therefore be observed by employing tasks with cognitive loads that differ more greatly than ours did.

In sum, the present NIRS study clearly indicated additional activation in channels 6, 9, 12, and 15 during the n-back after negative stimulation. It thus indicated the ability of the NIRS system to capture emotion-related processing, though further research is necessary for the interpretation of this activation and the roles of personal traits for individual differences of brain activity, particularly adopting other experimental designs, cognitive tasks, and self-report questionnaires, for example. It is interesting that the emotional effect was observed only in brain activations during the n-back task but not in behavioral performance because it implies that brain activation could be an effective index of emotion regulation. By adopting the NIRS system, the study of emotion regulation can be more easily studied in various participants, such as children and psychiatric participants.

#### **ACKNOWLEDGMENTS**

We would like to thank the participants for their involvement in the experiment that induced unpleasant emotions. This study was supported by grants from JST CREST, Grant-in-Aid from MEXT (21118005) and JSPS (22240026).

#### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 September 2013; accepted: 21 January 2014; published online: 10 February 2014.*

*Citation: Ozawa S, Matsuda G and Hiraki K (2014) Negative emotion modulates prefrontal cortex activity during a working memory task: a NIRS study. Front. Hum. Neurosci. 8:46. doi: 10.3389/fnhum.2014.00046*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Ozawa, Matsuda and Hiraki. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### Sensitivity of fNIRS to cognitive state and load

### *Frank A. Fishburn1, Megan E. Norr 2, Andrei V. Medvedev3 and Chandan J. Vaidya2,4\**

*<sup>1</sup> Interdisciplinary Program in Neuroscience, Georgetown University Medical Center, Washington, DC, USA*

*<sup>2</sup> Department of Psychology, Georgetown University, Washington, DC, USA*

*<sup>3</sup> Center for Functional and Molecular Imaging, Georgetown University Medical Center, Washington, DC, USA*

*<sup>4</sup> Children's National Medical Center, Children's Research Institute, Washington, DC, USA*

#### *Edited by:*

*Nobuo Masataka, Kyoto University, Japan*

#### *Reviewed by:*

*Hasan Ayaz, Drexel University, USA Yukika Nishimura, University of Tokyo, Japan*

#### *\*Correspondence:*

*Chandan J. Vaidya, Department of Psychology, Georgetown University, 306 White-Gravenor Hall, Washington, DC 20057, USA e-mail: cjv2@georgetown.edu*

Functional near-infrared spectroscopy (fNIRS) is an emerging low-cost noninvasive neuroimaging technique that measures cortical bloodflow. While fNIRS has gained interest as a potential alternative to fMRI for use with clinical and pediatric populations, it remains unclear whether fNIRS has the necessary sensitivity to serve as a replacement for fMRI. The present study set out to examine whether fNIRS has the sensitivity to detect linear changes in activation and functional connectivity in response to cognitive load, and functional connectivity changes when transitioning from a task-free resting state to a task. Sixteen young adult subjects were scanned with a continuous-wave fNIRS system during a 10-min resting-state scan followed by a letter n-back task with three load conditions. Five optical probes were placed over frontal and parietal cortices, covering bilateral dorsolateral PFC (dlPFC), bilateral ventrolateral PFC (vlPFC), frontopolar cortex (FP), and bilateral parietal cortex. Activation was found to scale linearly with working memory load in bilateral prefrontal cortex. Functional connectivity increased with increasing n-back loads for fronto-parietal, interhemispheric dlPFC, and local connections. Functional connectivity differed between the resting state scan and the n-back scan, with fronto-parietal connectivity greater during the n-back, and interhemispheric vlPFC connectivity greater during rest. These results demonstrate that fNIRS is sensitive to both cognitive load and state, suggesting that fNIRS is well-suited to explore the full complement of neuroimaging research questions and will serve as a viable alternative to fMRI.

**Keywords: working memory, n-back, functional connectivity, resting state, fronto-parietal**

#### **INTRODUCTION**

Unprecedented technical advances in the past 20 years have made functional magnetic resonance imaging (fMRI) the primary neuroimaging modality for cognitive neuroscience. However, there are some notable drawbacks to fMRI that limit its utility in imaging young children and those with developmental disorders. First, head motion leads to substantial artifacts due to its relatively low temporal resolution and minimal constraint on head mobility in the scanning apparatus. Offline motion correction algorithms are effective for small movements, but larger movements necessitate subject exclusion, requiring oversampling as high as 30% in children with disorders such as Autism Spectrum Disorders (ASD), Attention Deficit Hyperactivity Disorder (ADHD), and Epilepsy (Yerys et al., 2009). In healthy young children (e.g., 4–6 years) who can comply with task instructions, the exclusion rate due to head motion was up to 40% (Yerys et al., 2009). Excessive head motion poses even more of a limitation for examining functional connectivity, the temporal co-activation of multiple brain regions, because even very small movements (e.g., *<*1 mm) introduce a systematic bias toward underestimating functional connectivity between distant regions (Power et al., 2012; Van Dijk et al., 2012). As the primary working hypothesis of some developmental disorders (e.g., ASD) is reduced long-distant functional connectivity, use of fMRI for those populations is particularly limiting. Second, the MR scanning environment is intimidating for many children. The enclosed nature of the scanning apparatus often produces feelings of claustrophobia, and the loud noise is fear-inducing for young children and autistic children with sensory hypersensitivity. Thus, despite its robust properties as a neuroimaging modality, fMRI is poorly suited for a large subset of pediatric and clinical populations. Thus, it is imperative to develop alternate neuroimaging modalities for investigating task-based and functional connectivity research questions.

Functional near-infrared spectroscopy (fNIRS) is an emerging non-invasive brain imaging modality for recording cortical hemodynamic activity. The method projects near-infrared light through the scalp and records optical density fluctuations resulting from metabolic changes within the brain. Similar to fMRI, cerebral blood flow is used as a proxy for neuronal activity. Both the spatial resolution and penetration depth of fNIRS are dependent upon the distances between light sources and detectors. The result is that the spatial resolution of fNIRS is on the order of 2.5– 3 cm and is capable of imaging depths of 1–2 cm (McCormick et al., 1992), making it well-suited for imaging cortical regions. This technique is particularly resilient to contamination from head motion since the optodes are affixed to the head and thus move with the subject. The silent operation and unenclosed scanning environment make fNIRS more amenable to subjects that have sensory hypersensitivity or claustrophobia. These qualities of fNIRS make it particularly suitable for use with pediatric populations, including those with developmental disorders. While fNIRS has been used in functional neuroimaging for almost 30 years (Ferrari et al., 1985), it remains unclear whether fNIRS has the requisite sensitivity to serve as an alternative to fMRI. To that end, it is important that fNIRS be validated against cognitive phenomena with known neural bases. While fNIRS has potential for use with developmental and clinical populations, it is necessary to first validate its sensitivity for cognitive processes in healthy adults. Furthermore, in order for fNIRS to be considered as a viable alternative to fMRI for examining developmental disorders, it is vitally important that its sensitivity be validated on cognitive processes commonly affected in those disorders. The present study examines the sensitivity of fNIRS to changes in cognitive state (e.g., resting to task) and task load during working memory, a component process of higher cognition that is disrupted in numerous developmental and psychiatric disorders.

Working memory is a temporary buffer for active maintenance and manipulation of goal-relevant information that critically depends upon the integrity of prefrontal cortex and its connections with posterior brain regions (Miller and Cohen, 2001). fMRI studies have consistently shown activation within the dorsolateral prefrontal cortex (dlPFC) and posterior parietal cortex, with significant left-hemisphere lateralization in prefrontal cortex for verbal working memory tasks (for a meta-analysis, see Owen et al., 2005). More specifically, studies have shown that activation in left dlPFC scales linearly with working memory load (Braver et al., 1997; Jansma et al., 2000; Veltman et al., 2003), indicating load-dependent recruitment of dlPFC. Common manipulations of load in verbal working memory tasks involve linear increases in the size or temporal lag of to-be-remembered information. For example, on the n-back task, letters are presented serially with instructions to detect target letters that repeat, successively (low load, termed 1-back) or with 2 or 3 intervening trials (higher load, termed 2-back, and 3-back, respectively). Working memory capacity predicts higher cognitive ability indexed by general intelligence (Kane et al., 2005; Oberauer et al., 2005) and reasoning (Süß et al., 2002). It increases during development (Gathercole et al., 2004) and those age-related increases relate to frontal-parietal white-matter maturation (Nagy et al., 2004) and activation (Olesen et al., 2003). Working memory is impaired in several developmental disorders (Alloway et al., 2009) and its training improves higher cognition, such as reasoning (Jaeggi et al., 2008) and attention in ADHD (Klingberg et al., 2005). Further, training-related changes are reflected in frontal and parietal activation (Olesen et al., 2004). Thus, working memory is an optimal candidate for validation of fNIRS as it is crucial for higher cognition, sensitive to developmental pathology and intervention, and depends upon prefrontal and parietal cortices.

A number of fNIRS studies have examined the effect of varying working memory load on activation. The studies have generally found that higher working memory load tends to produce greater activation within dlPFC (Hoshi et al., 2003; Li et al., 2010; Ayaz et al., 2012; Molteni et al., 2012). Further, fNIRS has demonstrated sensitivity to group differences in activation during working memory, based on gender (Li et al., 2010), ADHD diagnosis (Ehlis et al., 2008), schizophrenia diagnosis (Lee et al., 2008), dopamine receptor genotype (Herrmann et al., 2007), and mild cognitive impairment in the elderly (Niu et al., 2013b). However, no fNIRS study has demonstrated dlPFC activation that scales linearly with working memory load. As the utility of fNIRS is contingent upon its robustness as an imaging modality, it is important to demonstrate that fNIRS is sensitive enough to detect the linear relationship between prefrontal activation and working memory load that has been documented with fMRI (Braver et al., 1997; Jansma et al., 2000; Veltman et al., 2003). Additionally, no study has examined functional connectivity during working memory with fNIRS. Thus, it is unknown whether fNIRS is capable of detecting working memory load-dependent changes in functional connectivity. Task-evoked functional connectivity measurement by fNIRS, is itself novel with only a handful of studies to date (Chaudhary et al., 2011; Medvedev et al., 2011; Hall et al., 2013).

It is presently not known whether fNIRS is sensitive to changes in cognitive state. Sensitivity to changes in cognitive state from drowsy to wakeful to cognitively engaged can be detected reliably with scalp-based electroencephalography (Schomer and Da Silva, 2010). Recent fMRI studies have shown how cortical functional networks change as subjects transition from resting/awake to cognitively engaged states (Gordon et al., 2012a,b). Specifically, functional connectivity was greater during working memory than rest between dlPFC and inferior parietal cortex (Gordon et al., 2012b). State-dependent changes are important to understand as they depend upon genetic factors (Gordon et al., 2012b) and can reflect consolidation associated with learning (Lewis et al., 2009). Further, task-free resting state, itself, is sensitive to individual variation in a variety of affective and behavioral traits (Vaidya and Gordon, 2013). Thus, demonstrating that fNIRS is sensitive to cognitive state is important to establish its versatility as a tool that is as suitable for the full complement of research questions as other neuroimaging modalities.

Previous investigations have found that resting-state networks can be detected with fNIRS (White et al., 2009; Lu et al., 2010; Mesquita et al., 2010; Zhang et al., 2010a,b), and are stable across testing sessions (Zhang et al., 2011). Resting-state networks have been shown to be segregated within different frequency bands (Sasai et al., 2011) and correlate with networks detected by simultaneous fMRI (Sasai et al., 2012). Graph theory approaches have also been successfully applied to resting-state fNIRS (Niu et al., 2012), demonstrating its sensitivity to the topological organization of resting-state networks and that these measurements are stable across testing sessions (Niu et al., 2013a). However, no studies have used fNIRS to investigate functional connectivity differences between the resting state and a cognitive task.

The present study addressed two questions: First, is fNIRS sensitive to load-dependent working memory changes in activation and functional connectivity in prefrontal-parietal regions? Second, is fNIRS sensitive to functional connectivity differences between working memory and a task-free resting state? Healthy young adult subjects were imaged during a 10-min task-free resting state followed immediately by a verbal n-back task, with three loads, 1-back, 2-back, and 3-back. We hypothesized that: (1) activation within prefrontal cortex would scale with n-back load, (2) fronto-parietal functional connectivity would scale with n-back load, and, (3) fronto-parietal functional connectivity would be greater during task than rest.

#### **METHOD**

#### **SUBJECTS**

Sixteen Georgetown University undergraduates (6 male; 1 lefthanded) ages 18–24 years (Mean ± *SD* = 20*.*3 ± 1*.*7) participated in the study for payment. Participants were not using psychotropic medication (e.g., stimulants, anti-depressants, anxiolytics) and had no history of neurological injury or disease, seizure disorder, or psychiatric diagnosis. All participants provided informed consent according to guidelines of the Georgetown University Institutional Review Board, which approved the protocol.

#### **TASK PROCEDURE**

fNIRS sessions consisted of a 10-min resting-state run in which the subjects were instructed to close their eyes and remain awake, followed by a 6.5 min n-back task. The sequence of rest and task was not counter-balanced due to previous research showing that task-induced changes in functional connectivity persist after task completion (Evers et al., 2012; Gordon et al., 2012a; Harmelech et al., 2013). During the n-back task, participants were presented with a series of single consonant letters and instructed to press a button with their dominant hand when the presented letter was the same as the one presented *n* letters ago. Subjects were tested on three blocks of each of the three load conditions: 1-back, 2-back, and 3-back. The load condition order was pseudorandomized using a modified Latin square. Each block consisted of 9 trials, each lasting 3000 ms, with the letter exposed for 500 ms followed by a lag of 2500 ms. Each 27-s block was followed by a 14-s interval of fixation to allow the hemodynamic response to return to baseline. Subjects practiced the n-back task prior to the scanning session.

#### **IMAGING PROCEDURE**

Optical signals were recorded on a two-wavelength (690 and 830 nm) continuous-wave CW5 imaging system (TechEn, Inc., Milford, MA). Data were collected from detectors in parallel at a sampling rate of 41.7 kHz, with each laser modulated at a different frequency to allow subsequent offline demodulation and separation of source-detector pairs (i.e., channels). The 40 optical channels were comprised of 12 sources and 29 detectors, arranged into 5 probes, covering bilateral parietal cortex, bilateral prefrontal cortex, and frontal pole. Participants were fitted with a 128-channel HydroCel EEG cap (Electrical Geodesics, Inc., Eugene, OR) prior to probe placement. The cap provided a consistent frame of reference for positioning optical probes. Optode coordinates (provided in **Table 1**) in 10–20 reference space were estimated by triangulation with the three nearest EEG electrodes, using the electrode coordinates provided by the manufacturer. The NFRI software package (Singh et al., 2005) was then used to generate interpolation kernels for projection of channel data onto the brain surface, with interpolation only taking place between channels on the same probe (**Figure 1**).

**Table 1 | Optode positions computed from distances to neighboring EEG electrodes.**


*Source optodes are denoted with S, and detector optodes are denoted with D. Source optodes in PFC were located along the boundary between dlPFC and vlPFC, with detector optodes above and below giving rise to dlPFC and vlPFC channels, respectively.*

#### **BEHAVIORAL DATA ANALYSIS**

Behavioral data were lost for 2 subjects due to computer malfunction. Subject accuracy was computed for each load condition by taking the mean percentage of correct trials. Reaction time was computed by taking the mean across correct trials within each load condition. Repeated-measures ANOVAs and paired *t*-tests were performed for both accuracy and reaction time.

**FIGURE 1 | NIRS probe configuration.** Upper left panel shows the right parietal and right frontal probes and the upper right panel shows the medial frontal probe. Bottom panel is a rendering of estimated channel positions on a template brain. Colors indicate the channel sets used for interpolation.

#### **fNIRS PREPROCESSING**

Data were low-pass filtered with a high-order (400) FIR filter at 0.8 Hz and downsampled to 20 Hz. Raw optical density signals were converted to hemoglobin concentration changes using the modified Beer–Lambert law (Cope and Delpy, 1988) with the HOMer software package (Huppert et al., 2009). The oxy-Hb signal has previously been shown to correlate with blood flow better than the deoxygenated signal (Hoshi et al., 2001), thus interpretations focus on the oxygenated signal. Results for deoxygenated data are provided in the supplementary materials.

#### **fNIRS ANALYSIS—ACTIVATION**

Channel timecourses were modeled with a general linear model (GLM) in NIRS-SPM (Ye et al., 2009). Regressors were generated by convolving the weighted task boxcar function with the canonical hemodynamic response function provided by SPM8 (Friston et al., 1994). Data were corrected for low frequency drift by detrending using the wavelet-MDL algorithm (Jang et al., 2009) and corrected for serial correlations, such as those produced by physiological noise sources, using the HRF precoloring method (Worsley and Friston, 1995) implemented within NIRS-SPM (Ye et al., 2009). In order to separate the effects of load-dependent and load-independent activation, two regressors were generated: (1) a load-independent regressor in which all n-back blocks were weighted equally, and (2) a load-dependent regressor in which each n-back block was weighted by its load (i.e., 1, 2, or 3). Channel-wise beta values were compared across subjects for outliers. Subjects that had two or more adjacent channels with beta values over 3 standard deviations from the group mean were excluded from further analysis. Three subjects were excluded in this manner, thus reported results are from *N* = 13. The channel-wise beta values from the remaining subjects were then interpolated into voxel space. T-contrasts were then used to generate statistical parametric maps of activation for each regressor. A *p*-value correction was applied to control the familywise error rate using the Lipschitz-Killing curvature-based Euler characteristic (EC) approach (Li et al., 2012). Activation maps were thresholded at a corrected threshold of *p <* 0*.*05.

#### **fNIRS ANALYSIS—FUNCTIONAL CONNECTIVITY**

As fMRI studies have primarily found functional resting-state networks in the 0.01–0.10 Hz frequency range, both the restingstate data and n-back data were filtered around this range. This step also prevented high-frequency physiological artifacts from biasing the results. To this end, a band-pass Fourier filter was applied using the publicly available iFilter script for Matlab (Filter parameters: center = 0.035 Hz, width = 0.04 Hz, shape = 10; corresponding to a pass-band of approximately 0.009–0.09 Hz).

#### *Load-dependent*

For each n-back load, the individual block timecourses were concatenated and the Pearson correlation coefficient was computed between all channel-pairs. A Fisher's r-to-Z transformation was then applied to normalize the variance of the correlation values. For each channel-pair, the transformed correlation values were regressed against the corresponding n-back load. The t-statistic of the estimated beta value (i.e., the beta value divided by its standard error) was used in a one-sample *t*-test across subjects. Channels were grouped into 7 anatomical regions: left/right parietal (P), left/right ventrolateral PFC (vlPFC), left/right dorsolateral PFC (dlPFC), and frontal pole (FP). For each region-pair, a one-sample *t*-test was performed on the channel-wise *t*-statistic against the null hypothesis that the mean channel-wise *t*-statistic was less than the corresponding critical value at *p <* 0*.*05. A Bonferroni correction was applied to the region-wise *p*-values to control for multiple comparisons. The final significance threshold was set at *p <* 0*.*05.

#### *State-dependent*

To determine the contribution of cognitive state to functional connectivity, the resting-state and n-back scans were compared. Beginning and ending sections of the resting-state scan were removed to match the duration of the n-back task. In order to mitigate the influence of activation on the state-wise functional connectivity comparison, activation was regressed from the n-back fNIRS data. A GLM was created in NIRS-SPM with each load-level as a separate regressor. This GLM was necessary in order to remove non-linear load effects that would not be captured by the linear GLM used in the activation analyses. The residuals from the estimation were then used in the connectivity analyses. The results from the connectivity analyses without this regression step are provided in **Supplementary Figures 4**, **5**. Regression of activation was not applied in the load-dependent functional connectivity analysis, as different n-back loads are directly comparable and make an interpretable contribution to load-dependent changes in functional connectivity. The Pearson correlation was computed between all pairs of channels and Fisher's r-to-Z transformation was applied. For each channel-pair, a paired *t*-test (n-back *>* rest) was computed across subjects. For each region-pair, a one-sample *t*-test was performed on the channel-wise *t*-statistic. A Bonferroni correction was applied to the region-wise *p*-values to control for multiple comparisons. The *p*-value threshold was set at *p <* 0*.*05.

### **RESULTS**

#### **TASK PERFORMANCE**

One-Way repeated-measures ANOVA with Greenhouse-Geisser correction revealed a main effect of load on accuracy [*F(*1*.*768*,* <sup>17</sup>*.*677*)* = 4*.*043, *p <* 0*.*05]. Paired *t*-tests indicated that mean accuracy for 1-back (98.7 ± 1.9%) and 2-back (98.3 ± 3.0%) were near ceiling and did not differ (**Figure 2A**). Accuracy for the 3-back (94.6 ± 4.8%) condition was significantly lower than the 1-back [*t(*10*)* = 2*.*292, *p <* 0*.*05] and the 2-back [*t(*10*)* = 2*.*236, *p <* 0*.*05].

One-Way repeated-measures ANOVA with Greenhouse-Geisser correction revealed a main effect of load on reaction time [*F(*1*.*272*,* <sup>12</sup>*.*722*)* = 7*.*697, *p <* 0*.*05]. Paired *t*-tests indicated that 1-back (559 ± 106 ms) was performed faster than the 2-back (665 ± 218 ms), *t(*10*)* = 2*.*423, *p <* 0*.*05, and the 3-back (800 ± 207 ms), *t(*10*)* = 4*.*513, *p <* 0*.*005, while the 2-back and 3-back did not differ (**Figure 2B**).

to 3-back trials on the verbal n-back task. Activation for the loadindependent regressor, in which all blocks were weighted equally, did not survive correction.

#### **LOAD-DEPENDENT FUNCTIONAL CONNECTIVITY**

Oxy-Hb functional connectivity increased with increasing n-back loads: (1) between hemispheres for parietal cortex, L-Par— R-Par, and prefrontal cortex, L-vlPFC—R-vlPFC, L-dlPFC—RdlPFC, L-vlPFC—R-dlPFC; (2) between frontal and parietal regions within the left (L-Par—L-dlPFC) and right (R-Par—RdlPFC) hemispheres, across hemispheres (L-Par—R-dlPFC), and between parietal and frontopolar cortex (R-Par—FP, L-Par— FP); (3) between adjacent regions in frontal cortex, L-vlPFC— L-dlPFC, R-vlPFC—R-dlPFC, FP—L-dlPFC, FP—R-dlPFC; and (4) within all regions: L-Par, R-Par, L-vlPFC, R-vlPFC, L-dlPFC, R-dlPFC, FP (**Figures 4A,B**).

### **LOAD-DEPENDENT ACTIVATION**

Load-dependent oxy-Hb activation was significant in bilateral dlPFC (**Figure 3**), showing that the engagement of these regions increased linearly as working memory load increased from 1-back

#### **STATE-DEPENDENT FUNCTIONAL CONNECTIVITY**

Oxy-Hb functional connectivity increased from the resting-state run to n-back task: (1) between frontal and parietal regions, L-Par—R-vlPFC, R-Par—R-vlPFC; (2) between adjacent frontal regions, L-dlPFC—FP, R-dlPFC—FP; and (3) within bilateral

**FIGURE 4 | Channel-wise (A) and region-wise (B) load-dependent functional connectivity matrices.** Warm colors denote a load-dependent increase in functional connectivity, while cool colors denote load-dependent decrease in functional connectivity. Abbreviations: Par, Parietal; vlPFC, ventrolateral prefrontal cortex; dlPFC, dorsolateral prefrontal cortex; FP, frontal pole.

dlPFC, R-dlPFC, L-dlPFC (**Figure 5**). In contrast, functional connectivity decreased from the resting-state run to the n-back task: (1) between homologous frontal regions: L-vlPFC—RvlPFC, L-dlPFC—R-vlPFC. Thus, task-engagement resulted in an increase of functional connectivity between right vlPFC and bilateral parietal cortex, within bilateral dlPFC, and between bilateral dlPFC and FP. Functional connectivity decreased from rest to task between right vlPFC and left PFC (vlPFC and dlPFC).

#### **DISCUSSION**

The present study addressed two questions: (1) whether fNIRS is sensitive to load-dependent working memory changes in activation and functional connectivity in prefrontal-parietal regions, and (2) whether fNIRS is sensitive to functional connectivity differences between a working memory task and a task-free resting state. Activation was found to increase linearly with working memory load in bilateral PFC. Functional connectivity increased with working memory load between frontal and parietal regions, between hemispheres for homologous frontal and parietal regions, and locally (i.e., within regions and between adjacent regions). Change in cognitive state, from resting to working memory, changed functional connectivity such that it increased in fronto-parietal connections but decreased in inter-hemispheric frontal connections. These results collectively demonstrate that fNIRS detected functional neural changes associated with modulation of cognitive load and state in frontal and parietal cortices.

Working memory load-dependent activation increased linearly in bilateral dorsolateral prefrontal cortex, with stronger activation in the left hemisphere. This load-dependent prefrontal activation is consistent with previous fMRI findings (Braver et al., 1997; Jansma et al., 2000; Veltman et al., 2003). Braver et al. (1997) used a verbal n-back with loads of 0-, 1-, 2-, and 3-back and found load-dependent activation in bilateral dlPFC and left vlPFC. Jansma et al. (2000) used a spatial n-back task with loads 0-, 1-, 2-, and 3-back and found load-dependent activation in bilateral dlPFC and parietal cortex. Veltman et al. (2003) used a verbal n-back with loads of 0-, 1-, 2-, and 3-back and found load-dependent activation in bilateral dlPFC, left vlPFC, and left parietal cortex. Although found in fMRI studies, parietal activation did not survive corrected threshold in the present study. It has been shown that parietal regions have a longer scalp-to-brain distance than frontal regions and that this increased distance results in lower signal-to-noise ratio, as measured by correlation with simultaneous fMRI (Cui et al., 2011). Thus, the increased distance between scalp and brain may have impeded detection of activation in parietal cortex. Most previous fNIRS studies using a verbal n-back task examined load effects in a pairwise manner: subtracting the mean signal change of 1-, 2-, and 3-back each from 0-back (Hoshi et al., 2003), comparing mean signal change of 1-, 2-, and 3-back in an ANOVA (Li et al., 2010), and using a GLM to compare 1-, 2-, 3-back each with 0-back and baseline (Molteni et al., 2012). Ayaz et al. (2012) used a repeatedmeasures ANOVA to find a main effect of load on activation in left PFC, though *post-hoc* analyses showed only that 3-back had greater activation that 0- and 1-back. The present study is the first to use fNIRS to test for linear increases of activation spanning multiple working memory loads. As optode coverage was limited to prefrontal, parietal, and frontal pole, it is unknown whether other regions such as visual or temporal cortices were also sensitive to load-related increase. Establishing the sensitivity of fNIRS to prefrontal load-dependent modulation provides a useful tool for detecting both immature and disordered functional anatomy underlying higher order cognition. Working memory capacity predicts a variety of higher cognitive functions including reading ability (Swanson and Jerman, 2007), reasoning and problem solving (Süß et al., 2002), and general intellectual function as indexed by IQ (Kane et al., 2005; Oberauer et al., 2005). Prefrontal response to working memory demands depends upon dopaminergic activity (Aalto et al., 2005; McNab et al., 2009). Therefore, a load-dependent working-memory fNIRS probe is likely to be a useful tool in detecting disturbances in prefrontal functioning supporting higher cognition.

Functional connectivity was found to increase with working memory load between frontal and parietal regions, between hemispheres for homologous frontal and parietal regions, and locally (i.e., within regions and between adjacent regions). This finding supports the view that working memory is supported in a load-dependent manner by communication between prefrontal and parietal cortices, as well as between hemispheres. Notably, fronto-parietal functional connectivity increased with load for dlPFC, but not vlPFC. These findings are in accord with previous fMRI research showing load-dependent functional connectivity between contralateral and ipsilateral prefrontal and parietal regions, with stronger parietal connectivity with dlPFC than with vlPFC (Narayanan et al., 2005). This is the first demonstration of fNIRS sensitivity to functional connectivity changes related to working memory load, establishing the utility of fNIRS for probing task-evoked functional connectivity. It is of particular importance that fNIRS be shown to have sensitivity to load-related changes in functional connectivity, as this may allow functional connectivity to serve as a proxy for structural connectivity in those who cannot be imaged with traditional methods. Structural brain connectivity is typically assessed using Diffusion Tensor Imaging (DTI), an MRI technique that estimates the integrity of white-matter tracts. The reliance upon MRI precludes its use with a large subset of the developmental and clinical populations. This is particularly troublesome for developmental disorders such as ASD, which are associated with disruptions in connectivity (Courchesne and Pierce, 2005; Just et al., 2012). While fNIRS does not provide structural information directly, functional connectivity may provide indirect structural information. Functional connectivity depends, at least in part, upon the quality of structural connections between regions, and previous fMRI work has shown that functional connectivity predicts white matter integrity (Gordon et al., 2011). By measuring the relative changes of functional connectivity across varying task loads, the strength of the underlying structural connections may be estimated. In this way, fNIRS could prove to be a valuable tool for assessment of brain connectivity in populations that cannot currently be reached by DTI. However, fNIRS must first be capable of detecting connectivity differences across workloads. The present demonstration that fNIRS is sensitive to changes in functional connectivity resulting from working memory load provides further support for the potential of fNIRS in this domain.

The flexible engagement and disengagement of cognitive resources for serving current goals is the hallmark of mature cognition. Set-shifting, the ability to switch between response sets, is a form of cognitive flexibility that continues developing through early adulthood (Kalkut et al., 2009). Furthermore, set-shifting is impaired in developmental disorders of executive function such as ASD (Maes et al., 2011). Therefore, this form of flexibility is a vulnerable component of executive function. We reasoned that the simplest case of such flexibility is the transition from a resting to a task-performing state—the resting state can be thought of as one of unconstrained attention (as subjects are told to not think of anything in particular but to stay awake) when contrasted with n-back performance where attention has to be constrained to evaluating letters for n-back targets. fMRI studies show that fronto-parietal functional connectivity becomes stronger as subjects transition to a task from being at rest, and most importantly, individual variation in the magnitude of state-related functional connectivity changes predicted attentional function in everyday life (Gordon et al., 2012a,b). Here, we found that increased fronto-parietal functional connectivity was accompanied by reduced interhemispheric frontal connectivity, as subjects transitioned from rest to the task. A task-related decrease in functional connectivity between homologous prefrontal regions may be the result of a task-related increase of functional lateralization. Activation is commonly found to be stronger in the left hemisphere in verbal working memory paradigms (Owen et al., 2005). It stands to reason that an unconstrained resting state may have greater inter-hemispheric connectivity than a task that places demands on functions that are strongly lateralized. These changes were not driven by load-related variability in functional connectivity, because loads were regressed out. In children with connectivity abnormalities, such as those with ASD, state-related changes in functional connectivity suggested lack of engagement of task-selective circuitry and predicted variability on inattention symptoms among school-aged children (You et al., 2013). Thus, the availability of an accessible imaging modality that is sensitive to state-related changes in functional communication will be useful for investigation of both normal and disordered cognition. This demonstration of the sensitivity of fNIRS to cognitive state is an important step toward measuring cognitive flexibility.

The present findings need to be considered in the context of the following methodological issues: First, while fNIRS has excellent temporal resolution and resilience to artifacts arising from head motion, the spatial resolution is inferior to fMRI. Although fitted EEG caps were used to position NIRS probes in reference to standard 10–20 coordinates, the location of channels with respect to underlying brain regions could not be independently verified. These factors make precise localization of fNIRS signals particularly difficult. However, the spatial resolution of fNIRS is low enough (2–3 cm) that these relatively small imprecisions should have only a minimal impact on the results. Second, while the medial frontal channels are likely to overlap with frontopolar cortex, there is some difficulty in interpretation due to greater depth of cortex as it approaches the medial longitudinal fissure. Thus, medial frontal channels may have covered regions where cortex was further from the scalp, potentially weakening the signal relative to lateral prefrontal cortices. Third, while the deoxygenated hemoglobin signal tended to show similar patterns of activation (**Supplementary Figure 1**) and functional connectivity (**Supplementary Figures 2**, **3**) to the oxygenated signal, the patterns were not identical. Given that task-evoked influx of oxygenated hemoglobin far outpaces oxygen metabolism, it is not surprising that the oxygenated hemoglobin signal is more robust than the deoxygenated signal. Further, deoxygenated hemoglobin signal tends to have lower signal-to-noise ratio than oxygenated, due in part to lesser tissue penetration of the short-wavelength light associated with deoxygenated signal. Fourth, task performance did not scale linearly with n-back load. This is not surprising as performance is limited by the sensitivity of the task to measure differences and is subject to ceiling and floor effects. In addition, subjects are often able to accommodate increased workloads without a significant change in performance simply by increasing effort. In contrast, brain activation is closely linked to effort and is thus expected to be more sensitive to changes in workload than behavior. This is evidenced by studies showing group differences in brain activation where performance does not differ (Matsuo et al., 2006; Herrmann et al., 2007). Therefore it is not necessary for task performance to scale with working memory load. Despite the limitations, our findings show some useful attributes of prefrontal-parietal functioning, specifically sensitivity to working memory load and cognitive state. This sensitivity makes fNIRS a useful imaging modality for a large segment of children and adults who cannot be reached by fMRI. As this field matures, some consensus will emerge regarding data processing steps and parameter choices that ought to make it possible to compare results across studies more reliably.

In sum, this study demonstrates the utility of fNIRS for detection of activation and functional connectivity related to cognitive load and state. These findings are particularly important as they provide a basis for the use of fNIRS as an alternative to fMRI in studies of executive function, particularly in pediatric and clinical populations that are not amenable to fMRI. Working memory is an important domain in this regard, as it develops over the course of childhood and adolescence and is subverted in developmental disorders. This study is the first to show that fNIRS has the requisite sensitivity to detect activation and functional connectivity that increase linearly with increasing working memory load, and one of the first to demonstrate that fNIRS can reliably detect differences related to cognitive state (e.g., rest versus task). In order for fNIRS to be adopted for widespread use, it is vital to first demonstrate its sensitivity to activation and functional connectivity during cognitive processes of interest.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www*.*frontiersin*.*org/journal/10*.*3389/fnhum*.* 2014*.*00076/abstract

#### **Load-dependent Deoxy-Hb activation**

Load-dependent activation for the deoxy-Hb signal did not survive correction, but was patterned similarly to oxy-Hb with bilateral frontal activation (**Supplementary Figure 1**). Loadindependent activation was not observed.

#### **Load-dependent Deoxy-Hb functional connectivity**

Load-dependent deoxy-Hb functional connectivity reached corrected threshold in frontal cortex only, increasing with greater n-back loads: for (1) fronto-parietal connections (FP-lP, FPrP); (2) frontal interhemispheric connections (lvlPFC-rvlPFC); (3) parieto-parietal connections (lP-rP); and local connections (rvlPFC-rdlPFC, lP, lvlPFC, ldlPFC, FP, rdlPFC, rvlPFC, rP; **Supplementary Figure 2**).

#### **State-dependent Deoxy-Hb functional connectivity**

Deoxy-Hb functional connectivity was greater during the n-back run than the resting state run between (**Supplementary Figure 3**): (1) frontal and parietal regions (FP-lP, rdlPFC-lP, rvlPFC-lP, rdlPFC-rP); (2) between parietal regions (lP-rP); and within regions (rP, lP, rdlPFC). Functional connectivity was greater for resting-state than n-back: (1) for all significant connections with left frontal regions (lvlPFC-ldlPFC, lvlPFC-FP, ldlPFC-FP, ldlPFC-rdlPFC, lvlPFC-rvlPFC, ldlPFCrvlPFC); (2) between frontal and parietal regions: rP-lvlPFC; and (3) within frontal regions (ldlPFC, rvlPFC, FP).

#### **State-dependent functional connectivity without task regression**

Oxy-Hb signal showed task-related functional connectivity that was much more prominent than in the task-regression case for: (1) fronto-parietal connections (lvlPFC-lP, ldlPFC-lP, rdlPFC-lP, lvlPFC-rP, ldlPFC-rP, FP-rP, rdlPFC-rP); (2) between hemispheres for parietal (lP-rP); and (3) between bilateral dlPFC and FP (ldlPFC-rdlPFC, ldlPFC-FP, rdlPFC-FP). Similarly, the deoxygenated signal showed greater fronto-parietal (lvlPFC-lP, ldlPFC-lP, FP-lP, rdlPFC-lP, FP-rP, rdlPFC-rP), parieto-parietal (lP-rP, lP-lP, rP-rP), and fronto-frontal (vlPFC-FP, dlPFC-FP) task-related functional connectivity (**Supplementary Figure 5**) relative to the task-regressed results. The increase in task-related functional connectivity is interpreted as the result of bias from the presence of task-related activation.

#### **Supplementary Figure 1 | Load-dependent activation from the deoxygenated hemoglobin signal shared a similar pattern with the**

**oxygenated signal, but did not reach significance.** *p <* 0*.*05, uncorrected.

**Supplementary Figure 2 | Channel-wise (A) and region-wise (B) load-dependent functional connectivity matrices for the deoxygenated signal.** Warm colors denote a load-dependent increase in functional connectivity, while cool colors denote load-dependent decrease in functional connectivity. Abbreviations: Par, Parietal; vlPFC, ventrolateral prefrontal cortex; dlPFC, dorsolateral prefrontal cortex; FP, frontal pole.

#### **Supplementary Figure 3 | Channel-wise (A) and region-wise (B) state-dependent functional connectivity matrices from deoxygenated hemoglobin signal.** Warm colors denote connections where functional connectivity was greater during the n-back than at rest, while cool colors denote greater connectivity at rest. Abbreviations: Par, Parietal; vlPFC, ventrolateral prefrontal cortex; dlPFC, dorsolateral prefrontal cortex; FP, frontal pole.

**Supplementary Figure 4 | Channel-wise (A) and region-wise (B) state-dependent functional connectivity matrices without performing regression on the task data (oxygenated hemoglobin signal).** Warm colors denote connections where functional connectivity was greater during the n-back than at rest, while cool colors denote greater connectivity at rest. Abbreviations: Par, Parietal; vlPFC, ventrolateral prefrontal cortex; dlPFC, dorsolateral prefrontal cortex; FP, frontal pole.

**Supplementary Figure 5 | Channel-wise (A) and region-wise (B) state-dependent functional connectivity matrices without performing regression on the task data (deoxygenated hemoglobin signal).** Warm colors denote connections where functional connectivity was greater during the n-back than at rest, while cool colors denote greater connectivity at rest. Abbreviations: Par, Parietal; vlPFC, ventrolateral prefrontal cortex; dlPFC, dorsolateral prefrontal cortex; FP, frontal pole.

#### **REFERENCES**


Zhang, Y.-J., Lu, C.-M., Biswal, B. B., Zang, Y.-F., Peng, D.-L., and Zhu, C.-Z. (2010b). Detecting resting-state functional connectivity in the language system using functional near-infrared spectroscopy. *J. Biomed. Opt.* 15, 047003. doi: 10.1117/1.3462973

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 October 2013; accepted: 30 January 2014; published online: 20 February 2014.*

*Citation: Fishburn FA, Norr ME, Medvedev AV and Vaidya CJ (2014) Sensitivity of fNIRS to cognitive state and load. Front. Hum. Neurosci. 8:76. doi: 10.3389/fnhum. 2014.00076*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Fishburn, Norr, Medvedev and Vaidya. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## A problem-solving task specialized for functional neuroimaging: validation of the Scarborough adaptation of the Tower of London (S-TOL) using near-infrared spectroscopy

#### *Anthony C. Ruocco1 \*, Achala H. Rodrigo1, Jaeger Lam1, Stefano I. Di Domenico1, Bryanna Graves <sup>1</sup> and Hasan Ayaz <sup>2</sup>*

*<sup>1</sup> Clinical Neurosciences Laboratory, Department of Psychology, University of Toronto Scarborough, Toronto, ON, Canada*

*<sup>2</sup> School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA, USA*

#### *Edited by:*

*Kazuo Hiraki, University of Tokyo, Japan*

#### *Reviewed by:*

*Sotaro Shimada, Meiji University, Japan Matsuda Goh, University of Tokyo, Japan*

#### *\*Correspondence:*

*Anthony C. Ruocco, Clinical Neurosciences Laboratory, Department of Psychology, University of Toronto Scarborough, 1265 Military Trail, Toronto, ON M1C 1A4, Canada e-mail: anthony.ruocco@gmail.com* Problem-solving is an executive function subserved by a network of neural structures of which the dorsolateral prefrontal cortex (DLPFC) is central. Whereas several studies have evaluated the role of the DLPFC in problem-solving, few standardized tasks have been developed specifically for use with functional neuroimaging. The current study adapted a measure with established validity for the assessment of problem-solving abilities to design a test more suitable for functional neuroimaging protocols. The Scarborough adaptation of the Tower of London (S-TOL) was administered to 38 healthy adults while hemodynamic oxygenation of the PFC was measured using 16-channel continuous-wave functional near-infrared spectroscopy (fNIRS). Compared to a baseline condition, problems that required two or three steps to achieve a goal configuration were associated with higher activation in the left DLPFC and deactivation in the medial PFC. Individuals scoring higher in trait deliberation showed consistently higher activation in the left DLPFC regardless of task difficulty, whereas individuals lower in this trait displayed less activation when solving simple problems. Based on these results, the S-TOL may serve as a standardized task to evaluate problem-solving abilities in functional neuroimaging studies.

**Keywords: functional near-infrared spectroscopy (fNIRS), Tower of London, validation, dorsolateral prefrontal cortex, problem-solving, executive functioning, deliberation**

#### **INTRODUCTION**

The term *problem-solving* refers to a multifaceted higher-order cognitive function directed toward identifying problems with the current state, and generating and implementing potential solutions to achieve a goal state (Simon and Newell, 1971). Problem-solving is considered a cyclical process comprising several interacting non-sequential stages (**Figure 1**). Broadly, problem-solving can be summarized to involve three partially overlapping cognitive operations: problem recognition, definition, and representation (Pretz et al., 2003). Problem *recognition* concerns the extent to which a problem is directly presented to an individual, or if it requires discovery or creation by the problem-solver (Getzels, 1982). Problem *definition* refers to the precision with which a problem's scope and goals are delineated, ranging from a problem that is clearly defined to one that is indistinct, the former usually associated with a presented problem and the latter with one that is discovered or created by the problem-solver. Last, problem *representation* is thought to encompass four components: a description of the initial state of the problem, a description of the goal state, a set of allowable operators (i.e., actions taken to move from one state to another), and a set of constraints (Pretz et al., 2003).

Several cognitive tests have been developed to assess problem-solving ability in an objective and standardized manner. *Tower tests* are among the most commonly administered tests of problem-solving for both research and clinical purposes (for a review, see Sullivan et al., 2009). These tasks normally present individuals with three placeholders (usually pegs or pockets) and multiple balls or discs which can be put onto each placeholder. Conventional tower tasks typically differ in the lengths of the placeholders (equal or unequal lengths), colors of the balls or discs (monochromatic or polychromatic), and sizes of the balls or discs (equal or unequal circumferences). Based on the threecomponent model of problem-solving described by Pretz et al. (2003), the conventional tower task can be characterized as a presented problem (*recognition*) whose initial and goal states, allowable operators, and constraints, are precisely demarcated (*definition*). Features which typically differ from one variant of the task to another, however, are the number of allowable operators required to achieve the goal state, and the specific constraints (or "rules") imposed on the problem-solving task (*representation*). Therefore, conventional tower tasks do not measure the cognitive processes underlying discovered or created problems (i.e., problem types that are more challenging to evaluate in a controlled and standardized fashion). Perhaps due to their narrow

problem definition and ease of standardization, tower tasks have enjoyed widespread clinical application (Culbertson and Zillmer, 1998; Culbertson et al., 2004) and extensive validation to identify the component cognitive functions which may underlie performance on these tasks (Welsh et al., 1999; Unterrainer et al., 2004).

Researchers attempting to understand the neural underpinnings of problem-solving ability have adapted a variety of paperand-pencil tower tasks for administration by computer. Perhaps the most commonly adapted tower task is known as the Tower of London (TOL) (Shallice, 1982). Modifications to this task have typically involved changes to its representational features, namely, the number of allowable operators to achieve the goal state and the mode by which allowable operators may be implemented (i.e., not by physically moving the balls or discs from one placeholder to the next, but instead by mentally visualizing each move). Using functional magnetic resonance imaging, functional near-infrared spectroscopy (fNIRS) and positron emission tomography, several studies have used these tasks to delineate the neural circuitry underlying problem-solving ability<sup>1</sup> (**Table 1**). While the parameters for each variant of the TOL varied from one study to another, the prefrontal cortex (PFC) was reliably activated across all studies, including its anterior, inferior, and dorsolateral (DLPFC) aspects (Baker et al., 1996; Boghi et al., 2006; Wagner et al., 2006; Just et al., 2007; Den Braber et al., 2008; Fitzgerald et al., 2008; Campbell et al., 2009; Zhu et al., 2010; De Ruiter et al., 2011; Kaller et al., 2011; Stokes et al., 2011; Hahn et al., 2012). Activation was also routinely observed in a number of other cortical and subcortical regions, including the parietal cortex, premotor region, anterior cingulate cortex, insular cortex, caudate, and thalamus (Baker et al., 1996; Beauchamp et al., 2003; Cazalis et al., 2006; Just et al., 2007; Campbell et al., 2009; Den Braber et al., 2010). With increasing "difficulty" or "task load" (i.e., the greater the number of allowable operators, or "moves," required to achieve the target configuration), higher levels of activation were observed primarily within the left DLPFC as well as the parietal cortex bilaterally (Rasmussen et al., 2006; Den Braber et al., 2008, 2010), areas which also have showed significant functional connectivity during performance on this task (Just et al., 2007). The left DLPFC has been linked specifically to the extraction of goal information and the generation of an internal problem representation, whereas the right DLPFC may be more strongly associated with working memory and mental transformations (Newman et al., 2003, 2009; Van Den Heuvel et al., 2003; Wagner et al., 2006; Ruh et al., 2012).

Taken together, these studies have provided important information about the role of the DLPFC in problem-solving on the TOL. These investigations, however, incorporated adapted computerized variants of this task that varied along a number of task parameters (see **Table 1**) that could influence the patterns of DLPFC activation observed in one study to another. These included the number of "moves" (or allowable operators) required to achieve the target configuration (which varied from 2 to 6), the mode of responding (which ranged from a forced-choice two-alternative format to one which required examinees to press one of seven buttons on a keypad), and the baseline or "control" task used for comparison with more complex problem-solving trials (which ranged from a blank screen or fixation crosshair to the use of 0- and 1-move problems). Variations in these task parameters could contribute to discrepant findings observed across studies, potentially obscuring subtle distinctions in DLPFC activation patterns that may be associated with dissociable cognitive functions.

The primary aims of the present study were to develop and validate a new computerized version of the TOL designed specifically for neuroimaging, called the Scarborough adaptation of the Tower of London (or S-TOL). An experimental task like the S-TOL is said to be valid for measuring an intended cognitive function when it produces measurement outcomes (e.g., patterns of neural activation) that are consistent with the theory of response behavior that guided its construction (Borsboom et al., 2004; Borsboom, 2005). Given that the S-TOL was developed to measure problem-solving ability using an established paradigm associated with a reasonably well-defined pattern of regional brain activation, we anticipated that similar neural activations would be elicited by the S-TOL to provide converging support for its validity as a problem-solving task.

Validity of the S-TOL was also examined on the basis of the relationship between a trait known as deliberation and activity in the DLPFC across levels of task difficulty (i.e., lower vs.

<sup>1</sup>The authors of nearly all functional neuroimaging studies employing adapted versions of the Tower of London task described these tasks as measures of *planning*. Whereas paper-and-pencil tower tasks are often used to measure planning ability, the primary index of this cognitive function on these tasks is the duration of time that an examinee deliberates before physically attempting their first move on each problem. Computerized versions of these tasks adapted for functional neuroimaging studies typically did not ask examinees to physically manipulate the balls or discs. Instead, examinees were normally only required to indicate the minimum number of moves necessary to achieve each goal state. Therefore, these tasks typically did not directly measure planning ability and are referred to in this paper as tests of problem-solving ability.


#### **Table 1 | Task parameters for computerized adapted versions of the Tower of London used in functional neuroimaging studies published since 2000.**

*(Continued)*


*fMRI, functional magnetic resonance imaging; fNIRS, functional near-infrared spectroscopy; PET, positron emission tomography. aData reported are for healthy control participants.*

higher). *Deliberation* is defined as "the tendency to think carefully before acting" (McCrae and Costa, 2010, p. 24). Within the context of a laboratory task like the S-TOL, we expected participants who are disposed to engage with problem-solving items in a more thoughtful manner (i.e., individuals higher in deliberation) to exhibit a more consistent pattern of activation within those PFC regions that are critical for problem-solving (i.e., DLPFC), regardless of task difficulty. This association between deliberation and task difficulty would constitute further evidence that the S-TOL is valid for studying problem-solving processes by demonstrating that response behavior on the S-TOL varies systematically with the quality of task engagement to which respondents are disposed on problem-solving tasks.

Importantly, the S-TOL was intentionally designed with an eye toward its potential for translation to clinical samples (i.e., individuals with psychiatric and neurological disorders). Therefore, the task was constructed with the aim of achieving high levels of accuracy, even in individuals with possible central nervous system dysfunction, to reduce the likelihood of frustration on the task and to facilitate comparisons between activation blocks containing lower and higher difficulty problems. This approach in designing the S-TOL was considered essential to ensure that any observed differences in patterns of brain activation between clinical and non-clinical groups are not confounded with differences in accuracy on the task. Accordingly, this task incorporated problems requiring no more than three moves to achieve a target configuration, and utilized a simplified response format (i.e., a forced-choice "yes" or "no" answer to the same task instruction across all trials), thereby reducing the need for complex response devices and minimizing demands on working memory. The development and validation of a standardized tower task that can be readily incorporated into neuroimaging protocols could increase consistency of methods across studies and facilitate comparisons of results between clinical and non-clinical samples.

#### **MATERIALS AND METHODS**

#### **PARTICIPANTS**

An initial sample of 43 healthy adults provided informed written consent to participate in this study. Four participants were subsequently excluded because they obtained atypically inaccurate performances (*<*60%) on zero-move (ZM) trials of the S-TOL (i.e., these participants may not have understood or complied with instructions), and one participant was excluded due to technical problems with an event-marker file for the neuroimaging task. The final sample comprised 38 adults who were recruited from the University of Toronto Scarborough's undergraduate research participant pool as well as the surrounding University community. Participants were largely right-handed (76.3%) and female (60.5%) with an average 13.8 years (*SD* = 1*.*7) of formal education. The ethno-racial composition of the sample according to 2011 Canadian census categories was as follows: Chinese (34.2%), White (21.1%), South Asian (13.2%), Black (5.3%), Filipino (5.3%), Latin American (5.3%), Japanese (2.6%), Korean (2.6%), Southeast Asian (2.6%), West Asian (2.6%), and Other (5.3%). Prior to commencing the neuroimaging protocol, participants completed a brief screening measure to collect demographic information and to rule out the presence of any serious manual, ophthalmic, neurologic (i.e., seizure disorder, severe head injury), or psychiatric illness (i.e., psychosis, bipolar disorder).

#### **PROCEDURE**

This research was conducted in accordance with Canada's 2nd edition of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans and was approved by the Social Sciences, Humanities and Education Research Ethics Board at the University of Toronto. After a complete description of the study, participants were seated in a dimly-lit room in front of a computer monitor and a keyboard. Participants were asked to sit comfortably and interact with the computer using the mouse with their right hand. Prior to beginning each task, instructions were presented on the monitor and read aloud by the experimenter. Testing did not proceed until participants acknowledged that they understood all instructions completely. After finishing all procedures, participants were compensated for their time with course credit or \$10 for each hour of the experiment. After the participant's forehead was cleaned using an alcohol swab, the fNIRS probe was positioned over the forehead and secured at the back of the head using Velcro® straps. The fNIR Imager 1000® (fNIR Devices, Potomac, MD) is a continuous-wave fNIRS system described in previous studies conducted by our research group (Ruocco et al., 2010; Ayaz et al., 2012; Rodrigo et al., 2014). Two wavelengths of light (730 and 850 nm) were measured continuously at 500 ms intervals in 16 channels with 1.25 cm penetration. The probe was aligned with the electrode positions F7, FP1, FP2, and F8 (which correspond to Brodmann areas 9, 10, 45, and 46) based on the international 10–20 EEG system (Jasper, 1958). Specific details regarding probe placement are provided in Ayaz et al. (2006). **Figure 2** displays the spatial location of each channel of the fNIRS system. Image reconstruction was rendered using the topographic tools

available in fNIRSoft® Professional Edition (Ayaz, 2010), which provides spatial visualization of fNIRS activation data using magnetic resonance imaging templates as described in Ayaz et al. (2006). After completing fNIRS procedures, participants were seated in a testing room and asked to complete a personality inventory that evaluated traits related to decision-making and impulsiveness.

**near-infrared spectroscopy system.**

#### **SCARBOROUGH ADAPTATION OF THE TOWER OF LONDON (S-TOL)**

Participants completed a newly adapted computerized version of the TOL which was designed according to original descriptions of this task as presented in Shallice (1982). Two boards were visually presented in color on a computer screen (**Figure 3**). The task began with the following on-screen instructions which were also read aloud by the examiner:

On this task, you will see two boards: one at the top of the screen and one at the bottom. The board at the top of the screen is called the target bard and the board at the bottom of the screen is your board. Each peg has a different size. The first peg can hold three colored balls. The second peg can hold two colored balls. The third peg can hold one colored ball. Your job is to decide how many times you need to move the colored balls, from one peg to another, to make your board look like the target board. You will have 7 s to study the two boards, afterward; you will always be asked the same question: Can you solve this in exactly two moves? You will have 3 s to decide your answer.

Participants were also provided with two rules: (1) they could move only one ball at a time, and (2) they could not put more balls

**FIGURE 3 | Sample computerized stimuli from the Scarborough adaptation of the Tower of London task.** The left panel displays sample problems requiring a minimum of two moves to solve the problem (left) or zero moves to solve the problem (right). Stimuli were designed based on descriptions provided in Shallice (1982).

on a peg than what it could hold. Participants completed three practice trials and had the opportunity to ask questions about the task. After the practice trials, participants were given the following instruction:

Remember to stay as still as you can, and to keep your hand on the mouse at all times. Think through each problem carefully. Be sure to decide yes or no as accurately as possible within the 3-s time limit. It is more important to be accurate in your decision than to give your answer quickly.

The task consisted of two trial types: multiple-move (MM) trials, which included problems that could be completed in a minimum of either two or three moves, and ZM trials, during which participants observed two boards displaying identical configurations (i.e., no moves were required) (**Figure 3**, right panel). MM trials included a combination of problems that either did or did not require an intermediate step to achieve the goal state (**Figure 3**, left panel). The correct answer was "yes" for two-move trials and "no" for three-move and ZM trials, the latter included as a check to ensure that participants remained engaged during ZM trials. The advantage of using the same instruction for MM and ZM trials was that the maintenance of this instruction in working memory was equivalent between trials. Each trial began with a 7-s study period which was followed by a 3-s window during which participants were asked to respond using a mouse by clicking on a box labeled either "yes" or "no." Trials were grouped into blocks containing six of either MM or ZM trials, and blocks were separated by a 30-s rest period when participants were asked to fixate on a crosshair at the center of the screen. Blocks were alternated, starting with a MM block, over a total of six repetitions for each block. Accuracy and response times (RT) were recorded for MM and ZM trials.

#### **TRAIT DELIBERATION**

Participants completed the NEO Personality Inventory-3 (NEO-PI-3; McCrae et al., 2005a), a 240-item self-report personality inventory that was designed to measure the major domains of personality based on the Five-Factor Model (FFM) of personality (McCrae and Costa, 1987). Of interest to the current study was a trait facet scale, referred to as *Deliberation*, which reflects the tendency for individuals to think things through before acting or speaking. Examinees are asked to rate their answers to items on a five-point Likert-type scale: strongly disagree, disagree, neutral, agree, and strongly agree. The NEO-PI-3 shows strong convergence and similar (if not superior) psychometric properties when compared to its predecessor, the NEO-PI-Revised (McCrae et al., 2005b; De Fruyt et al., 2009). These measures have demonstrated excellent reliability and validity across a large number of studies (see McCrae et al., 2005a).

#### **STATISTICAL ANALYSES**

#### *Signal processing*

Raw fNIRS light intensities were manually screened to exclude channels which had poor signal quality (i.e., very low signal or saturation) and subsequently underwent signal processing to exclude physiological artifacts using a low-pass filter with a finite impulse response and a linear phase filter with an order of 20 and a cut-off frequency of 0.1 Hz (Izzetoglu et al., 2004; Ayaz et al., 2012). Following these procedures, channels that were identified as being problematic using a sliding-window motion artifact rejection (SMAR) technique (Ayaz et al., 2010) were analyzed and confirmed rejected through visual inspection. On average, 2.67 (*SD* = 2*.*06) channels were excluded for each participant primarily because of saturation with ambient light when contact with the skin was not optimal or in lateral channels due to interference from hair shafts. Time synchronization markers denoting the beginning and end of each block were delivered to the fNIRS acquisition device using a serial connection. Based on the markers that separated MM and ZM blocks, data for local baseline segments and activation segments were extracted and compared. The activation segments consisted of blocks (i.e., ZM and MM) that were 60-s in duration. Each block consisted of 6 trials that began with a 7-s observation period and a 3-s response period. The local baseline segments included the first 10-s of each block, representing 20 observations sampled at 500-ms intervals at the beginning of each block. The primary measure of interest in this study was oxygenated hemoglobin (oxy-Hb), although deoxygenated hemoglobin and total hemoglobin measurements were also collected but not reported because oxy-Hb is more commonly associated with neural activity.

#### *Statistical plan*

In order to control for neural activation associated with visual attention and working memory (i.e., maintaining task instructions in mind), primary analyses contrasted ZM and MM blocks. Therefore, the main difference between ZM and MM blocks was that the latter required participants to solve problems that required a minimum of two or three moves (rather than none) to achieve the goal state. According to the Related-Samples Wilcoxon Signed Rank Test, participants made more errors on MM as compared to ZM trials, *z* = 4*.*19, *p <* 0*.*001, *r* = 0*.*70. To control for differences in accuracy between MM and ZM blocks, a criterion of 90% correct responses was applied to both trial types. This procedure identified a subset of participants (*n* = 24) that obtained similar performances across MM and ZM conditions, *t(*23*)* = 2*.*01, *p* = 0*.*05. Contrasts of activation associated with MM and ZM conditions for these highly accurate participants (*n* = 24) were visualized separately from the less accurate participants (*n* = 14) to highlight differences in functional activation between these groups. All data were visualized on a standard MRI template (**Figure 2**).

The fNIRS time-series data were analyzed with multilevel models (Bryk and Raudenbush, 1992; Kenny et al., 1998). Multilevel models are regression models that feature fixed effects as well as random effects (i.e., parameters distributed according to some probability distribution). Multilevel models confer a number of advantages over traditional repeated measures ANOVA. Two such advantages that are relevant to the present investigation concern the inclusion of unbalanced data and the flexibility to incorporate continuous predictors. Multilevel models have been recommended for psychophysiological research (Bagiella et al., 2000), and have been employed in our previous studies using fNIRS (e.g., Di Domenico et al., 2013; Rodrigo et al., 2014).

Within the context of the present study, multilevel models take into account that the data points comprising each participant's experimental time-series measurements (i.e., oxy-Hb measurements taken at intervals of 500-ms) are nested within the respective participants and that the number of data points may be unbalanced across participants due to signal processing. Thus, variance in the dependent variable (i.e., oxy-Hb) is partitioned into within-person (Level-1) and between-person (Level-2) components, allowing predictor terms to be represented at both the level of the experimental condition (i.e., the MM and ZM conditions) and at the level of the participant, respectively. In the primary analyses, we examined the Level 1 effect of problemsolving difficulty on the S-TOL across 16 fNIRS channels, controlling for Type I error (*p <* 0*.*05) using the False Discovery Rate (FDR) approach (Benjamini and Hochberg, 1995). Problemsolving was effect-coded (*ZM* = −1; *MM* = 1) in all multilevel analyses. Thus, a two-unit change on this effect-code represents the unstandardized mean difference in oxy-Hb across the ZM and MM conditions.

All multilevel models were estimated in R Core Team (2013) using the *multilevel* and *nlme* packages (Bliese, 2009). We estimated random intercept models, nesting the experimentally demarcated time-series data within each participant. To account for the temporal autocorrelation in the time-series, all models were conservatively estimated using an unstructured covariance matrix and the "between-within" method of estimating degrees of freedom (Schluchter and Elashoff, 1990). The descriptive statistics for the oxy-Hb time-series are provided in **Table 2**. The intraclass correlations across the fNIRS channels ranged from 0.03 to 0.28 indicating a small but significant degree of dependence among participants' nested data points and a substantial amount of within-person variation during the time-course of the S-TOL as expected.

#### **Table 2 | Descriptive statistics for oxy-Hb time-series across fNIRS channels.**


*All intraclass correlations are significant at p < 0.0001. N* = *total number of data points, aggregated across all participants; n* = *total number of participants with available data. The numbers of data points across fNIRS channels are unbalanced due to filtering.*

#### **RESULTS**

#### **BEHAVIORAL PERFORMANCE AND PERSONALITY TRAITS**

On the S-TOL, all participants (*N* = 38) attained 97.5% (*SD* = 3*.*72) accuracy on ZM problems, 88.9% (*SD* = 15*.*8) on two-move problems, and 89.2% (*SD* = 12*.*73) on three-move problems. Mean RT was 161 ms (*SD* = 914) for correct ZM problems, 896 ms (*SD* = 155) for two-move problems, and 1018 ms (*SD* = 206) for three-move problems.

*T*-scores (with a normative mean of 50 and *SD* of 10) for the participants' self-reported levels of deliberation as measured by the NEO-PI-3 was within normal limits (*M* = 45*.*1, *SD* = 10*.*7). There were no significant correlations (Spearman's rho) between Deliberation scores and accuracy indices from the S-TOL (**Table 3**; all *p*'s *>* 0.05).

#### **SOLVING COMPLEX vs. SIMPLE PROBLEMS**

The results of multilevel analyses comparing oxy-Hb across the MM and ZM conditions for all participants (*n* = 38) is presented in **Table 4**. Significantly greater increases in oxy-Hb for MM compared to ZM were observed in two distinct clusters, the first encompassing the anterior aspects of the left inferior/middle frontal gyrus (left DLPFC; channels: 1, 2, 3, and 4), and the second, the right superior frontal gyrus (right medial PFC channels: 10, 11, and 12). Conversely, significantly less oxy-Hb for MM as compared to ZM conditions was observed in two clusters, the first centered over the medial PFC (channels 5, 7, and 9), and the second comprising a single channel (16) over the most anterior aspect of the right inferior frontal gyrus (IFG). Channels 6, 8, 13, 14, and 15 showed no difference in oxy-Hb changes across the MM and ZM conditions. These results are displayed in **Figure 4**.

**Table 3 | Spearman's rho correlations between impulsive personality traits and accuracy on the Scarborough adaptation of the Tower of London.**


### **Table 4 | Multilevel analyses comparing oxy-Hb levels for**



*\*\*p < 0.001, \*p < 0.05. All models were estimated with an unstructured covariance matrix and the between-within method of estimating degrees of freedom. Significance levels are FDR corrected.*

#### **HIGHLY ACCURATE vs. LESS ACCURATE PROBLEM-SOLVING**

**Table 5** reports the results of multilevel analyses comparing oxy-Hb across the MM and ZM conditions for participants who were matched for accuracy at the 90% accuracy threshold (*n* = 24). Significant increases in oxy-Hb compared to ZM were observed in 12 channels, encompassing the anterior aspects of the right (channels: 10, 11, 12, 13, 14, 15, and 16) and left (channels 1, 2, 3, 4, and 6) DLPFC. Conversely, significant decreases in oxy-Hb compared to ZM were observed in four channels encompassing the medial PFC (channels: 5, 7, 8, and 9). The results of these analyses are portrayed in **Figure 5**.

Fourteen participants did not meet the 90% accuracy threshold. **Table 6** reports the results of multilevel analyses comparing oxy-Hb across the MM and ZM conditions for these participants. Significant increases in oxy-Hb compared to ZM were observed in seven channels, encompassing the anterior aspects of the left middle/IFG (channels: 1 and 3), and the medial PFC (channels: 7, 8, 9, 10, and 11). Conversely, significant decreases in oxy-Hb compared to ZM were observed in six channels, encompassing the left middle/IFG (channels: 2, 4, and 6), and right middle/IFG (channels: 14, 15, and 16). Channels 5, 12, and 13 did not demonstrate a significant change in oxy-Hb across

the MM and ZM conditions. These results are portrayed in **Figure 6**.

#### **ANCILLARY ANALYSES: DELIBERATION AND LEFT DLPFC ACTIVATION**

Beyond analyses of accuracy on the S-TOL and its relationship to PFC activity on simple and complex problems, we examined the personality trait deliberation as an external validator of the aforementioned findings. As previously mentioned, deliberation refers to the tendency to think carefully before acting. The left DLPFC is crucial for problem-solving, specifically, the extraction of goal information and the generation of an internal problem representation. Accordingly, we predicted that deliberation would moderate the relationship between problem-solving and left DLPFC activation on the S-TOL. Specifically, we hypothesized that individuals higher in trait deliberation would show consistently high levels of left DLPFC activation across both higher and lower levels of task difficulty. Less deliberate individuals were hypothesized to demonstrate greater left DLPFC activation when solving problems of higher difficulty but less activation when solving simpler problems.

For this ancillary analysis, the left DLPFC was defined as our region of interest using channels 1, 2, 3, and 4. The timemarked data points from these channels were aggregated with list-wise deletion because some fNIRS data were missing at random. The aforementioned data screening and filtering yielded a total of 18, 305 data points across 21 participants for analysis of the left DLPFC. To test our hypothesis that Deliberation might moderate activity in the left DLPFC on the S-TOL,

**Table 5 | Multilevel analyses comparing oxy-Hb levels for participants who were matched for high accuracy (***N* **= 24).**


*\*\*p < 0.001, \*p < 0.05. All models were estimated with an unstructured covariance matrix and the between-within method of estimating degrees of freedom. Significance levels are FDR corrected.*

we estimated a multilevel model that examined the Level 1 effect of problem-solving, the Level 2 effect of Deliberation, and the problem-solving × Deliberation cross-level interaction in the prediction of oxy-Hb. The last term of this model was of particular interest because it tested whether or not withinperson differences in oxy-Hb across the ZM and MM conditions varied as a function of between-person differences in trait Deliberation.

As expected, this analysis uncovered a significant cross-level interaction between problem-solving and Deliberation on the S-TOL [*b* = −0*.*01, *SE* = 0*.*00, *t(*18282*)* = −12*.*51, *p <* 0*.*001]. In order to probe the nature of this significant interaction, the effect of problem-solving in the left DLPFC was examined at high (+1 *SD*) and low (−1 *SD*) levels of Deliberation (West and Aiken, 1991). As hypothesized, this analysis revealed that activation in the left DLPFC was higher in the MM relative to the ZM condition for participants who reported lower levels of Deliberation [*b* = 0*.*11, *SE* = 0*.*01, *t(*18282*)* = 18*.*13, *p <* 0*.*0001], as compared to those who reported higher levels of this trait [*b* = 0*.*00, *SE* = 0*.*01, *t(*18282*)* = 0*.*10, *p* = 0*.*92]. That is, whereas those participants who reported higher levels of Deliberation did not show a significant difference in oxy-Hb across the ZM and MM conditions, those participants who reported lower levels of this trait showed higher activation in the MM condition relative to the ZM condition. Furthermore, Deliberation was not significantly related to oxy-Hb during the MM condition [*b* = 0*.*00, *SE* = 0*.*01, *t(*19*)*, *p* = 0*.*92], but it was marginally associated with increased activation during the ZM condition [*b* = 0*.*01, *SE* = 0*.*01, *t(*19*)* = 2*.*04, *p* = 0*.*06]. This latter result suggested that participants who self-reported higher levels of Deliberation may have been more engaged in problem-solving on the S-TOL even on ZM trials. This significant interaction is illustrated in

**Figure 7** and highlights that activation of the left DLPFC during the S-TOL varied predictably as a function of individual differences in Deliberation.

#### **DISCUSSION**

The present study described a new computerized version of the TOL (Shallice, 1982) designed to improve on the shortcomings of alternative variants of this task which were developed for neuroimaging purposes. Other computerized versions of this task employed in neuroimaging studies differed along several task parameters, including the number of moves required to achieve the target configuration, the mode of responding, and the baseline or "control" task used for comparison with more complex problem-solving trials. Variations in these important task attributes could lead to different findings across studies and obscure neural activation patterns associated with problem-solving on this task. The S-TOL was designed to include trials that required no more than three moves to achieve a goal configuration and responses were made using a simple two-alternative forced-choice modality. The baseline task utilized problems that required a minimum of zero moves to achieve the target configuration, and the same task instruction was employed on two- and three-move problems as for ZM problems in order to control for working memory load across task conditions. Using 16-channel fNIRS, activation of the PFC was evaluated by comparing hemodynamic changes in oxy-Hb for conditions requiring two or three moves with those requiring no moves to achieve the target configuration.



*\*\*p < 0.001, \*p < 0.05. All models were estimated with an unstructured covariance matrix and the between-within method of estimating degrees of freedom. Significance levels are FDR corrected.*

Consistent with expectations, participants achieved high levels of accuracy on both two- and three-move problems on the S-TOL (∼89%). When activation on these problems was contrasted with ZM problems, increases in oxy-Hb were observed primarily in the left DLPFC and right medial PFC. Conversely, decreased activation was observed in superior channels in the medial PFC and a single channel in the right IFG. The results of this study partly converge with prior neuroimaging research using other versions of the TOL which found largely bilateral DLPFC activation during the active completion of problems requiring two or more moves to achieve a goal configuration (Baker et al., 1996; Dagher et al., 1999; Newman et al., 2003; Boghi et al., 2006; Rasmussen et al., 2006; Wagner et al., 2006; Just et al., 2007; Den Braber et al., 2008; Fitzgerald et al., 2008; De Ruiter et al., 2009; Ruh et al., 2012). Bilateral engagement of the DLPFC during problem-solving on the TOL, however, has been challenged by research which suggests that the right and left homologs of this region may subserve distinct problem-solving functions, namely, those involved in search depth and goal hierarchy, respectively (Kaller et al., 2011). Search depth refers to the degree of interdependence between consecutive steps in problem-solving, whereas goal hierarchy reflects the degree to which the configuration of the goal state makes the order of single steps either clearly evident or ambiguous. The S-TOL was intentionally designed to contain problems that varied only in search depth (i.e., either no or one intermediate and interdependent move was required to achieve the goal configuration on three-move problems) (see **Figure 8**). All goal hierarchies, however, were unambiguous. Therefore, the observation of predominantly left DLPFC activation on the S-TOL is consistent with Kaller et al. (2011) which found similarly lateralized DLPFC activity on unambiguous problems with greater search depth. In addition, the present study extended these findings by revealing

**FIGURE 6 | Areas of significant activation and deactivation associated with solving two- and three-move problems on the Scarborough adaptation of the Tower of London (S-TOL) task for participants who did not meet the 90% accuracy threshold (***N* **= 14).** Areas of significant activation are denoted in red, and areas showing significant deactivation are in blue. Data represent *t*-scores for the contrast of multiple-move and zero-move conditions (*p <* 0*.*05, False-Discovery Rate-corrected).

that increased activity in the right portion of the medial PFC may also play a role in solving unambiguous problems which vary in search depth.

Whereas increased activation was observed in primarily left DLPFC for two- and three-move problems, significant deactivation was also detected in dorsal aspects of the medial PFC and right IFG. Many studies using the TOL did not report on areas which showed significant deactivation on this task; however, Boghi et al. (2006) found pronounced reductions in activity in the medial PFC using a modified TOL task. Interestingly, comparisons of activation within the medial PFC and right IFG for highly accurate vs. less accurate participants revealed distinct

**FIGURE 8 | Sample three-move problems from the Scarborough adaptation of the Tower of London task that are low (left) vs. high (right) in search depth.** These sample problems require a minimum of three moves to solve the problem and they either require an intermediate move (left) or do not require an intermediate move to achieve the target configuration (right).

patterns of activity in these regions: highly accurate individuals showed greater deactivation in the medial PFC, whereas less accurate participants showed less activation within the right IFG. It is important to note that these apparent differences in regional brain activity were observed even though the range of accuracy scores on the S-TOL was restricted.

An extensive body of research has shown that whereas the performance of attention-demanding cognitive tasks is typically associated with increased activation in the DLPFC and decreased activation in the medial PFC (Fox et al., 2005), states of passive rest and self-focused attention are typically associated with increased activity in the medial PFC (Gusnard et al., 2001). In light of these previous studies, we speculate that the observed differences in activation across levels of accuracy reflect differences in the degree to which participants cognitively immersed themselves in the S-TOL. Specifically, we suggest that participants who performed the S-TOL with greater accuracy may have subjectively engaged themselves in the S-TOL to a greater extent. Indeed, the medial PFC is a region that is crucial for self-referential thought (e.g., Abraham, 2013; Araujo et al., 2013; D'argembeau, 2013; Moran et al., 2013) and research from the field of motivational psychology highlights that transient decreases of self-focused attention are typically reported when people experience themselves as being subjectively immersed in an activity (Csikszentmihalyi, 1990). In keeping with these ideas about the presently observed differential pattern of PFC activity across levels of accuracy on the S-TOL, other fNIRS work has shown that more extensive deactivation in the medial PFC during active task performance is associated with greater success in inhibiting a motor response (Rodrigo et al., 2014). Moreover, using positron emission tomography, Beauchamp et al. (2003) asked participants to complete a computerized TOL over four separate scanning sessions to observe changes in neural systems associated with learning on this task. They found that as performance on the TOL improved over subsequent sessions, the medial PFC showed a concomitant decrease in activity. Future neuroimaging studies using the S-TOL (and other problem-solving tasks) should continue to examine how performance outcomes are associated with different patterns of neural activation across the PFC and should specifically consider the contribution of participants' degree of subjective task immersion and self-focused attention to medial PFC deactivation.

Reduced activity in the right IFG was an unexpected finding given that inhibition and attentional control, functions frequently ascribed to this region (Hampshire et al., 2010), are not typically referenced in theoretical models of problem-solving. Presumably, inhibitory control is an important component cognitive function in problem-solving ability—indeed, when solving a problem, individuals must evaluate and select one allowable operator from the larger set of allowable operators, which themselves must be inhibited at each step of the problem-solving process (Ward and Allport, 1997). To investigate the potential role of the right IFG in problem-solving, activation during MM vs. ZM problems on the S-TOL was compared between highly accurate and less accurate participants. Less accurate participants showed attenuated activity in the right IFG, whereas highly accurate participants displayed greater activation in this region. Accuracy on the S-TOL may therefore be subserved, at least in part, by the efficiency with which inhibitory processes are engaged during active problemsolving on items requiring at least two or three moves to achieve a target configuration. This speculation is also supported by neuropsychological research which indicates that accuracy on the TOL is strongly related to performance on tests of inhibitory control (Welsh et al., 1999; Miyake et al., 2000). It should be noted, however, that although lower levels activation were observed in the left and right IFG for less accurate participants, a region within the left IFG appeared to show higher activation. This perhaps suggests that the left IFG plays a unique role in problemsolving that extends beyond attention related task engagement. Future research should investigate this observation to further delineate the contributions of different regions of the PFC to problem-solving ability.

As an external validator of the current findings, patterns of left DLPFC activation on the S-TOL were examined in relation to a personality trait descriptor of decision-making, referred to as deliberation. This trait reflects the extent to which individuals think carefully before acting or speaking (McCrae and Costa, 2010). More deliberate individuals showed similar levels of activation in the left DLPFC on both lower and higher difficulty problems. These findings suggest that individuals who tend to think problems through carefully before acting may engage neural processes necessary for effective problem-solving regardless of task difficulty. Individuals that described themselves as less deliberate, however, had less activation in the left DLPFC while solving lower difficulty problems but higher activation on higher difficulty problems. Theoretically consistent associations between left DLPFC activation and individual differences in trait deliberation provide convergent validity for the S-TOL as a task that may effectively probe neural systems involved in problem-solving ability.

#### **LIMITATIONS AND FUTURE DIRECTIONS**

A number of limitations should be considered when interpreting the current findings. First, a greater number of females were recruited in this study. There is emerging evidence suggesting that females completing the TOL may show greater activation bilaterally in the DLPFC and right parietal cortex than their male counterparts (Boghi et al., 2006). Subsequent research should achieve a greater balance in female and male participants to evaluate possible differences between these groups in PFC activation. Second, the S-TOL did not include highly complex problems (i.e., those requiring a minimum of four or more moves to reach the goal configuration). This task was intentionally designed to evaluate problems requiring a minimum of two or three moves with the aim of ultimately translating the S-TOL to clinical populations that may have difficulty correctly answering more complex problems. Indeed, prior neuroimaging studies using variants of the TOL reported accuracy that ranged from 46 to 82% for five-move problems in non-clinical samples, although nearly one-third of studies published since 2000 did not report accuracy on the TOL (see **Table 1**). A difficulty with incorporating more complex problems in neuroimaging studies employing block designs is that lower levels of accuracy may be confounded with neural activation differences between clinical and non-clinical groups. The S-TOL achieved reasonably high levels of accuracy on two- and three-move problems, although generalizations derived from this task may be limited to relatively less complex problem-solving tasks. Third, larger sample sizes would provide more statistical power to evaluate the relationship between trait measures of problem-solving (e.g., deliberation) and neural activation on the S-TOL. The current exploratory findings provided preliminary support for the relationship of self-reported deliberation to left DLPFC activation on the S-TOL; however, examination of other potentially relevant personality traits (e.g., impulsiveness, excitement-seeking) would be permitted with suitably larger sample sizes. Indeed, fNIRS may hold potential to significantly advance personality neuroscience by providing researchers with a cost-effective tool to gather large sample sizes that are necessary to provide adequate power for personality-based research. Fourth, it should be noted that more research is needed to further validate the S-TOL above and beyond our initial evidence presented here, and to provide evidence demonstrating its specific advantages over other computerized TOL tasks. Furthermore, more research is needed with clinical samples to determine whether the S-TOL may indeed be suitable for translational research with these groups. Finally, the S-TOL and other conventional tower tasks are unable to identify the neural substrate underlying problems that are discovered or created. Rather, the S-TOL is a presented problem with clearly defined task parameters. It will be important for future neuroimaging studies to push the traditional boundaries of problem-solving research to explore the neural underpinnings involved in solving problems requiring discovery or creation by the problem-solver.

#### **AUTHOR CONTRIBUTIONS**

Anthony C. Ruocco designed the study, developed the problemsolving task, and wrote the majority of the manuscript. Achala H. Rodrigo assisted in the development of the problem-solving task, conducted statistical analyses, and wrote portions of the manuscript. Jaeger Lam and Bryanna Graves wrote portions of the manuscript. Stefano I. Di Domenico and Hasan Ayaz assisted with statistical analysis and wrote portions of the manuscript.

#### **ACKNOWLEDGMENTS**

This research was supported by a New Investigator Award (Funding Reference Number: MSH–130177) from the Canadian Institutes of Health Research to Dr. Ruocco. Mr. Rodrigo was supported by an Alexander Graham Bell Canada Graduate Scholarship from the Natural Sciences and Engineering Research Council of Canada.

#### **REFERENCES**


of London planning task: implication for effortful processing. *Eur. J. Neurosci.* 17, 2219–2225. doi: 10.1046/j.1460-9568.2003.02633.x


**Conflict of Interest Statement:** The optical brain imaging instrumentation utilized in the present research was manufactured by fNIR Devices, LLC. Dr. Hasan Ayaz was involved in the development of the technology and thus offered a minor share in fNIR Devices, LLC. All other authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 01 November 2013; accepted: 13 March 2014; published online: 28 March 2014.*

*Citation: Ruocco AC, Rodrigo AH, Lam J, Di Domenico SI, Graves B and Ayaz H (2014) A problem-solving task specialized for functional neuroimaging: validation of the Scarborough adaptation of the Tower of London (S-TOL) using near-infrared spectroscopy. Front. Hum. Neurosci. 8:185. doi: 10.3389/fnhum.2014.00185*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Ruocco, Rodrigo, Lam, Di Domenico, Graves and Ayaz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Differences in time course activation of dorsolateral prefrontal cortex associated with low or high risk choices in a gambling task

#### *Stefano Bembich1, Andrea Clarici <sup>2</sup> \*, Cristina Vecchiet 1, Giulio Baldassi 1, Gabriele Cont <sup>1</sup> and Sergio Demarini <sup>1</sup>*

*<sup>1</sup> Institute for Maternal and Child Health, IRCCS "Burlo Garofolo", Trieste, Italy*

*<sup>2</sup> Psychiatric Clinic Unit, Department of Medical, Surgical and Health Sciences, University of Trieste, Trieste, Italy*

#### *Edited by:*

*John J. Foxe, Albert Einstein College of Medicine, USA*

#### *Reviewed by:*

*Ippeita Dan, Jichi Medical University, Japan Colm Gerard Connolly, University of California San Francisco, USA*

#### *\*Correspondence:*

*Andrea Clarici, Psychiatric Clinic Unit, Department of Medical, Surgical and Health Science, University of Trieste, Via Paolo de Ralli 5, 34138 Trieste, Italy e-mail: clarici@units.it*

Prefrontal cortex plays an important role in decision making (DM), supporting choices in the ordinary uncertainty of everyday life. To assess DM in an unpredictable situation, a playing card task, such as the Iowa Gambling Task (IGT), has been proposed. This task is supposed to specifically test emotion-based learning, linked to the integrity of the ventromedial prefrontal cortex (VMPFC). However, the dorsolateral prefrontal cortex (DLPFC) has demonstrated a role in IGT performance too. Our aim was to study, by multichannel near-infrared spectroscopy, the contribution of DLPFC to the IGT execution over time. We tested the hypothesis that low and high risk choices would differentially activate DLPFC, as IGT execution progressed. We enrolled 11 healthy adults. To identify DLPFC activation associated with IGT choices, we compared regional differences in oxy-hemoglobin variation, from baseline to the event. The time course of task execution was divided in four periods, each one consisting of 25 choices, and DLPFC activation was distinctly analyzed for low and high risk choices in each period. We found different time courses in DLPFC activation, associated with low or high risk choices. During the first period, a significant DLPFC activation emerged with low risk choices, whereas, during the second period, we found a cortical activation with high risk choices. Then, DLPFC activation decreased to non-significant levels during the third and fourth period. This study shows that DLPFC involvement in IGT execution is differentiated over time and according to choice risk level. DLPFC is activated only in the first half of the task, earlier by low risk and later by high risk choices. We speculate that DLPFC may sustain initial and more cognitive functions, such as attention shifting and response inhibition. The lack of DLPFC activation, as the task progresses, may be due to VMPFC activation, not detectable by fNIRS, which takes over the IGT execution in its second half.

**Keywords: multichannel NIRS, DLPFC, risk, Iowa Gambling Task, attention shifting, response inhibition**

#### **INTRODUCTION**

There is a general agreement in the role of the prefrontal cortex in decision-making (DM), under conditions of complexity and unpredictability (e.g., Bechara et al., 2000a; Bechara, 2001; Clark et al., 2003; Wise, 2008; Gläscher et al., 2012). According to Damasio's "somatic marker hypothesis" (Damasio, 1994), patients with ventromedial prefrontal cortex (VMPFC) damage develop severe impairments in personal and social DM, while other intellectual abilities are usually well preserved. There appears to be an important deficit in the emotional learning process: the somatic activation of an emotional response is no longer associated with DM, in uncertain conditions, therefore impairing the DM process itself.

In order to assess the complex nature of human DM under uncertainty, the Iowa Gambling Task (IGT) has been developed (Bechara et al., 1994). It is a playing card task which simulates real-life decisions. The IGT is supposed to specifically test emotion-based learning, linked to DM under uncertainty. For example, during the task, participants show an anticipatory skin conductance response associated with risky choices (Bechara et al., 1999), while patients with a damage in VMPFC, both fail the test and do not show anticipatory skin conductance responses (Bechara et al., 1994, 2000b). Additionally, VMPFC may play a role in the continuous updating of expectations about reward and punishment based on experience (Fellows and Farah, 2005; Oya et al., 2005; Stocco and Fum, 2007).

Other areas of the prefrontal cortex may play a relevant role in DM processes associated with the IGT execution. Among these, the most important seems to be the dorsolateral prefrontal cortex (DLPFC), whose role in a gambling task has been demonstrated both by clinical and neuroimaging studies. Patients with a lesion in the right DLPFC showed significantly worse performances than patients with focal lesions in other areas (with the exclusion of VMPFC) or controls (e.g., Manes et al., 2002; Clark et al., 2003). Moreover, Fellows and Farah (2005) found that eliminating the need for a reversal learning from IGT improved the performance of VMPFC patients, but not that of DLPFC patients. In addition, using positron emission tomography (PET), a significant activation in right DLPFC was found during IGT execution (e.g., Ernst et al., 2002, 2003; Bolla et al., 2003, 2005).

In a recent functional magnetic resonance imaging (fMRI) research, a modified double deck version of IGT was used to study the role of prefrontal cortex in task rule learning (Hartstra et al., 2010). A differential involvement of DLPFC, anterior cingulated cortex (ACC)/pre-supplementary motor area (pre-SMA) and medial orbital frontal cortex (med-OFC) was observed, as IGT execution progressed. DLPFC and ACC/pre-SMA were more active during the first phase of IGT performance, while med-OFC was more active in later phases. However, differential cortical activation according to different choice risk levels was not studied.

So far IGT has been studied only by fMRI (e.g., Lin et al., 2008; Lawrence et al., 2009) or PET (e.g., Ernst et al., 2002). Consequently, the execution of the original task had to be adapted to the spatial and technical limitations of such devices (e.g., implementing a computerized version of IGT or limiting cerebral activity detection to a restricted part of the test). As a possible alternative, closer to real life situations, multichannel near-infrared spectroscopy (NIRS) may be used (e.g., Cazzell et al., 2012). It permits detection of regional hemoglobin concentration changes in the cortex induced by brain activity with fairly good spatial resolution and very high temporal resolution (Maki et al., 1995; Taga et al., 2003), even if detection is limited to the cortical surface. NIRS cortical monitoring is non-invasive, does not require strict motion restriction during measurements (subjects are not put into a scanner), is safe (Ito et al., 2000) and resistant to movement artifacts. These characteristics make it particularly suitable for research concerning also complex cognitive functions with awake healthy subjects in a relatively unrestricted environment.

Although DLPFC activation during IGT has been observed mainly in the initial phases of the performance (Hartstra et al., 2010), there are no studies assessing if such activation is differentially associated with high- or low-risk choices as the task goes on. The aim of this pilot exploratory study was to further assess, by multichannel NIRS, the dynamic contribution of DLPFC to the IGT execution over time, in healthy adults. We tested the hypothesis that DLPFC would be differentially activated by low- and high-risk choices as the IGT execution progressed.

#### **MATERIALS AND METHODS PARTICIPANTS**

A total of 11 participants (seven females), aged between 21 and 38 years (mean: 27.7 ± 5.7), participated to this study. They were all healthy and there was no history of neurological, sensory, psychiatric or addictive disorders. All were right handed, as assessed by the Edinburgh Questionnaire (Oldfield, 1971). Written informed consent was obtained from each participant after full procedural and technical explanation of the experiment. The ethics committee of the Institute for Maternal and Child Health IRCCS "Burlo Garofolo"—Trieste (Italy), where all tests were conducted, approved the research.

#### **THE IOWA GAMBLING TASK**

We administered the IGT as proposed in the original study of Bechara et al. (1994), using physical playing cards. Participants chose 100 cards from four decks (A, B, C, D) in any sequence they preferred. There were two advantageous or low risk decks (C and D), and two disadvantageous or high risk decks (A and B). The schedule of rewards and punishments differed in decks: in deck A (disadvantageous/high risk) and deck C (advantageous/low risk) there were smaller but more frequent unpredicted punishments, while in deck B (disadvantageous/high risk) and deck D (advantageous/low risk) punishments were greater but less frequent. The only difference from the original task consisted in the value of rewards and punishments and in the currency (Euros rather than Dollars). Participants were actually given a total amount of C 2000 (fake money) at the beginning of the task. When playing on disadvantageous decks, participants won C 100 for every card turned, but incurred in losses between C 150 and C 1250. When playing with advantageous decks, they won C 50 for every card turned and incurred in losses between C 25 and C 250. If subjects chose all the cards in decks A and B they lost comprehensively C 1000, if they chose all the cards in decks C and D they won comprehensively C 1000. Every win or loss actually modified the amount of money held by the participant, and he/she was invited to consider the amount of fake money possessed, won or lost as real and own money. The aim of the game was to win as much money as possible, and to avoid disadvantageous decks. A total net score of less than 50 advantageous choices is the cutoff value that indicates that participants are not choosing cards advantageously.

#### **MULTICHANNEL NIRS RECORDING**

In order to detect cortical activity, we used the Hitachi ETG-100 OT device (Hitachi Medical Corporation, Tokyo, Japan), which is a multichannel NIRS system (Maki et al., 1995) recording simultaneously from 24 channels on the cortex. It emits near-infrared light at two wavelengths, 780 and 830 nm, and the reflected light is sampled once every 100 ms. This device estimates changes in the concentration of oxy-hemoglobin (HbO2) deoxyhemoglobin and total hemoglobin in response to stimulation in the unit of mM•mm, that is the product of the hemoglobin concentration changes expressed in millimolar and the optical path length expressed in millimeters.

For detecting DLPFC activation during the IGT, we used a 4 × 4 plastic fibers holder containing 16 optical fibers (or optodes) of 1 mm in diameter, 8 emitters and 8 detectors, which were placed 3 cm apart and allowed 24 detecting channels. The fibers holder was positioned above the frontal lobes of participants using the international 10–20 EEG placement system (Jasper, 1958). The anterior row of channels (**Figure 1**) was put on the virtual line joining Fp1 and Fp2 points, placing channel 23 on Fpz. When the holder was positioned, it was made sure that the fibers touched the scalp. The device automatically detects whether the contact is adequate to measure emerging photons.

Cortical areas covered by each channel were identified referring to the methods developed by Tsuzuki et al. (2007) and by Singh et al. (2005), which allow fNIRS data to be probabilistically registered to the standard Montreal Neurological

Institute (MNI) space also without using a 3D digitizer, as in our case. The probabilistic localization for each channel is given in **Table 1**. Cerebral labeling is based on the brain atlas constructed by Tzourio-Mazoyer et al. (2002). To have an estimate of channel projections on a rendered brain, we used the "Spatial registration of NIRS channels location" function of the NIRS-SPM version 4\_r1 software (Ye et al., 2009), which is a SPM5 and MATLAB based software package (http://bisp*.*kaist*.*ac*.*kr/ NIRS-SPM). Using the "Stand alone" option, not needing MRI images, we obtained a spatial representation of the channel locations on a rendered brain, referring to the MNI coordinates reported in **Table 1**.

#### **PROCEDURE**

We administered the IGT in a soft lighted and sound isolated room, and participants sat on a chair in front of a desk. In normal adults, the cerebral vascular response takes about 10–12 s to be completed (Wobst et al., 2001; Meek, 2002). Thus, we adjusted the IGT test to detect DLPFC activation associated to each choice: approximately 12 s were allowed between choosing cards. Specifically, the participant could make each choice only after the presentation of a green slide on a computer screen, which was positioned in front of him or her on the same table where the IGT was performed, behind the four card packs. The slide was presented for 6 s, then there was a break of other 6 s, when the computer screen turned black. Then, the green slide was shown again, for the successive IGT choice and so it went on for the 100 picks of the entire DM task (**Figure 2**). We hypothesized that, separating each choice from the following one by 12 s, it would allow us to assess the DLPFC activation associated with each choice (Schroeter et al., 2004), with a limited interference on the IGT learning processes (Gupta et al., 2009). The entire experimental procedure was completed in about 25 min.

#### **DATA ANALYSIS**

Our analyses focused on the increase of HbO2 concentration, which is considered an estimate of cerebral activation (e.g., Meek, 2002). Possible components of the HbO2 signal related to slow fluctuations of cerebral blood flow and heartbeat noise were



*R, right; L, left; F, frontal; Orb, orbital; sup, superior. \*Channels located on the nasion-Inion virtual line.*

removed by bandpass filtering between 0.02 and 1 Hz and, in order to prevent movement artifacts, a further filter was used to remove detections with rapid changes in HbO2 concentration (signal variations *>* 0.1 mM•mm over two consecutive samples). We also visually checked the signals recorded in each channel of all subjects, in order to detect low signal-to-noise ratio due to bad transmission of near-infrared light (e.g., due to hair obstruction). There was no need to exclude any registrations from data analysis due to a bad signal-to-noise ratio.

An event design was used and the active channels were identified by means of paired *t*-tests. In each participant, for every channel, an arbitrary baseline was calculated as the mean in relative changes of HbO2 in the 2 s before the onset of the green slide. The hemodynamic response associated with each choice was calculated as the mean change in HbO2 concentration over the 10 s after the onset. To identify the activated channels, e.g., those showing a HbO2 increase, we performed one-tailed paired *t*-tests and compared, for each channel, the baseline and the choice associated hemodynamic response. In order to evidence possible changes in DLPFC activation associated with different choice risk levels, as the performance went on, the time course of task execution was divided in four periods, each one consisting of 25 choices, and DLPFC activation was distinctly analyzed for low and high risk choices in each period.

To control Type I error in multiple testing situations, we used a false discovery rate (FDR) approach (Genovese et al., 2002; Singh and Dan, 2006), that controls the proportion of false positives among channels that are significantly detected. We selected a *q* value of 0.05, so that there were no more than 5% false positives (on average) in the number of channels emerging with significant contrasts. Statistical analyses were conducted using SPSS version 13.0 for Windows (SPSS Inc., Chicago, IL, USA).

#### **RESULTS**

All but one participant in our sample chose advantageously, showing an IGT net score above the cut-off threshold of at least 50 choices from advantageous decks. **Table 2** reports single performances.

During the first task period (1st–25th choice), five channels passed the FDR threshold (*P <* FDR 0.05) in association with low risk choices (advantageous card decks C and D). They were

**FIGURE 2 | Illustration of the experimental procedure concerning the first two choices.** Each choice could be made just after the presentation of the green slide, but within 12 s (e.g., the interval between choices we selected to allow the detection of a complete hemodynamic response associated with each decision). The black screen was inserted to fill the 12 s inter-choice interval creating an alternation with the green slide, which had to be again presented to allow the next choice. This was repeated for 100 times.


channel 13 (*t*<sup>10</sup> = −2*.*818; *P* = 0.009), located on left superior frontal gyrus, channel 15 [*t*(10) = −6*.*471; *P <* 0*.*001], located on right superior frontal gyrus, channel 16 [*t*(10) = −3*.*454; *P* = 0.003], located on right/left superior and medial frontal gyrus, channel 19 [*t*(10) = −3*.*042; *P* = 0.006], located on right superior frontal gyrus, and channel 22 [*t*(10) = −3*.*222; *P* = 0.0045], located on right superior/orbital frontal gyrus (**Figures 3**, **4**). Instead, in association with high risk choices (disadvantageous card decks A and B), no channel passed the FDR threshold.

During the second task period (26th–50th choice), in association with low risk choices, only channel 19 [*t*(10) = −3*.*910; *P* = 0*.*0015], located on right superior frontal gyrus, passed the FDR threshold. In association with high risk choices, on the other hand, eight channels passed the FDR threshold (*P <* FDR 0.05). They were channel 2 [*t*(10) = −5*.*858; *P* = 0.006], located on right/left superior and medial frontal gyrus, channel 3 [*t*(10) = −4*.*300; *P* = 0.001], located on left superior frontal gyrus, channel 9 [*t*(10) = −3*.*841; *P* = 0.0015], located on right/left superior and medial frontal gyrus, channel 12 [*t*(10) = −3*.*441; *P* = 0.003], located on right superior frontal gyrus, channel 13 [*t*(10) = −3*.*349; *P* = 0.0035], located on left superior frontal gyrus, channel 15 [*t*(10) = −4*.*078; *P* = 0.001], located on right superior frontal gyrus, channel 16 [*t*(10) = −2*.*688; *P* = 0.0115], located on right/left superior and medial frontal gyrus, and channel 19 [*t*(10) = −4*.*382; *P* = 0.0005], located on right superior frontal gyrus (**Figures 5**, **6**).

During the third and fourth task period (51st–100th choice), no channel passed the FDR threshold (*P <* FDR 0.05), for either low- or high-risk choices.

Four channels were activated in association with both low- or high-risk choices. They were channels 13, 15, 16, and 19, covering left, right and central portions of DLPFC. Additionally, in association with low-risk choices, also the right frontal pole was activated (channel 22), while, in association with high-risk choices, the DLPFC activation extended more posteriorly, to channels 2, 3, and 9.

**FIGURE 3 | Cortical location, on a rendered brain, of channels found significantly activated in association with low risk choices (evidence in green), during the first IGT period.**

**FIGURE 5 | Cortical location, on a rendered brain, of channels found significantly activated in association with high risk choices (evidence in pink), or in high and low risk choices (evidence in pink/green), during the second task period.**

#### **DISCUSSION**

We used multichannel NIRS to study if DLPFC is differentially activated, over time, by low or high risk choices during the execution of the complete administration of IGT, a playing card task developed to study DM under uncertainty. We have found that DLPFC significantly contributes to the IGT execution in the first half of the task. Additionally, we found different time courses in its activation as the task went on, associated with low or high risk choices. Specifically, DLPFC was shown to be activated by low risk choices in the first period (1st–25th choice) and by high risk choices in the second period (26th–50th choice) of IGT performance. Activated DLPFC areas associated with low or high risk choices partially overlapped in both left, right and central portions of the DLPFC region monitored. In association with low risk choices (first period), along with activation of left/right superior frontal gyrus and left/right medial frontal gyrus (channels 13, 15, 16, and 19), also the right frontal pole was activated (channel 22). In association with high risk choices (second period), the DLPFC activation was more extensive and involved more posterior areas, possibly including the supplementary motor area (channels 2 and 3). In both cases, activation was slightly lateralized on the right side. No significant variations in DLPFC activity emerged during the second half of IGT execution, regardless of choice risk level.

The involvement of DLPFC, especially with a right lateralization, in IGT execution has already been shown. Clinically, a bad IGT performance was observed in patients with right DLPFC lesions (Clark et al., 2003; Fellows and Farah, 2005) and, by neuroimaging, DLPFC cortical areas were found activated during the IGT test (Ernst et al., 2002, 2003; Bolla et al., 2003, 2005). Besides DLPFC, other areas may play a role in DM, such as VMPFC (Bechara et al., 2000a), the amygdala (Bechara, 2001; Bar-On et al., 2003) and the insula (Lin et al., 2008; Lawrence et al., 2009), but such areas are too deeply located to be detected by NIRS devices. Primary and secondary sensory areas (Bechara, 2001; Bar-On et al., 2003) are involved in DM as well, but they were outside our region of interest. More recently, Hartstra et al. (2010) have demonstrated, by fMRI, that activation of DLPFC during

IGT execution changes over time: it is significant in the first part of the modified task they used and becomes non-significant later. These authors speculate that earlier activation observed in DLPFC may be attributed to the involvement of working memory, which would be necessary for active learning of task rules (see also Bechara et al., 1998; Clark et al., 2003; Fellows and Farah, 2005; Martinez-Selva et al., 2006). In addition, they hypothesize that later med-OFC activation is associated with the representation of learned reward values, guiding successive choices. In agreement with the previous study, we showed an early significant involvement of DLPFC in IGT performance as well, but we found a differential activation according to choice risk level. DLPFC was activated earlier by low risk choices and later by high risk choices. Study designs differed, as Hartstra et al. (2010) aimed to study the role of prefrontal cortex in rules learning, while we aimed to show the DLPFC contribution to IGT performance in the setting of high or low risk choices. In this regard, a meta-analysis of fMRI studies has demonstrated that risk-related DLPFC activation occurs when a choice is involved (Mohr et al., 2010). Moreover, it has been recently found that DLPFC mediates the discrimination of IGT disadvantageous choices (Christakou et al., 2009), and the activation intensity in such region has been positively correlated with task performance (Lawrence et al., 2009).

Our results on the differential DLPFC activation over time, in association with low or high risk choices during the IGT, may seem to indicate that working memory is not the only neurocognitive resource supplied by DLPFC. We speculate that two superior cognitive functions, mediated by DLPFC as well, may explain our observations: attention shifting and response inhibition. It is worth noting that, in IGT, high risk choices are those with a greater immediate reward, but with a delayed greater loss. On the contrary, low risk choices are associated with lower immediate rewards, but with a delayed gain. DLPFC activation showed by our participants seems to indicate a shift in attention, as the task went on. Their attention shifted from the decks with lower immediate rewards, in the first task period (1st–25th choice), to those with greater losses, in the second task period (26th– 50th choice). In both cases, an attention response, mediated by DLPFC (e.g., Manes et al., 2002), was elicited by those decks (and related choices) that seemed to be more disadvantageous (see also Christakou et al., 2009; Lawrence et al., 2009). Moreover, in the second period of the task, when DLPFC was activated in association with high risk choices, frontocentral cortical areas (channels 9 and 16) and the supplementary motor area (channels 2 and 3) were also activated. Both regions have been associated with response inhibition and behavioral control (Elliott et al., 2000; van Gaal et al., 2008; Fassbender et al., 2009).

Our experimental procedure aimed to assess DLPFC activation associated with low or high risk choices, as they were made. Consequently, the activation of cortical regions with a role in response inhibition was detected when the high-risk decision was made. We speculate that the extended DLPFC activation in the second IGT quarter, may reflect a conflict between previously learned and newer rules, in order to make the best decision. The activation involving the right frontal pole in the first quarter may indicate an attempt to elaborate more abstract task rules (e.g., Venkatraman and Huettel, 2012), as an initial approach to an unknown complex task. Concerning the absence of DLPFC activation in the third and fourth period of the IGT, we speculate that it may be due to VMPFC taking over the task, which is undetectable by fNIRS. Indirectly, this hypothesis is supported by Hartstra et al. (2010), who found only a later involvement of medial prefrontal cortex regions (med-OFC in their case) in IGT performance. Moreover, Lawrence et al. (2009) found a linear decrease in lateral OFC and pre-supplementary motor area activation as IGT execution progressed.

Our study has some obvious limitations. First, we enrolled a limited number of subjects and it should be considered a pilot study. Although functional neuroimaging studies are typically performed in small populations, our results need to be confirmed in a larger sample. Characterizing participants for personality traits and mood state would further improve such a study, since it has been proved that these variables have an influence on IGT performance (Buelow and Suhr, 2009). Second, multichannel NIRS has a poorer spatial resolution than fMRI. However, previous research has shown a good correlation between these two techniques (Toronov et al., 2001; Strangman et al., 2002). Third, NIRS measurement of hemoglobin variations are limited to the lateral surface of the cerebral cortex. Fourth, the experimental design we chose did not allow us to discriminate among the specific prefrontal functions involved in the early distinction between low- and high-risk choices. Finally, some possible systemic effects we have not directly checked, e.g., blood flow variation in the scalp during the experiment, could have had an influence on our results. However, this was probably not the case because an asymmetric pattern of cortical activation was observed.

In conclusion, this study has further deepened the complex role of DLPFC to early DM under uncertainty process, as assessed by IGT. We have shown that, in the first half of the performance, DLPFC activation differs over time, according to the risk level of the choice. We propose that, beside working memory, other complex functions, such as attention shifting and response inhibition, are involved in sustaining the DLPFC role in discriminating disadvantageous choices during the task. Thereafter, as rule learning establishes, DM under uncertainty seems to be prevalently mediated by VMPFC. Thus, it emerges a wide, complex and temporally dynamic involvement of the prefrontal cortex in decisions concerning the ordinary uncertainty of everyday life.

#### **ACKNOWLEDGMENT**

This study has been financed by the grant N. 50/11 of the Institute for Maternal and Child Health—IRCCS "Burlo Garofolo"— Trieste, Italy.

### **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 September 2013; accepted: 06 June 2014; published online: 24 June 2014. Citation: Bembich S, Clarici A, Vecchiet C, Baldassi G, Cont G and Demarini S (2014) Differences in time course activation of dorsolateral prefrontal cortex associated with low or high risk choices in a gambling task. Front. Hum. Neurosci. 8:464. doi: 10.3389/ fnhum.2014.00464*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Bembich, Clarici, Vecchiet, Baldassi, Cont and Demarini. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Near-infrared spectroscopy (NIRS) in functional research of prefrontal cortex

Nobuo Masataka<sup>1</sup> \*, Leonid Perlovsky <sup>2</sup> and Kazuo Hiraki <sup>3</sup>

*<sup>1</sup> Primate Research Institute, Kyoto University, Inuyama, Japan, <sup>2</sup> Psychology Department, Northeastern University, Boston, MA, USA, <sup>3</sup> Department of General Systems Studies and Center for Evolutionary Cognitive Sciences, The University of Tokyo, Tokyo, Japan*

Keywords: near infrared spectroscopy, prefrontal cortex (PFC), functional magnetic resonance imaging (fMRI), oxy-hemoglobin, hemodynamics, cognition, emotions

Functional near-infrared spectroscopy (fNIRS) uses specific wavelengths of light to provide measures of cerebral oxygenated and deoxygenated hemoglobin that are correlated with the functional magnetic functional imaging (fMRI) BOLD signal. fNIRS has emerged during the last decade as a promising non-invasive neuroimaging tool and has used to monitor various types of brain activities during motor and cognitive tasks with increasing interest from research communities. One can see classical comprehensive reviews of such investigations elsewhere (Hoshi, 2003; Koizumi et al., 2003; Obrig and Villringer, 2003). While various MRI methods documented in the literatures have provided essential information about the brain systems concerning these issues up to the present, they have two major limitations, their requirement for participant immobility and their high operational cost. Particularly, the former rules out the use of such techniques for investigating brain dynamics during every activities such as walking and running. On the other hand, fNIRS can be used under less body constraints than other imaging modalities require. Moreover, it is quiet (no operating sound). Thus, the conditions for performing fNIRS measurements are much more comfortable for human as well as non-human subjects (Wakita et al., 2010). In addition, it provides higher temporal resolution (faster sampling frequency). In this respect, fNIRS is useful for studying brain activity under more "natural" and therefore much more variable conditions though it can only monitor cortical regions with less spatial resolution (usually in the centimeter range).

#### Edited and reviewed by:

*Srikantan S. Nagarajan, University of California, San Francisco, USA*

#### \*Correspondence:

*Nobuo Masataka, masataka@pri.kyoto-u.ac.jp*

Received: *22 November 2014* Accepted: *27 April 2015* Published: *12 May 2015*

#### Citation:

*Masataka N, Perlovsky L and Hiraki K (2015) Near-infrared spectroscopy (NIRS) in functional research of prefrontal cortex. Front. Hum. Neurosci. 9:274. doi: 10.3389/fnhum.2015.00274*

Since fNIRS is a non-invasive, safe and portable method, topics that NIRS researches could address are highly variable, and more diverse than those with other non-invasive imaging research techniques. That fact makes fNIRS an ideal candidate for monitoring activity of prefrontal cortex, which has been undertaken with respect to more broadly based issues across the broad expanse of research communities. In fact, many academic journals in diverse areas publish findings obtained from pioneering fNIRS measurements of prefrontal regions. We must admit that this diversity of fNIRS applications makes difficult to access accumulated experience. Protocols of fNIRS recordings from various investigations are not necessarily well-standardized and not necessarily well-grounded on available knowledge of appropriate fNIRS use. One of the purposes of the current research topic is to address this issue across both theoretical and empirical aspects.

In order to realize the high potential of fNIRS, effective discrimination between physiological noise originating from forehead skin hemodynamic and cerebral signals is a pre-requisite. Main sources of physiological noise are global and local blood flow regulation processes on multiple time scales. Having attempted to identify the main physiological noise contributions in fNIRS forehead signals and to develop a method for physiological de-noising of fNIRS data, Kirilina et al. (2013) documents a set of physiological regressors, which are used for physiological de-noising of fNIRS signals. New imaging methods to fuse fNIRS measurements and fMRI data are presented to reveal the spatiotemporal dynamics of the hemodynamic responses with high spatiotemporal resolution across the brain (Yuan and Ye, 2013). Recognition algorithm is described for recognition whether one taps the left hand or the right hand (Hai et al., 2013). In the study, data with noises and artifacts collected from a multi-channel system is pre-processed using a Savitzky-Golay filter for getting more smooth data. These results are confirmed by another study (Wakita, 2014).

A more ambitious and challenging attempt is set out by two groups of scientists independently from each other, but in a similar manner, to explore the possibility that fNIRS can serve as a replacement of fMRI in clinical settings. In order to examine whether fNIRS is equipped with necessary sensitivity as a potential alternative of fMRI, they investigate the sensitivity to detecting linear changes in activation and functional connectivity in response to cognitive load, and functional connectivity changes when transitioning from a task-free resting state to a task performance (a letter n-back task with three load conditions). The results demonstrate that fNIRS is sensitive to both cognitive load and state (Herff et al., 2014). Interesting, almost identical findings are reported by another group of scientists (Fishburn et al., 2014). These findings strongly indicate the fNIRS-measured prefrontal activity to discriminate cognitive states in real life. Arguments that support this conclusion as well as suggestive evidence for the claim are presented (Derosiere et al., 2013; Harrivel et al., 2013; Yoshino et al., 2013a,b).

The most intensively investigated issue in the functional research of prefrontal cortex is that concerning working memory, particularly executive function. It is not surprising therefore that several articles deal with the issue in the current research topic. A review (Moriguchi and Hiraki, 2013) summarizes quite concisely recent advancements by fNIRS research on the development of executive function in children. It particularly concentrates on the lateral prefrontal cortex, focusing on inhibitory control and cognitive shifting. The remaining four research articles deal with divergent issues, e.g., music (Ferreri et al., 2013), story-telling (Moro et al., 2013), the Scarborough adaptation of the Tower of London (Ruocco et al., 2014), and gambling (Bembich et al., 2014). While actual issues addressed in the studies are variable, they commonly deal with encoding and decoding processed of memories stored in the working memory system for our dairy activities. This diversity of topics addresses multifaceted aspects of applying NIRS to various scientific disciplines such as pediatrics, psychology, psychiatry, neurophysiology.

#### Another research direction addresses emotional influences on prefrontal activity, the number of reports in this area has been increasing. In particular, processing emotional visual stimuli to produce these effects is drawing attention. An article (Doi et al., 2013) reviews unique characteristics of fNIRS as an effective tool for investigating the role of the prefrontal cortex in emotional processing. Admitting several obstacles in the application of fNIRS to emotion research, they discuss the implications of recent findings to assess the effects of prefrontal activation on emotion, specifically addressing the methodological challenges of NIRS measurements with respect to the area of emotion research. They discuss potentials of the two research fields for investigating (i) biological pre-dispositions influencing prefrontal responses to emotional stimuli and (ii) neural mechanisms underlying the bi-directional interaction between emotion and action, have much to gain from the use of fNIRS. The latter issue is also addressed by another group of scientists (Sato et al., 2014), who reports that the mood-cognition interaction occurs in the lateral prefrontal cortex, a brain region known to associate with the executive function of the working memory system closely (Moriguchi and Hiraki, 2013). Other two studies concentrate on recordings in the activation of medial prefrontal cortex when viewing visual images (non-art pictures or art pieces). In both studies, some modulation of the brain activation is confirmed. However, it is more robust when presenting art pieces (Kreplin and Fairclough, 2013) than non-art pictures (Ozawa et al., 2014). Among such art pieces, the activation was more robust when the presented images were evaluated as more esthetic than when they were evaluated as less esthetic. Based on the findings, the authors of the article conclude the activated brain region is involved during positive evaluation of visual art that may be related to judgment of pleasantness or attraction.

In summary, this research topic has provided a forum for scientists planning functional studies of prefrontal brain activation using fNIRS. We hope this will serve as a reference repository of knowledge from these fields as well as a conduit of information from leading researchers. In addition it offers an extensive crossreferencing system that will facilitate search and retrieval of information about NIRS measurements in activation studies. Researchers interested in fNIRS would benefit from an overview about its potential utilities for future research directions.

### References


validation of the Scarborough adaptation of the Tower of London (S-TOL) using near-infrared spectroscopy. Front. Hum. Neurosci. 8:185. doi: 10.3389/fnhum.2014.00185


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Masataka, Perlovsky and Hiraki. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.