João R. Sato
Thomas P. White
Claudinei E. Biazoli Jr.- 1Centre of Mathematics, Computation and Cognition, Universidade Federal do ABC, Santo Andre, Brazil
- 2School of Psychology, University of Birmingham, Birmingham, UK
A commentary on
A test-retest dataset for assessing long-term reliability of brain morphology and resting-state brain activity
by Huang, L., Huang, T., Zhen, Z., and Liu, J. (2016). Sci. Data 3:160016. doi: 10.1038/sdata.2016.16
A transformation toward open neuroscience is ongoing (Milham, 2012), and increases the availability of high-quality, open-access neuroimaging datasets (Poline et al., 2012; Mennes et al., 2013). Consequently, a new set of analytical approaches, including discovery science (Biswal et al., 2010) and focus on individual rather than group-level effects (Finn et al., 2015; Miranda-Dominguez et al., 2014), are increasingly accessible. However, moving to single-subject statistics raises specific concerns that must be addressed. Notable amongst these is the test-retest reliability of fMRI-based metrics (Dubois and Adolphs, 2016). Recently, Huang et al. (2016) provided a test-retest neuroimaging dataset (BNU2), with an inter-scan interval in the order of months, allowing investigation of the temporal reliability of features extracted from rs-fMRI. In the spirit of the data-sharing initiatives, the “Consortium for Reliability and Reproducibility in Functional Connectomics” (CoRR) publicly released this data (Zuo et al., 2014).
Recently, Finn et al. (2015) investigated the existence of functional “connectome fingerprints.” The authors hypothesized that, despite overall similarity in connectivity patterns across subjects, portions of brain connectome variability would be fairly singular to each individual (Mueller et al., 2013; Gordon et al., 2015; Laumann et al., 2015; Xu et al., 2016). This notable study demonstrated that, using only the functional connectivity profile extracted from an fMRI scanning session, it was possible to identify the same subjects from their profiles from a second session a few days later. Interestingly, this hypothesis of a functional connectivity fingerprint was also supported by previous work (Miranda-Dominguez et al., 2014), which additionally showed that such individual signatures exist not only in humans but also in non-human primates.
However, the extent to which connectome profile stability can be generalized to more extended timescales remains largely untested. Furthermore, the vast majority of the functional connectome studies to date focus on timescales of seconds to minutes or years to decades (Poldrack et al., 2015; Huang et al., 2016, but see Xu et al., 2016). We thought that the BNU2 dataset is quite suitable to assess the reliability and stability of connectome fingerprints on an intermediate timescale of months. In fact, the released BNU2 dataset consists of anatomical and functional data from 61 healthy adults (19–23 years old) scanned under a resting-state protocol (eyes closed) in two sessions at an interval of 103–189 days. Further information about scanning parameters, demographical and quality metrics data can be found in Huang et al. (2016).
We preprocessed the data and extracted individual functional connectivity estimates using CONN toolbox version 15.g (Whitfield-Gabrieli and Nieto-Castanon, 2012) with standard MNI152 pipeline and parameters. Conservative options (discarding volumes with displacement >0.5 mm and global-signal z-value >3) for scan motion censoring were applied, since motion artifacts are a well-recognized source of error in functional connectivity studies using fMRI. The pairwise bivariate correlations (functional connectivity) among 333 cortical regions-of-interest (ROIs) were obtained using the Gordon et al. (2016) parcellation. Considering the upper triangular values of the individual correlation matrices as the subject connectivity profile, a functional connectome fingerprinting analyses was then carried out. The similarity between the two profiles was then measured with Spearman's correlation coefficient. The within-subject correlation between the two sessions determines the accuracy as it reflects the proportion of subjects correctly identified. Note that the expected accuracy by chance is 1/61 = 1.6%.
We expected to reproduce the original results from connectome fingerprint studies (Miranda-Dominguez et al., 2014; Finn et al., 2015) if the individual profiles are stable over months. In order to do so, we attempted to identify the subjects in the second session based on the profiles similarity to the first session. As a second step, we calculated the intra- and inter-subject similarities between the two sessions for each subject. The inter-subject similarity was calculated by random sampling an individual at the second session. We also sought to investigate how large-scale networks connectivity varies within and between subjects in the timescale of months. Each brain parcel was labeled for the conventional resting-state networks, as provided by the Gordon atlas. Thus, we conducted the two previously described analyses' steps considering all ROIs and each network separately. Based on previous findings of within and between subject variability of network connectivity (Mueller et al., 2013; Miranda-Dominguez et al., 2014; Zuo and Xing, 2014; Chen et al., 2015; Finn et al., 2015; Poldrack et al., 2015), we expected increased discriminability of individuals for heteromodal associative networks.
The results are shown in Figure 1. A high accuracy of 85% for the whole-brain connectivity profile was found. Moreover, accuracies were above 90% for the default mode and the fronto-parietal networks. Interestingly, accuracies for primary sensory and motor networks were lower. It is likely that the ability to uniquely discriminate individuals relies on features with both low within and high between-subject variability over time. For all the networks investigated, we noticed a tendency for higher similarity within subjects than between them. Overall, these results suggest stable connectome fingerprints exist over months and are in agreement with the previously reported inter-individual variability of networks including heteromodal areas. However, caution should be taken when interpreting differences in accuracy between networks as the number and extent of ROIs varies. Since each ROI signal is based on average across voxels, networks with larger parcels may have superior signal-to-noise ratios. Moreover, the number of ROIs may be related to redundancy of information in the connectivity matrices, which would also affect accuracies. Remarkably, the networks which presented the lowest subjects identification accuracies have <8 ROIs.
Figure 1. Functional networks—Box-plots of intra (red) an inter-subject (green) correlations between the first and second resting state fMRI sessions. The analyses are carried out separated by networks and the connectome fingerprinting accuracy is highlighted as percentage. Cing. Oper., Cingulo-opercular; Cing. Par., Cingulo-parietal; DMN, Default-mode network; Dorsal Attn, Dorsal attention; Fronto-par, Fronto-parietal; Retr. Temp., Retrosplenial-temporal; SMh, somatomotor-hand; SMm, somatomotor-mouth; Ventral Attn, Ventral attention.
Results from individual-based fMRI metrics can be framed by the usual concepts of validity and reliability (Dubois and Adolphs, 2016). However, an inherent issue of the approaches like those proposed here is the extent to which validity and reliability can be disentangled. In other words, it is possible to state that connectome fingerprints are stable over the months, which would constitute a claim for the validity of the underlying neural phenomena. Alternatively, but not mutually exclusively, it is also possible that test-retest reliability varies between subjects and networks. We argue that continuous effort for data-sharing, in the spirit of the CoRR and other initiatives, is of paramount importance as disentangling these factors will ultimately depend on accumulating evidence for the stability of connectome fingerprints across different timescales and with large datasets. Establishing the stability of these measures, in turn, will be essential to investigate true effects of development on the connectomes. Furthermore, adopting comparable acquisition parameters and open and reliable data processing will be necessary to further assure the validity of remarkable findings such as individually unique connectivity profiles.
Author Contributions
JS preprocessed and analyzed the data. All authors wrote the manuscript.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
The authors are grateful to Huang et al. (2016) and the “Consortium for Reliability and Reproducibility in Functional Connectomics” (CoRR) for providing the data. The authors also acknowledge and are thankful to the two reviewers who provided valuable comments to improve this letter.
References
Biswal, B. B., Mennes, M., Zuo, X. N., Gohel, S., Kelly, C., Smith, S. M., et al. (2010). Toward discovery science of human brain function. Proc. Natl. Acad. Sci. U.S.A. 107, 4734–4739. doi: 10.1073/pnas.0911855107
Chen, B., Xu, T., Zhou, C., Wang, L., Yang, N., Wang, Z., et al. (2015). Individual variability and test-retest reliability revealed by ten repeated resting-state brain scans over one month. PLoS ONE 10:e0144963. doi: 10.1371/journal.pone.0144963
Dubois, J., and Adolphs, R. (2016). Building a science of individual differences from fMRI. Trends Cogn. Sci. 20, 425–443. doi: 10.1016/j.tics.2016.03.014
Finn, E. S., Shen, X., Scheinost, D., Rosenberg, M. D., Huang, J., Chun, M. M., et al. (2015). Functional connectome fingerprinting: identifying individuals using patterns of brain connectivity. Nat. Neurosci. 18, 1664–1671. doi: 10.1038/nn.4135
Gordon, E. M., Laumann, T. O., Adeyemo, B., Huckins, J. F., Kelley, W. M., and Petersen, S. E. (2016). Generation and evaluation of a cortical area parcellation from resting-state correlations. Cereb. Cortex 26, 288–303. doi: 10.1093/cercor/bhu239
Gordon, E. M., Laumann, T. O., Adeyemo, B., and Petersen, S. E. (2015). Individual variability of the system-level organization of the human brain. Cereb. Cortex doi: 10.1093/cercor/bhv239. [Epub ahead of print].
Huang, L., Huang, T., Zhen, Z., and Liu, J. (2016). A test-retest dataset for assessing long-term reliability of brain morphology and resting-state brain activity. Sci. Data 3:160016. doi: 10.1038/sdata.2016.16
Laumann, T. O., Gordon, E. M., Adeyemo, B., Snyder, A. Z., Joo, S. J., Chen, M. Y., et al. (2015). Functional system and areal organization of a highly sampled individual human brain. Neuron 87, 657–670. doi: 10.1016/j.neuron.2015.06.037
Mennes, M., Biswal, B. B., Castellanos, F. X., and Milham, M. P. (2013). Making data sharing work: the FCP/INDI experience. Neuroimage 82, 683–691. doi: 10.1016/j.neuroimage.2012.10.064
Milham, M. P. (2012). Open neuroscience solutions for the connectome-wide association era. Neuron 73, 214–218. doi: 10.1016/j.neuron.2011.11.004
Miranda-Dominguez, O., Mills, B. D., Carpenter, S. D., Grant, K. A., Kroenke, C. D., Nigg, J. T., et al. (2014). Connectotyping: model based fingerprinting of the functional connectome. PLoS ONE 9:e111048. doi: 10.1371/journal.pone.0111048
Mueller, S., Wang, D., Fox, M. D., Yeo, B. T., Sepulcre, J., Sabuncu, M. R., et al. (2013). Individual variability in functional connectivity architecture of the human brain. Neuron 77, 586–595. doi: 10.1016/j.neuron.2012.12.028
Poldrack, R. A., Laumann, T. O., Koyejo, O., Gregory, B., Hover, A., Chen, M. Y., et al. (2015). Long-term neural and physiological phenotyping of a single human. Nat. Commun. 6:8885. doi: 10.1038/ncomms9885
Poline, J. B., Breeze, J. L., Ghosh, S., Gorgolewski, K., Halchenko, Y. O., Hanke, M., et al. (2012). Data sharing in neuroimaging research. Front. Neuroinform. 6:9. doi: 10.3389/fninf.2012.00009
Whitfield-Gabrieli, S., and Nieto-Castanon, A. (2012). Conn: a functional connectivity toolbox for correlated and anticorrelated brain networks. Brain Connect. 2, 125–141. doi: 10.1089/brain.2012.0073
Xu, T., Opitz, A., Craddock, R. C., Wright, M. J., Zuo, X. N., and Milham, M. P. (2016). Assessing variations in areal organization for the intrinsic brain: from fingerprints to reliability. Cereb. Cortex 26, 4192-4211. doi: 10.1093/cercor/bhw241
Zuo, X.-N., Anderson, J. S., Bellec, P., Birn, R. M., Biswal, B. B., Blautzik, J., et al. (2014). An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1:140049. doi: 10.1038/sdata.2014.49
Keywords: replicability, brain networks, brain imaging, reproducibility, fingerprints
Citation: Sato JR, White TP and Biazoli CE Jr. (2017) Commentary: A test-retest dataset for assessing long-term reliability of brain morphology and resting-state brain activity. Front. Neurosci. 11:85. doi: 10.3389/fnins.2017.00085
Received: 13 December 2016; Accepted: 07 February 2017;
Published: 22 February 2017.
Edited by:
Xi-Nian Zuo, Institute of Psychology (CAS), ChinaCopyright © 2017 Sato, White and Biazoli. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: João R. Sato, joao.sato@ufabc.edu.br