*Correspondence:
This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology.
Edited by: Lynne D. Roberts, Curtin University, Australia
Reviewed by: Eric-Jan Wagenmakers, University of Amsterdam, Netherlands
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
The need for improved methodology for psychological research has recently received much attention. The primary recommendation has been increased emphasis on confirmatory or replication research that is carefully planned with adequate sample size and is pre-registered (Wagenmakers et al.,
In the fall of 2012, we opened a study registry at the University of Edinburgh's Koestler Parapsychology Unit (KPU) (KPU Registry,
Based on experience working in regulated medical research, the second author has long advocated that the standard research methods for academic psychology were not adequate for controversial research like parapsychology and that formal, pre-registered, well-powered confirmatory research was needed (Kennedy,
Distinguishing between exploratory and confirmatory research is important for study registration. Registration has high value for confirmatory research, but less value for exploratory research. Exploratory research is typically the creative step that is the starting point for a line of research, whereas confirmatory research provides the convincing evidence that makes science valid and self-correcting. This distinction is usually straightforward for regulated medical research—Phase 1 and Phase 2 studies are exploratory, and Phase 3 studies are confirmatory (National Library of Medicine,
We recommend that each registered hypothesis or analysis be classified as exploratory or confirmatory. For a study pursuing only exploratory analyses, registration in the KPU registry is considered optional. The strongest evidence pertaining to an effect comes from registered confirmatory research. Meta-analysis of exploratory research does not eliminate the need for well-designed confirmatory research (Cooper and Hedges,
Confirmatory research has two key characteristics. First, confirmatory research can provide evidence that the hypothesis of interest is false as well as true. This implies that the study has adequate sample size and that the measurement methods and experimental interventions are established. Studies are exploratory if they involve the development of measurement methods or new experimental interventions that could confound the interpretation of evidence for or against the primary hypothesis. Also, non-significant results for an underpowered study are ambiguous because the results could be due to low power rather than to the experimental hypothesis being false.
Second, all analysis decisions that could affect the confirmatory results are made prior to the start of data collection. These decisions include the specific statistical methods, the criteria for acceptable evidence, any transformations or adjustments to the data, and any criteria for excluding or deleting data.
We recommend that the analysis decisions noted above be included in the registration information for confirmatory hypotheses. This amount of detail about the planned analysis is greater than typically required to register a clinical trial, but is less extensive than the statistical analysis plan that regulatory agencies expect for confirmatory studies (International Conference on Harmonisation,
We recommend that the submitted registration information is reviewed to verify that the required information has been provided. Reviews of the KPU registrations found deficiencies for virtually all initial submissions. Common omissions included not specifying whether analyses were one or two sided and not specifying the prior probability distributions for Bayesian analyses.
Ambiguities and inconsistencies about the independent and dependent variables also occurred. In one case the planned hypothesis test was presented as confirmatory and the type of statistical test was pre-specified as an ANOVA. However, the scores from a questionnaire were the independent variable for the ANOVA, and the experimenter did not specify the criteria for mapping the scores to discrete categories for the analysis. For a confirmatory analysis, the criteria for assigning the categories needed to be pre-specified in order to document that the experimenter did not explore different criteria during data analysis and select the criteria that produced the most favorable results. This specification was requested as part of the registration review, and the ambiguity was eliminated.
We now believe that impartial, detailed review of the completeness and consistency of registration information is essential. We have found that authors of registry submissions welcome the review process and recognize that it strengthens study registration and enhances the credibility of the study. The primary clinical trials registry, ClinicalTrials.gov (
Registries maintain information about studies indefinitely and are pivotal for literature reviews. For medical research, study registries are often the starting point for reviews. Widely used and easily searched registries allow reviewers to find efficiently the strongest evidence and unpublished studies. In addition, registries increasingly provide links or abstracts for the study results. The KPU registry encourages experimenters to provide links or information about the experimental results that can be subsequently posted with the registration information.
In addition to preventing common research biases, public study registration promotes scientific efficiency and reduces unintended duplication of research effort (International Committee of Medical Journal Editors,
Basic study registration publicly documents the key planned methodology for a study, but does not evaluate whether the methodology is adequate. For example, the registration process does not consider whether other statistical methods would be preferable, whether the planned experimental procedures preclude alternative explanations, or whether the planned hypotheses are meaningful. These types of questions are most effectively handled with peer review prior to registration of the final study plan. Peer review of the planned methodology can be obtained by privately circulating a description of the study among colleagues or by posting the description on the internet and inviting comments prior to formal registration on a public registry. Journals that will accept a study based on peer review of the planned methodology enhance study quality and are increasingly available (Chambers,
Study registration also does not prevent fraud by an experimenter. Other methodological practices are needed to prevent fraud and are appropriate for confirmatory research (Stroebe et al.,
The KPU registry and the major medical registries have substantially simpler, faster registration processes and more flexible publication options than the registered replication reports for specific journals, but also have substantially greater structure than the self-registration process of Open Science Framework (OSF). OSF provides online processes for managing scientific documents, data, and collaboration, and includes an option for registration (OSF,
Registration at OSF consists of making a copy of the electronic study documents and assigning a date-time stamp to the copy. A registration copy cannot be changed or deleted, but the experimenter controls the content of a registration, how many registrations are made, and whether a registration is kept private or made public. This process allows an experimenter to examine the study results before deciding whether to make the study and/or registration public (OSF,
For comparison, the standards for clinical trials and for the KPU registry are that registrations are controlled by an independent organization that has certain minimum registration requirements and makes the registrations irreversibly public before data collection starts. This eliminates the options to keep or make registrations private if the results are unfavorable. If the experimenter can examine the study results before deciding whether to make a registration public, experiments with favorable outcomes can be presented as pre-registered, but experiments with unfavorable outcomes may be kept privately in the file drawer as the experimenter moves on to other higher priorities. This substantially compromises the value of study registration. A relatively simple registration process with greater structure and improved search capabilities could be implemented within OSF or could be a feature of new registries.
Our recommendations for study registration are concisely listed below. This list identifies key factors for registration and may be useful for those planning to register a study or managing a study registry. The recommendations are:
public registration before data collection has begun;
registrations cannot be removed or made private after data collection has started;
each hypothesis or analysis classified as exploratory or confirmatory;
methodology for confirmatory research specified in sufficient detail to document that all decisions that affect the outcome were made prior to any knowledge of the study data;
registration information independently reviewed for completeness and clarity;
history of changes publicly displayed for any revisions to the registration information after data collection has begun;
registration information openly and freely available to anyone (no website login or membership required);
registrations easily and reliably searched to find all registered studies on a particular topic or by a particular researcher—for literature reviews and future verification of original study plans;
formal or informal peer review of the planned study prior to registration;
abstracts or links provided for final study results.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.