Edited by: Holmes Finch, Ball State University, USA
Reviewed by: Fernando Marmolejo-Ramos, Univerity of Adelaide, Australia; Pietro Cipresso, Istituto di Ricovero e Cura a Carattere Scientifico Istituto Auxologico Italiano, Italy
*Correspondence: Stéphane Buffat, Département Action et Cognition en Situation Opérationnelle, Institut de Recherche Biomédicale des Armées, BP73, 91223 Brétigny-sur-Orge Cedex, France; Cognition and Action Group, Cognac G, Service de Santé des Armées, CNRS, Université Paris Descartes, UMR-MD 4 - 8257, 45 Rue des Saint Pères, 75270 Paris Cedex 06, France; CNRS, UMS RISC 3332, 29 Rue d'Ulm, 75005 Paris, France; Laboratoire des Systèmes Perceptifs, Département d'études Cognitives, UMR-8248, CNRS, ENS, 29 Rue d'Ulm, 75005 Paris, France e-mail:
This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Studying object recognition is central to fundamental and clinical research on cognitive functions but suffers from the limitations of the available sets that cannot always be modified and adapted to meet the specific goals of each study. We here present a new set of 3D scans of real objects available on-line as ASCII files, OB3D. These files are lists of dots, each defined by a triplet of spatial coordinates and their normal that allow simple and highly versatile transformations and adaptations. We performed a web-based experiment to evaluate the minimal number of dots required for the denomination and categorization of these objects, thus providing a reference threshold. We further analyze several other variables derived from this data set, such as the correlations with object complexity. This new stimulus set, which was found to activate the Lower Occipital Complex (LOC) in another study, may be of interest for studies of cognitive functions in healthy participants and patients with cognitive impairments, including visual perception, language, memory, etc.
Sets of experimental visual stimuli are bread and butter for any research investigating cognitive functions in healthy individuals and patients. Continuous improvements of numerical editing and manipulation of images, as well as their dissemination through the Internet, helped the fast development and use of classes of visual stimuli coming in a variety of formats and representing a large diversity of “objects” whether natural or artificial. One such set comes from the seminal work of Snodgrass and Corwin (
Other visual data sets are often made up from photographs transformed into different numerical formats. Efforts were made to provide well-controlled sets (Bonin et al.,
We took a different, complementary, approach and designed a new set of ~140 visual stimuli, constructed by scanning real 3D objects (either “natural” or realistic toy versions of “natural” objects) with a laser scanner (Figure
With these stimuli, simple routines permit versatile transformations that can be performed in real time (see Figure
This stimulus set has been used in fMRI imaging studies that uncovered brain regions overlapping those already found to respond to objects (e.g., in the Lateral Occipital Cortex or LOC; Kourtzi and Kanwisher,
This stimulus set must be normalized to ensure that objects are consistently recognized across observers and associated meta-data should be made available to a large community. To that aim it is necessary to collect a large amount of data with numerous participants. In this regard, a web-based protocol has the advantages of easily and quickly collecting numerous answers via the world wide web, (Birnbaum,
The aim of the present work is: (i) to advertise the stimuli stored in the OD3D available database and to present the results of the normative tests conducted with this set similar to Snodgrass and Vanderwart's (
The experiment was promoted through mailing to the RISC (“Relais d'information sur les sciences de la cognition,”
Participants ranged between 22 and 54 years old (Mean age 28.6 years, ±7.5; women/men ratio of 63:37), and were native French speakers. All reported having no neurological disease and having normal or corrected to normal vision. The experiment was done by visiting the experiment's website (
Participants gave their consent before starting the experiment and were explained that they were free to stop at any time and for any reason. If they stopped early, participants received a password to reconnect to the web site and to continue the experiment where they left it, if they wished.
In total, 430 connections were registered, corresponding to a total of 223 different participants. Two participants who gave responses unrelated with the task and 11 participants who responded too quickly, resulting in mostly empty files recorded, were excluded from the analyses. In total, we analyzed the data from 210 participants.
3D Objects from the OB3D database are free, open source available online (
On average, 105 points defined each object. Figure
With these stimuli, a large number of object transformations are possible, such as rotating objects, decreasing the number of dots, changing the size, and proportions, adding positional noise, changing color, generating scramble versions, mixing, and morphing between objects, etc. (see Figure
XML files were used for the Web-based experiment. For each object, the point of view was manually set, so as to ease the recognition of each object, and the names and categories were assessed by means of the French lexical database “Lexique 2” (New et al.,
The web-based experiment unfolded as follows: After a blank page, 100 dots randomly picked up amongst the all the dots from a cloud object were presented. The number of dots then linearly increased with time. The participants were instructed to press a key when they were confident they had recognized the object. After a key press, the number of dots presented on the screen was recorded, and the response screen was displayed. At this point, participants had to fill out a short questionnaire: (1) To give the name of the object, or to answer “does not know the object” (DKO), “does not now the name” (DKN), or “tip of the tongue” (TOT); (2) To rate the object familiarity on a 0–9 scale; (3) To indicate the category of the object (forced choice). Afterwards, participants pressed a key to start the next trial, and a new object was presented. A random stimulus sequence was generated for each participant. Because each trial takes time to perform, after each 20 trials, participants could stop the experiment. In this case, they received a link by email allowing them to resume the experience later at the trial where they stop. The instructions are presented in Table
Name | Identify the object as quickly and unambiguously as possible by writing only one name, the first name that comes to mind | Free text, or alternate checking box for DKO, DKN, and TOT answers |
Category | Indicate to which category the object belongs to | Forced choice (eight alternatives) |
Familiarity | Rate the level of familiarity with the object | Rating scale 0–9 |
Retrospective confidence judgment | Rate your confidence to the naming question | Rating scale 0–9 |
Number of dots | Each object will be displayed with an increasing number of dots on the screen. Press a key as soon as you believed you recognized the object | Starts with 100 dots randomly picked up amongst all the dots. Linear increase of dot number until a Key is pressed |
Free comment | Add any comment about the experiment or the object | Free text |
The web-based architecture was as follows:
The main functionalities of the experiment were written in HTML5 and JavaScript. The website was tested with all major web browsers and optimized desktop computers. Prior to the experiment proper, participants could test whether their browser supported WebGL (a JavaScript API for the rendering of interactive 3D graphics) and explained how to enable WebGL. All data were stored in a relational database, MySQL (number of dots needed to recognize the object, and the questionnaire answers). The database architecture is presented in Figure The experiment follows most of the recommendations of Birnbaum (
Ratings and correct answers
Whenever the minimum number of displayed dots was 150 or less, the corresponding trial was excluded as it most likely corresponded to manipulation errors (no answer whatsoever for the questions). Only three such trials were discarded from further analyses, which accounts for a high rate of participants correctly performing the task.
The analysis mostly follows the same logic as in Brodeur et al. (
In this equation,
We also computed correlations (linear regression analysis) between correct answers and perceived familiarity, perceived difficulty, and the mean number of dots required for object recognition.
Free comments were coded, and the most frequent were analyzed. Most analyses are Chi2 We also computed
In total, we collected 7200 answers, from 210 participants. The norms are summarized in Figure
The number of displayed dots allowing a correct naming provides a relevant quantitative indication of the minimum information needed to recognize and classify an object. The mean number of dots yielding correct recognition was 4560 (±6222), but varies widely, both across objects and observers, indicating varying degrees of ambiguity and uncertainty (see Supplementary Table
Mean Name agreement was 62% (±48%). This result is very close to the results of Brodeur et al. (
The mean for Category agreement was 70.2% (±4.58%). The categories with the highest mean for Category agreement were tools [84.3% (±3.64%)] and animals [82.6% (±3.80%)]. The categories with the lowest mean for Category agreement were furniture [50.0%, (±5.09%)] and others [21.8%, (±4.13%)]. With a mean
Mean DKN was 9.1% (±2.8%), mean DKO was 1.2% (±1.1%), and mean TOT was 0.4% (±0.6%). The sum of DKN and TOT is relatively high, which is consistent with the trade-off between having a large number of objects and easily named list of objects. This result is also in agreement with the Name agreement found in this experiment.
Contrary to previous studies, we did not offer more than eight categories. However, the number of objects in each category was almost the same, which gave our participants a balanced set of objects.
The familiarity average ratings ranged over a scale from 0 to 9 (9 being very familiar). Mean Familiarity rate was 5.45 (±3.18), meaning participants were moderately or highly familiar with the objects. This is confirmed by performing a pairwise Welsch
The Retrospective Confidence Judgment (RCJ) ratings ranged over a scale from 0 to 9 (9 being very Confident with participant's own response). Mean RCJ rate was 5.71 (±3.13). Overall, it indicates that participants were somewhat confident in their answers as confirmed by a pairwise Welsch
Correlations in normalization studies help understanding how different dimensions relate to each other. Table
Name agreement | − | 0.888 | <0.0001 | |
Number of points | + | 0.341 | <0.0001 | |
Familiarity | − | 0.354 | <0.0001 | |
Retrospective confidence judgment | − | 0.523 | <0.0001 | |
Name agreement | Familiarity | + | 0.465 | <0.0001 |
Name agreement | Retrospective confidence judgment | + | 0.625 | <0.0001 |
Name agreement | Number of points | − | 0.4 | <0.0001 |
Category agreement | Familiarity | + | 0.298 | <0.0001 |
Category agreement | Retrospective confidence judgment | + | 0.41 | <0.0001 |
Category agreement | Number of points | − | 0.255 | <0.0001 |
Most previous studies have shown that modal name agreement and the
The correlation between
RCJ is positively correlated with name agreement: this relationship between accuracy and RCJ is consistent with the consensuality principle (Koriat,
The participants made 712 free comments, over a total of 7434 answers (9.7%). Overall, this is a good indication that the task was performed without any major issue for the participants. These comments were broke down by means of coding (see Table
No comment | None | The space was left blank | |
Explicitly no comment | None | The participant wrote an indication that she/he had no additional comment | “No” |
Personal comment | Internal | Some piece of humor, something personal | “I am a veterinary” |
Technical issue | External | There was a technical issue that could have impaired the experiment | “There was a brief black screen” |
Perspective/viewpoint | External | The way the object was depicted in 3D seemed unusual | “The viewpoint is weird for this object” |
Task difficulty | External | The task was thought to be too difficult | “This is too hard” |
Alternate name | Internal | The participant proposed another name | Answer is “Plane,” comment is “Dart” |
Sentence | Internal | The participant made a whole sentence to describe the object | “It would have been a mango if smoother” |
Confidence | Internal | The participant expressed a certain level of confidence | “I am unsure” |
Justification | Internal | The participant justified her/his choice | “The shoe has high heels” |
Change with time | Internal | The participant stated that she/he would have initially answered a name, and then changed the Name at some point during the trial | “I first took if for a tree. I was right to wait for it was a vehicle” |
The number of each coded comment is displayed in Figure
We performed additional analyses for the three most common comments (more than 20), “Alternate name,” “Sentence,” and “Justification.”
When the participants made a comment coded “Alternate name,” they were more often wrong when naming the object (Corrected Chi2 = 0.044; Corrected
We performed a Chi2 analysis between the answers coded “Sentence” and those coded otherwise, regarding the TOT variable. There is a significant difference between the two cases, in favor of the participants expressing whole sentences to try to explain the object they saw, but being unable to name it correctly. Because there are few TOT, we used Fischer's exact probability (Corrected Chi2 = 16.63; Fischer's Exact Prob. = 0.0031).
When considering the answers with “Justification,” we found that they were not different than without in terms of name agreement (Corrected Chi2 = 0.045; Corrected
We also pooled the comments in 3 Loci, “None,” “Internal,” and “External.” This gives some insight in the locus of control of the participants that made free comments. We found that 67% of the comments can be attributed to the internal locus, the remaining 33% being related to the external locus.
The present work describes a web-based experiment aimed at the normalization of a novel visual stimuli data set. This experiment was done in order to illustrate both the stimuli properties and on how valuable a web-based can be regarding database normalization. The OB3D is a free database of 3D objects that can be used by themselves, or embedded in virtual reality (VR) settings, with a comprehensive normalization. This data set is the first of its kind because one can easily customize the stimuli to fit with the experimental paradigm chosen by the researcher or clinician, and still be controlled for low-level vision cues.
The normalization results include RCJ in addition to the more widely reported parameters. First, we found normalization data consistent with the literature. Second, we provided additional value by providing a threshold in terms of the numbers of dots required reaching certain recognition rates. We believe that controllability of a stimulus is of paramount importance for neuropsychology tests. Other issues may arise, such as the necessity of control responses in a reference population (Rowe and Craske,
VR and interactive video gaming (Bioulac et al.,
Both approaches seem to be advantageous because they provide an opportunity to practice activities that are otherwise difficult to do in a clinical environment (e.g., at home), although it can still be administered in traditional therapeutic settings. In the latter case, the main advantages are better control and cost effectiveness. It can provide stimuli for individuals who have difficulty in imaging scenes. It can provide opportunities for those individuals who are too phobic to experience real situations, and it can also generate stimuli of greater magnitude than other more standard techniques such as whole alternative or even fantastic worlds (Riva,
Furthermore, VR programs benefit from being more interesting and even sometimes enjoyable than traditional therapy tasks. One of the immediate consequences is the higher numbers of repetitions the patients are willing to make. What makes these new tools interesting is their versatility. So far, they have been used in situations as diverse as stroke rehabilitation (Laver et al.,
Clinical psychologists work with all age groups from very young children to older people. In doing so, they work with people with mild, moderate, and severe mental health problems. They also help people suffering from learning disabilities, people with physical and sensory handicaps, brain injury, and even people who have alcohol and other drug problems. In addition, they can treat a wide range of physical health problems. The diversity of these clinical situations benefit from the use of virtual environments. Indeed, there are examples of the use of VR in the field of neuropsychology rehabilitation, in older adult psychology services, and in pediatric services. Their use within learning disabilities services in UK has also been discussed (Serino and Riva,
VR is at the same time technology, communication interface, and compelling experience. Because of population aging, and global economy uncertainty, free tools, such as tests, software (e.g., NeuroVR 2, Riva et al.,
We performed a web-based experiment aiming at normalizing a novel visual stimulus database made of 3D scans of “natural” objects. This kind of stimuli allows a controlled parametric tuning of several stimulus characteristics, as well as a large number of versatile transformations. In addition to classical normalization parameters, including RCJ, we measured a dot threshold estimating the information content needed for recognition and categorization. Overall, the present results are consistent with those reported in the literature with another kind of visual stimuli, indicating this stimulus set is well suited for use in a variety of experiments, with healthy subjects or patients.
In addition to the usual normalization data available with other image sets, the possibility to measure a recognition threshold, in terms of the numbers of dots, offers a quantitative evaluation of recognition performance, a feature rarely available with other stimulus sets.
To conclude, in this paper, we have shown that a web-based experiment is well suited to normalize a database aimed at providing visual stimuli (natural objects) for the research community. In addition, such normalization is especially important for clinical research, because the patients can have limited abilities to recognize some objects or some categories.
Conceived and designed the experiment: Stéphane Buffat, Jean Lorenceau. Scanned the objects: Frédéric Benmussa. Performed the experiment: Delphine Rider. Analyzed the data: Stéphane Buffat, Véronique Chastres. Contributed reagents/material/analysis tools: Alain Bichot, Véronique Chastres. Wrote the paper: Stéphane Buffat, Jean Lorenceau. Funded the project: Jean Lorenceau.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
We thank A. L. Paradis for her insightful suggestions.
The Supplementary Material for this article can be found online at: