Edited by: Gert Pfurtscheller, Graz University of Technology, Austria
Reviewed by: Christoph Guger, Guger Technologies OEG, g.tec Medical Engineering GmbH, Austria; Monica Risetti, Fondazione Santa Lucia, Italy
*Correspondence: Andrea Kübler, Department of Psychology I, Marcusstraße 9-11, 97070 Würzburg, Germany. e-mail:
This article was submitted to Frontiers in Neuroprosthetics, a specialty of Frontiers in Neuroscience.
This is an open-access article subject to an exclusive license agreement between the authors and the Frontiers Research Foundation, which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are credited.
Brain–computer interfaces (BCIs) enable paralyzed patients to communicate; however, up to date, no creative expression was possible. The current study investigated the accuracy and user-friendliness of P300-Brain Painting, a new BCI application developed to paint pictures using brain activity only. Two different versions of the P300-Brain Painting application were tested: A colored matrix tested by a group of ALS-patients (
Brain–computer interfaces (BCIs) are useful and accurate tools to provide communication for paralyzed people without the need of any voluntary muscular control (Kübler and Neumann,
One possible brain-signal for BCI-control is the P300 event-related potential. The P300 is a positive deflection in the EEG occurring approximately 300 ms after the presentation of a rare stimulus (oddball) in between frequently presented standard stimuli (oddball paradigm, Sutton et al.,
Today, additionally to the visual P300-BCI, also auditory and tactile P300-BCIs have been developed and their feasibility was demonstrated (Furdea et al.,
A pilot study addressing the usability of the P300-Brain Painting application was conducted and showed positive results (Kübler et al.,
The current study investigates accuracy and user-friendliness of the P300-Brain Painting application in healthy subjects and ALS-patients. We hypothesized that the P300-Brain Painting application would be usable with the same accuracy as the highly functional and precise P300-Spelling application, which is known to be usable for ALS-patients with accuracies above 70% over a time period of 40 weeks (Nijboer et al.,
The study consisted of two different experiments: In experiment I (in the further text referred to as Exp I) the P300-Spelling application and the P300-Brain Painting application were compared regarding accuracy and P300 amplitudes in healthy subjects and ALS-patients. On the basis of the results of Exp I, in experiment II (Exp II), a new P300-Brain Painting application matrix was developed and accuracy and P300 amplitude were compared with the P300-Spelling application. It was designed in black and white to avoid differences with regards to stimulus intensity and complexity and the cells of the matrix were organized in a more self-explanatory manner.
Exp I included 3 ALS-patients (two female, age 53 and 42 years; one male, age 54 years, see Table
Patient A | Patient B | Patient C | |
---|---|---|---|
Age | 53 | 42 | 54 |
Form of ALS | Familial | Sporadic | Sporadic |
Course of ALS | Spinal | Spinal | Spinal |
Artificial ventilation | Yes | No | No |
Artificial nutrition (PEG) | Yes | No | No |
Wheelchair | Yes | Yes | Yes |
Year of diagnosis | 2002 | 1996 | 2005 |
Participation BCI-studies since | End of 2004 | Beginning of 2005 | 2008 |
Residual muscular control | Mimic, minimal movement of the right hand, minimal head movement | Mimic, chewing, eye movement | Movement of the upper part of the body, minimal left hand movement, speech |
In Exp II, a group of 10 healthy volunteers (six female, age
EEG was measured using a 16-electrode cap (tin, 8 mm, Electro-cap International, Inc.). The electrodes F3, Fz, F4, T7, C3, Cz, C4, T8, Cp3, Cp4, P3, Pz, P4, Po7, Po8, and Oz were arranged according to the 10–20 international electrode system (Jasper,
Individual parameters for each subject necessary to control the device were calculated using the P300-GUI (part of the BCI2000 software package), an application developed in Matlab (The MathWorks™, MA, USA).
Stepwise linear discriminant analysis (SWLDA) was used for signal classification. Qualified predictor variables were selected and included in a multiple regression model. No initial model terms were provided at the beginning of the calculation. In multiple steps, the currently most significant predictor variable (with
The scalp distribution of the P300 is known to be centro-parietal with highest amplitudes over midline scalp sites (Sutton et al.,
The interest of the current study was focused on accuracy and P300 amplitudes as dependent variables. Type of application (P300-Spelling application vs. P300-Brain Painting application) was manipulated as independent variable. The influence of the application on accuracy and P300 amplitudes was tested using a within-subject design.
Due to their restricted mobility, ALS-patients were presented with Brain Painting in their home environment while healthy volunteers were tested in the laboratory of the Institute of Medical Psychology and Behavioural Neurobiology of the University Tübingen.
During the measurement, all participants were sitting in a comfortable chair or wheelchair at a distance of approximately 1 m from the screens. One screen displayed the matrix while another screen showed the developing picture (the “canvas” in the words of painting). After a screening necessary to adjust the BCI to the individual user, participants had to fulfill three different tasks: spelling a pre-set sentence (copy-spelling, Kübler et al.,
During the screening, subjects had to spell the words BRAIN and POWER using the copy-spelling (Kübler et al.,
The copy-spelling task consisted of spelling the sentence “(UWE,27.BüRO_LINKS!)” (engl: “(UWE,27.OFFICE_LEFT!)” where UWE is a male first name). This sentence was printed on a DIN A4 sheet of paper and placed in front of the screens instead of using the copy-spelling function included in the P300-Spelling application (the word to spell usually appears in the top line of a computer screen) to create equal test conditions as with the P300-Brain Painting application which does not have the copy function. This particular sentence was chosen to cover one letter in each row and column. To make it easier for the subjects to maintain their level of concentration, the sentence was split into three runs allowing for short breaks after the fifth and 13th selection. In case of an error, subjects were instructed not to correct it but to proceed with the next selection. In the similar copy-painting condition, subjects had to paint a pre-set picture matched to the pre-set sentence in number of choices (
For the patients, the number of flashes during screening was the same as for the healthy subjects and was individually adapted for the copy-spelling, copy-painting, and free-painting tasks. Patients A and C were presented with 15 flashes per row and column for all tasks resulting in a duration of 46.88 s per selection for the copy-spelling task and 51.88 s for the copy-painting task. Patient B was presented with five flashes per row and column resulting in a total duration of 20.63 s per selection during copy-spelling and 25.63 s during copy-painting.
After the screening, the copy-spelling and the copy-painting task, the subjects could paint an individual picture. There were no time constraints; subjects could indicate when they wanted to stop. The duration of the free-painting period was noted.
Before each session the matrices and task were explained to the participants who were asked to focus their attention on the item to be selected by counting the number of times the item was flashing.
In Exp I, a 6 × 8 matrix was used for the P300-Spelling application and the P300-Brain Painting application. For the P300-Spelling application, the letters of the German alphabet, the numerals 0–9 and some additional punctuation marks were displayed in white on a black background (see Figure
In Exp II, the P300 Spelling matrix was identical to Exp I (see Figure
The possibility of “memory usage” related differences in presentation timing between the icons of the P300-Brain Painting application and the letters of the P300-Spelling application could be ruled out because the BCI2000 software loaded all icons into a bitmap buffer where they were re-sampled to the resolution and color depth of the display used in the experiment. Therefore, any icon used the same amount of memory in the experiment and there was no size difference between grayscale, black and white or colored icons.
The group of ALS-patients was too small for statistical analysis; therefore individual data will be reported descriptively. For healthy subjects, all data was tested for Gaussian distribution using the Kolmogorov–Smirnov test. In case of Gaussian distribution, a paired-sample
All EEG-data was analyzed using BrainVisionAnalyzer version 2 (BrainProducts GmbH, Munich, Germany) and filtered with a low cutoff of 0.1 Hz and a high cutoff of 20 Hz. Data segments were extracted reaching from the start markers of each “flashing” period until 500 ms after the end marker of each flashing period. An interval of 100 ms before the start marker was used for baseline correction. The average ERPs were calculated for targets and non-targets separately, and a grand average was calculated across all participants separately for each of the two conditions. Peak detection was executed for the window of 200–500 ms after stimulus. Peak amplitude values per subject were exported to SPSS (SPSS Inc., 2008).
To provide a measurement of accuracy under conditions matching the “use in daily life” as close as possible (in which an environment free of artifact producing sources is unlikely), the EEG-data was not corrected for eye-blinks and artifacts.
The information transfer rate was calculated to measure the speed of command selection. Most publications employ a method suggested by Wolpaw et al. (
with
A more specific method accounting for different probabilities of undesired selection per cell is the formula for mutual information (Nykopp,
with
with
However, this method estimates the probability of the undesired selection per cell which requires a large number of data points for every selection. In the current study, this high number of data points was not available for a single subject because every selection has to be made only once or twice. Therefore, to give a complete overview of the information transfer rates and a comparison between group means, the current work reports the information transfer rates calculated with both formulas. Considering the strengths and weaknesses of both methods, the formula of Wolpaw et al. (
Prior to each part of the measurement (copy-spelling, copy-painting, free-painting) all participants completed questionnaires to collect psychological data concerning mood and motivation. A German version of the Questionnaire on Current Motivation for BCI2000 (Fragebogen zur Erfassung aktueller Motivation für BCI2000, FAM-BCI2000) (Rheinberg et al.,
At the beginning of the experiment, the ALS-patients completed a questionnaire to assess quality of life in ALS-patients (schedule for the evaluation of individualized quality of life, SEIQoL) (O'Boyle et al.,
The healthy controls completed an evaluation form at the end of the measurement, which is described in the next paragraph.
After the EEG measurement, each healthy participant completed a custom-made evaluation form gathering information about user satisfaction with the P300-Brain Painting application, the easiness of use comparing P300-Spelling and P300-Brain Painting application and subjective details about the level of concentration in each individual subject. The form consisted of nine questions. Answers were provided on a five-point Likert scale ranging from 1 (easy) to 5 (difficult) for questions one and two and 1 (not appropriate at all) to 5 (completely appropriate) for questions three to nine. Additionally, the questionnaire included a request for helpful suggestions. To compare subjective differences in easiness of use between the two applications, averaged ratings were compared.
The intention of this evaluation form was to gather information concerning possible subjective differences between the spelling and painting application and to use the information gathered to improve the application.
All calculations reported here (Exp I and Exp II) were based on 19 selections instead of the originally planned 20 selections due to impediments in the protocol. Additionally, in Exp II, EEG recordings of the first six selections of the painting task could not be included for one subject due to technical problems.
For all healthy subjects (Exp I and Exp II), the duration of the whole session including preparation of the measurement and removing the electrodes after the measurement varied between 1 and 2 h dependent on the time spent for free-painting. The duration of the copy-spelling and copy-painting tasks without breaks added up to 24.17 min.
On average, the time spent on painting a picture during free-painting was 29.10 min (SD ± 18.59) in the group of healthy subjects using the colored matrix (Exp I) and 17.30 min (SD ± 9.92) in the group of healthy subjects using the black and white matrix (Exp II).
For the ALS-patients, some extra time was needed to fulfill their medical needs (e.g., suction cleaning). The duration of one entire session varied between 2 and 3 h. The duration of copy-spelling and copy-painting without breaks added up to 32.91 min for patients A and C and 15.42 min for patient B.
Due to fatigue none of the patients completed the free-painting task after the copy-spelling and copy-painting tasks.
Individual and average accuracy, information transfer rates and P300 amplitudes per group are depicted in Table
Subject | Accuracy (%) | Information transfer rate (bits/min) | P300-Amplitudes (μV) | ||||||
---|---|---|---|---|---|---|---|---|---|
Accuracy copy-spelling | Accuracy copy-painting | Copy-spelling (Wolpaw et al., |
Copy-painting (Wolpaw et al., |
Copy-spelling target | Copy-spelling non-target | Copy-painting target | Copy-painting non-target | ||
ALS-Patients (Exp I) | 89.47 | 94.74 | 5.76 | 5.82 | 4.57 | 0.92 | 4.09 | 0.72 | |
47.37 | 26.32 | 4.77 | 1.41 | 3.51 | 1.17 | 8.78 | 1.52 | ||
100 | 89.47 | 7.15 | 5.19 | 4.95 | 1.29 | 4.45 | -0.42 | ||
Healthy subjects colored matrix (Exp I) | 78.95 | 57.89 | 6.68 | 3.51 | 6.51 | 2.87 | 8.04 | 4.29 | |
75.95 | 68.42 | 6.15 | 4.49 | 5.19 | 2.24 | 7.00 | 1.25 | ||
100 | 94.74 | 9.94 | 7.78 | 8.08 | 3.94 | 4.75 | 4.39 | ||
84.21 | 68.42 | 7.23 | 4.50 | 3.90 | 2.00 | 2.57 | 1.43 | ||
89.47 | 89.47 | 7.96 | 6.93 | 6.78 | 0.58 | 5.01 | 0.66 | ||
100 | 94.74 | 9.94 | 7.78 | 5.44 | 1.43 | 2.12 | 1.58 | ||
100 | 89.47 | 9.94 | 6.93 | 6.90 | 2.60 | 2.83 | 3.66 | ||
100 | 78.95 | 9.94 | 5.70 | 7.17 | 1.88 | 5.00 | 1.23 | ||
100 | 100 | 9.94 | 8.66 | 7.44 | 1.17 | 5.89 | 0.44 | ||
89.47 | 63.16 | 7.96 | 4.00 | 4.95 | 1.43 | 0.55 | 1.40 | ||
Healthy subjects black and white matrix (Exp II) | 94.74 | 84.21 | 8.94 | 6.30 | 9.54 | 0.75 | 7.16 | 1.04 | |
84.21 | 100 | 7.23 | 8.66 | 6.33 | 2.49 | 2.23 | 3.09 | ||
100 | 100 | 9.94 | 8.66 | 4.49 | 0.50 | 8.18 | 1.15 | ||
100 | 94.74 | 9.94 | 7.78 | 4.67 | -0.06 | 6.90 | 0.36 | ||
100 | 94.74 | 9.94 | 7.78 | 6.81 | 1.39 | 8.57 | 2.52 | ||
100 | 89.47 | 9.94 | 7.07 | 6.50 | 0.18 | 9.01 | -0.28 | ||
89.47 | 84.21 | 7.96 | 5.82 | 10.82 | 2.99 | 11.30 | 1.74 | ||
78.98 | 94.74 | 6.55 | 7.78 | 5.82 | 2.35 | 4.16 | 0.99 | ||
89.47 | 89.47 | 7.96 | 7.07 | 2.92 | 2.01 | 4.35 | 0.70 | ||
94.74 | 94.74 | 8.94 | 7.78 | 8.14 | 4.83 | 13.12 | 5.02 | ||
In Exp I, a significant drop in average accuracy between the copy-spelling task and the copy-painting task was found in the healthy group (
Patients’ individual P300 amplitudes and average P300 amplitudes per group of healthy subjects are depicted in Figure
In the group of healthy subjects using the colored matrix (Exp I), a significant drop in P300 target amplitudes from copy-spelling to copy-painting was found (t(9) = 2.76,
Target and non-target P300 amplitudes were significantly differentiable in the copy-spelling conditions (Exp I:
Additionally to the information transfer rates calculated using the formula of Wolpaw et al. (
The SEIQol index scores were 87.1 for patient A, 58.8 for patient B and 82.7 for patient C. Patient A reported communication (importance,
Patient B reported family (
For patient C, the five most important areas of life reported were family (
Depression was measured using the ADI-12 with a score of 17 for patient A, 18 for patient B and 26 for patient C of a total of 48. All patients were below the cut off (30) for Major Depressive Disorder (sensitivity 100%, specificity 83%). However, with a value of 26 patient C fell above the cut off (23) for the possibility of any affective disorder including minor depression (sensitivity 100%, specificity 60%) (Hammer et al.,
All data concerning mood and motivation collected with the VAS, SEL, and FAM-BCI2000 per group are depicted in Table
Subject | Session | FAM-BCI2000 | SEL | VAS | ||||
---|---|---|---|---|---|---|---|---|
Confidence | Incompetence fear | Interest | Challenge | Mood | Mood | Motivation | ||
2.50 | 1.20 | 4.80 | 5.00 | 4.00 | 8.00 | 9.00 | ||
2.75 | 1.00 | 5.00 | 5.25 | 4.20 | 10.00 | 9.00 | ||
2.75 | 1.00 | 4.80 | 5.25 | 4.00 | 10.00 | 10.00 | ||
2.75 | 1.00 | 7.00 | 7.00 | 3.40 | 8.10 | 8.50 | ||
2.75 | 1.00 | 7.00 | 7.00 | 3.70 | 8.50 | 8.10 | ||
2.75 | 1.00 | 7.00 | 7.00 | 3.60 | 8.90 | 8.60 | ||
2.75 | 1.80 | 6.60 | 6.00 | 4.00 | 8.70 | 8.50 | ||
x | x | x | x | x | x | x | ||
2.50 | 2.40 | 6.40 | 5.75 | 3.50 | 9.00 | 9.00 | ||
2.80 (0.56) | 1.84 (0.88) | 5.26 (1.03) | 4.02 (0.63) | 3.95 (0.35) | 7.67 (1.27) | 8.00 (1.43) | ||
2.65 (0.44) | 1.82 (0.78) | 5.48 (0.92) | 4.25 (0.79) | 4.10 (0.35) | 7.79 (1.63) | 8.43 (1.23) | ||
2.8 (0.35) | 1.7 (0.73) | 5.18 (1.58) | 4.25 (0.69) | 4.04 (0.48) | 7.49 (1.42) | 7.92 (1.63) | ||
2.83 (0.49) | 2.12 (1.00) | 5.22 (1.00) | 4.25 (1.22) | 3.85 (0.65) | 7.21 (1.82) | 8.31 (1.28) | ||
3.08 (0.38) | 2.13 (1.27) | 5.16 (0.83) | 4.18 (0.83) | 3.74 0.46) | 6.59 (1.55) | 8.39 (1.47) | ||
2.83 (0.55) | 1.74 (1.13) | 4.82 (1.48) | 3.85 (0.99) | 3.95 (0.30) | 7.41 (1.66) | 8.39 (1.47) | ||
In the group of healthy subjects using the colored matrix (Exp I), a significant correlation was found between challenge (FAM-BCI2000) before copy-painting and the accuracy of copy-painting (
In the group of healthy subjects using the black and white matrix (Exp II), challenge before free-painting was significantly correlated to the duration of free-painting (
Analyzing the evaluation form in the group of healthy subjects using the colored matrix (Exp I), a significant difference in clarity and easiness of use could be found between the matrices of the P300-Spelling (
In the group of healthy subjects using the black and white matrix (Exp II), no significant difference in clarity and easiness of use between the matrices of the P300-Spelling application (
The current study investigated whether the new P300-Brain Painting application would provide the same accuracy as the well established and highly accurate P300-Spelling application and to test the acceptance of the application in severely impaired patients. ALS-patients were able to use the P300-Brain Painting application with high accuracies. In the group of healthy subjects, a drop of accuracy between the P300-Spelling application and the P300-Brain Painting application which was found for the colored matrix could be prevented by adapting the painting matrix to that of the P300-Spelling application.
In the first experiment (Exp I), it was hypothesized that the P300-Brain Painting application is usable with the same accuracy and evokes P300 amplitudes comparable to the P300-Spelling application. The group of ALS-patients was too small for statistical analysis. However, the P300-Brain Painting application and the P300-Spelling application were usable with a high accuracy (above 89%) in two of the three patients, even if the shape of the their P300 differed from that of the classical P300 in healthy subjects (see Figure
Patient B with lower accuracy rates reported afterwards that she was suffering from pain during the entire day, which could have influenced her results. Moreover, the fewer number of sequences (
In the group of healthy subjects, our hypothesis of equal performance in both applications could not be confirmed. Lower accuracy and reduced P300 amplitudes were found for the P300-Brain Painting application as compared to the P300-Spelling application. These results might be explained by the perceived higher complexity and lower user-friendliness of the P300-Brain Painting matrix. This may imply that the participants needed more cognitive resources to decide which symbol to select, leaving less cognitive resources to focus on the target item for selection which is consistent with a focus of attention within working memory (Cowan,
Colors are perceived as salient stimuli among other symbols which in the P300-Brain Painting application could have influenced selective attention needed to detect the target stimulus (Johnston and Dark,
Concerning different aspects of mood and motivation on a descriptive level, higher values of interest, challenge, mood (VAS), and motivation (VAS) and lower values of incompetence fear were found for the group of ALS-patients compared to the group of healthy subjects. Two of the three patients had already experience with other P300-BCI applications which may explain the lower values of incompetence fear (Nijboer et al.,
Similarly, for healthy subjects we found a significant positive correlation between challenge and accuracy of copy-painting indicating that perceived challenge may be accompanied by more effort to perform better.
As an aside we would like to point out that none of the patients had Major Depressive Disorder, but moderate depressive symptoms were found in patient C. Patient C was still included in the study because he showed a well differentiable P300 and the mild depression was found not to interfere with the study design.
In the second experiment (Exp II), it was hypothesized that the drop in accuracy and P300 amplitudes between the P300-Spelling and the P300-Brain Painting application found in Exp I would disappear when using a black and white and functionally ordered matrix. This hypothesis could be confirmed as no significant difference was found between accuracy and P300 amplitude in the copy-spelling and copy-painting tasks.
The averaged results for accuracy and P300-amplitudes for the copy-spelling task did not differ between both experiments which supported the assumption that the equal performance during copy-spelling and copy-painting in Exp II was indeed due to an improvement in painting with the newly designed black and white matrix and not due to the unlikely possibility that the subjects in Exp II were scoring worse on the copy-spelling task while achieving equal results in the copy-painting task. Moreover, in line with these results, also the difference between the subjective evaluation of complexity and easiness of use between the matrices of the P300-Spelling and the P300-Brain Painting application disappeared. This was also in line with our hypothesis that the optical differences (Theeuwes,
Taken together, the current studies showed that the new P300-Brain Painting application is usable with high accuracy both in healthy subjects and severely impaired patients with ALS. Moreover, accuracies and P300 amplitudes found for the P300-Brain Painting applications are in line with prior studies which found accuracies and P300 amplitudes of >90%/5.90 μV (Kleih et al.,
One limitation for the user-friendliness of the P300-Brain Painting application is its low information transfer rate compared to spelling, where information transfer rates of up to 23 bits/min were reported (Kleih et al.,
We conclude that the P300-Brain Painting application is usable with high accuracy by healthy subjects and neurological, severely paralyzed patients alike and it allows the user creative expression, which is specifically enjoyed by patients. In the future, matrix designs should be restrained to black and white elements; at least at the beginning of training. As patients experienced no drop in performance, extensive training with the P300 matrix might reduce the effect of selective attention, which favors the most salient stimuli. Also, other input signals than event-related potentials (ERP) should be tested to further optimize functionality and user-friendliness of the Brain Painting application. However, already today the P300-Brain Painting application, when applied as leisure time activity, allows the patients to be productive and creative and to participate in the prolific society by means of exhibitions of their paintings.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
This work is supported by the European ICT Programme Project FP7-224631. This paper only reflects the authors’ views and funding agencies are not liable for any use that may be made of the information contained herein. Special thanks to the patients who participated in the study for their time, effort and enthusiasm.