Cognitive assessment of executive functions using brain computer interface and eye-tracking

New technologies to enable augmentative and alternative communication in Amyotrophic Lateral Sclerosis (ALS) have been recently used in several studies. However, a comprehensive battery for cognitive assessment has not been implemented yet. Brain computer interfaces are innovative systems able to generate a control signal from brain responses conveying messages directly to a computer. Another available technology for communication purposes is the Eye-tracker system, that conveys messages from eye-movement to a computer. In this study we explored the use of these two technologies for the cognitive assessment of executive functions in a healthy population and in a ALS patient, also verifying usability, pleasantness, fatigue, and emotional aspects related to the setting. Our preliminary results may have interesting implications for both clinical practice (the availability of an effective tool for neuropsychological evaluation of ALS patients) and ethical issues.


Introduction
Some of the most consistently reported cognitive changes in Amyotrophic Lateral Sclerosisi (ALS) regard frontal executive functions, in particular verbal fluency, together with attention, working memory, planning and abstract reasoning [1][2][3][4][5][6].However, the assessment of cognitive impairment still remains a problematic issue in ALS patients, because of the possible presence of severe physical disabilities, including movement impairment, paralysis in the advanced stages and dysarthria, which interfere with the outcome of traditional neuropsychological testing.In fact, all standard assessment tools involve verbal or motor responses.* Corresponding author.Email: p.cipresso@auxologico.it New technologies to enable communication have been recently used in several studies.Among these methods, Brain Computer Interface (BCI) and Eye Tracking (ET) are the most promising technologies.BCI uses neurophysiological signals as input commands to control external devices, while ET allows the measurement of eye position and movements.However, a comprehensive battery for cognitive assessment has never been implemented with these methodologies.
With regard to ET, to date, no applications have been developed, using it as a communication device in order to administer traditional cognitive tasks to patients.
The main disadvantage in the use of ET systems is that they require preserved ocular mobility, and the absence of relevant visual deficits; the former may be lost or altered in the final stages of the disease, and the latter may be present in patients of advanced age, thus forbidding the use of this device.So, in case of impaired ocular mobility, there is the EAI European Alliance for Innovation P. Cipresso et al.
2 need of a more direct interface between voluntary cortex activity and a technological device.BCI may offer an interesting answer to this issue with a growing number of different paradigms proposed.The most frequently used is the P300, an event related potential (ERP) elicited by infrequent task stimuli, that occurs 200-700 ms after stimulus onset; it is typically recorded over central-parietal scalp locations [8][9][10] and it can also be used by patients suffering from complete paralysis and impairment in oculomotor dysfunctions, such as locked-in patients.
It is notable, however, that 20% of subjects are not proficient in using BCI; this phenomenon is called "BCI illiteracy" [11] and it is due to the fact that some users do not produce brain activity detectable at the scalp level, independently from the health conditions: even about 10% of healthy subjects do not produce "usable" P300.
With regard to ALS patients, studies have shown that some of them produce less typical ERPs than healthy matched subjects [12]; [13].A previous ERP study in patients with sporadic ALS found that P3a and P3b amplitudes of ALS patients were lower and P3a latencies were significantly longer compared with the controls [14]; ERP recordings in non-demented patients with sporadic ALS also showed prolonged N200 and P300 latencies compared to healthy controls [15].Ogawa and colleagues [16], by employing neuropsychological measures, eventrelated potentials (ERPs) and clinical scales, studied a sample of patients with early-stage sporadic ALS.They found that patients with the bulbar-onset type showed marked prolongation of P3 latency compared to patients with the limb-onset type and controls.Furthermore, bulbar functional rating scale correlated with prolonged P3 latency and low P3 amplitude.Additionally, patients with bulbaronset ALS had consistently poorer cognitive test performance than those with limb-onset ALS [17].These results may represent a challenge for the use of P300 as an input signal in BCIs.Kübler and Birbaumer [7] investigated the relationship between the level of motor and physical impairment and the ability to use brain computer interface by comparing three different BCI systems (P300, SCP -Slow Cortical Potentials and SMRs -sensorimotor rhythms).They found no continuous decrement in BCI performance with physical decline, even in the completed locked-in state (CLIS) where no communication was possible.
Two important criteria in order to evaluate the feasibility of a BCI system are speed and accuracy [18].The former is related to the fact that the more rapidly a BCI can be controlled, the greater quantity of information can be produced by the user and the greater the chance for effective communication.Obviously, compared to verbal speech production, communication rate is severely reduced with BCI.With regard to accuracy, it consists of the percentage of correct selections per time interval.A wrong selection could turn into an error in communication, with both practical and psychological consequences for the user.In order to avoid this, the BCI system must be equipped with options that allow the user to correct wrong selections and a balance between speed and accuracy should be identified.BCI has been preliminarily employed in order to administrate cognitive testing [33][34][35].In a first study of Iversen and Colleagues [34], training was applied to two severely paralyzed ALS patients, during which the patients could learn to control certain components of their EEG in order to direct the movement of a visual symbol on a monitor.Following, a series of two-choice cognitive task were administered.
In a successive study, Iversen and Colleagues [35], employed the same SCP-EEG control in order to administrate a conditional-associative learning task to a latestage ALS patient, testing the ability to learn arbitrary associations among visual stimuli.In both studies, a good level of accuracy was observed in detecting patients performances, according to a within subjects experimental design.Patients were also able to understand the verbal instructions and to respond accordingly in the successive tasks.However, this method owns some important limitation: first, it requires an extensive pretraining in order to learn to control EEG, which can take some weeks; second, the method cannot be used for tasks based on recall or where a choice must be made among more than two stimuli.Perego and Colleagues [33] adapted a widely used clinical test (Raven Colored Progressive Matrix) to a SSVEP-BCI paradigm in order to verify whether BCI affects the performance due to fatigue and cognitive load.They found, in a healthy population, that the BCI-based administration did not affect performance.
More recently, an innovative study [19] aimed to evaluate the complementary use of P300 BCI and eye-tracking technologies both as Augmentative and Alternative Communication (AAC) systems and as cognitive assessment tools.
The aim of this study is to develop an adapted computerized version of a widely used test for the assessment of executive functions, i.e. the Verbal Fluency test, to be administered by means of BCI and ET devices.Moreover, we aimed to assess the overall system usability and user-friendliness.In order to fine-tune the overall testing setup, we performed a pilot study with 8 healthy subjects and one ALS patient.Specifically, participants were administered a phonemic and semantic verbal fluency test; measures of feasibility, pleasantness and fatigue were also collected.Emotional aspects related to the experimental setting, have been evaluated, too.In this paper we report the results of this pilot study.

Participants
Eight healthy subjects (4 females and 4 males), aged between 25 and 39 (M: 31.75,SD: 5.946), were recruited.They were all volunteers with a schooling degree ranging from 13 to 24 years of education (M: 19, SD: 4.276).All the subjects were experienced in the use of PC (50% fair or good and 50% excellent), some of them (50%) declared to have already used Brain Computer Interface or an Eyetracker system, and more than half (62.5%) had already participated into EEG experiments.Exclusion criteria were the presence of cardiovascular, neurological, sensory or psychiatric diseases.Participants were asked to avoid drinking caffeine or alcohol and smoking prior to the experimental test in order to prevent any effects of these substances on the central and autonomic nervous system.Besides, a male ALS patient with bulbar onset, (age: 46 years; years of education: 18), diagnosed by an expert consultant neurologist, according to the El Escorial Criteria (Brooks et al., 2000), was also included in the study.He was not experienced in the use of BCI or ET, while rated as good its experience in the use of PC.He has been recruited at the inpatient-outpatient Neuromuscular Clinic at the Department of Neurology of thr San Luca Hospital, IRCCS Istituto Auxologico Italiano in Milan.

System Setting
Test architecture (Figure 1) was composed of an ET system and a BCI device, both controlled by a laptop PC, connected to an external monitor, meant for the stimuli presentation (Display PC). Figure 1 shows the adopted test setup.The BCI device module was based on the g.USBAmp (7) biosignal amplifier, connected to an active electrode head cuff (5).The biosignal amplifier was connected to a portable laptop (6).This laptop was connected to an external monitor (2), where the stimuli were presented to the participant.The ET consisted of a high-speed infrared camera and the related illuminator (3), positioned just below the Display Monitor.The ET host computer (1) acquired eye-head information via the camera and processed them in real time.The two computers were connected by Ethernet cross-cable, for fast communications in order to synchronize the different acquisitions performed by the BCI and the ET.This allowed to extract interesting features from the combined use of both technologies, e.g. the screen eye-gaze patterns during the BCI tests.
On the Display PC a suitable custom software provided information of the general management of the tests and the sequence of the stimuli for the ET tests, while for the BCI, we used an adapted version of the widespread used BCI2000.To setup the BCI system we used a g.USBamp amplifier for the acquisition of 16 EEG channels.Channel names based on the International 10-20 locations were: FZ, C3, C4, CZ, CPZ, P3, P4, PZ, PO3, PO4, POZ, PO7, PO8, O1, O2, OZ; ground was placed in FPZ, and reference was located on the left ear lobe.

Eye-tracking
The ET was an Eyelink-1000 (SR Research Ltd., Mississauga, Ontario, Canada), consisting of a high-speed infrared camera and the related illuminator, positioned just below the Display Monitor.
Eye movement data consisted of moment-to-moment measures of the eyes' displacements along the vertical and horizontal axes (in millimeters) within the spatial working area of the monitor screen.
The pupil dilation and gazes were acquired, based on the pupil position and the corneal reflection on the frontal surface of participants' eyes (caused by an infrared light source), at 500 Hz (one record data per 2 milliseconds) by means of a EyeLink 1000 using custom software programmed in C.After the experiment, the signals were extracted and processed with custom software developed using MATLAB 7.2 (The Mathworks, Inc.; Natick, MA).

Additional equipments
The lab was equipped with a portable PC with an external monitor, for delivering the stimuli (consisting of letters on a keyboard) and acquiring EEG and ET data.Another PC was dedicated to manage the ET system and all the related processes.Finally, a chinrest was mounted on the desk to prevent subjects from moving during the ET sessions.

Signal processing and translational algorithms
To recognize P300 waves elicited by the flashing letters, we used the well-known BCI2000 software [36] customized for our purposes to accomplish technical requirements in the described EEG system with the 16 channels that we defined.Collected data were also analyzed using Matlab 7.10 (The Mathworks, Natick, MA), in order to investigate the recorded eye movements too.

Neuropsychological Tests
The experiment consisted of two different parts, one to perform calibration and testing with the ET and the other one with BCI.We used a modified version of the Phonemic and Semantic Verbal Fluency test.It has been recognized as the most sensitive tools in detecting cognitive impairments in ALS patients [5].For the Phonemic Fluency in the BCI session we measured the time taken by the subjects to think the word starting with the letters "A" and "H", alternately presented.A timer was started by the researcher right after the letter was presented to the subject.Then, the timer was stopped when the subjects indicated to be ready to effectively write the word with the BCI system and the experimenter started the system.The test finished when the participants had written at least 25-30 characters.Then the procedure was repeated for the Semantic Fluency, with the categories "furniture" and "means of transport."Concerning the ET assessment, we performed Phonemic and Semantic Verbal Fluencies using an adapted version of the Verbal Fluency Index [6] to adjust for ocular-motor components.In the Phonemic Verbal Fluency task, the subjects were required to write all the words starting with "Q" and "Z" letters, in one minute each (generation condition).Then the subjects were asked to copy exactly the same characters while the execution time was measured (control condition).The same procedure was repeated for the Semantic Fluency task, with "type of shoes" and "cooking ingredients" categories.The difference between time for the generation condition and that for the control condition was then calculated and used to determine the Verbal Fluency Index.[6]

ET Words-copy Test With Different Keyboards
Subjects were asked to copy a list of words, using two different kinds of keyboards.The first one was a standard keyboard, with letters arranged in alphabetical order (from A to Z), while the second one was a "scrambled" keyboard, with letters arranged in random order (Fig. 3).Execution times taken for both tasks were recorded.The aim of this test was to assess the user facilitation for the standard keyboard due to the reduced visual searching effort in comparison with the random arranged letters.

STAI form Y Questionnaire
The Italian version of the STAI form Y questionnaire was used to assess changes in two different types of anxiety, namely, anxiety detected as the subject's current state (STAI-Y1, i.e. state anxiety) and anxiety detected as a reasonably stable trait of the personality of an individual (STAI-Y2, i.e., trait anxiety).A total of four Self-reported STAI-Y1 were gathered before and after both BCI and Eyetracking sessions.One self-reported STAI-Y2 was gathered five minutes before the experimental session [22].

Self Assessment Manikin (SAM)
One of the most currently employed methods for identifying affective states in subjects during an experimental session is the Self Assessment Manikin (SAM).It is a non-verbal pictorial assessment technique that considers the two dimensions of "activation", namely, physiological arousal and emotional valence and directly measures the pleasure, arousal, and dominance associated with subjects' affective reactions.[23][24][25].A total of four Self-reported SAMs were gathered before and after both BCI and ET sessions.

Usability Inventory Post-test Questionnaire
Since there are no usability validated tests for ET and BCI systems, we developed a questionnaire composed of 19 items, based on the available literature [26][27][28][29].Our purpose was to evaluate the instruments' general usability and some specific variables such as fatigue, screen readability and perceived usefulness.

Experimental Procedures
Experiments were performed at the Applied Technology for Neuro-Psychology Laboratory and at the San Luca Hospital, both located at the IRCCS Istituto Auxologico Italiano in Milan, Participants were tested by a neuropsychologist and a senior psychology researcher.The experimenters were instructed to maintain a neutral vocal tone and a neutral behavior for the entire duration of the experimental session.Since the experiment was constituted by two parts (BCI and ET calibration and testing), the design was balanced between the participants: 4 subjects first performed the ET part, while the remaining 4 subjects first underwent BCI.Before and after each session, the participants were asked to fulfill the described questionnaires.
Both instruments required a calibration phase,.After the calibration, the neuropsychologist started the Phonemic and Semantic Fluency tests with BCI ET.
At the end of ET session, the subjects performed the ET words-copy test with the scrambled and standard keyboards.The total experiment duration was about two hours.

Results
Data were analyzed with the aid of the statistical software SPSS, version 17 (Statistical Package for the Social Sciences-SPSS for Windows, Chicago, Illinois, USA).
Nonparametric tests were preferred, even if several measures showed a normal distribution (according to Kolmogorov-Smirnov test).In the following paragraphs the main results of this preliminary study are presented.

Fluency Tests
In BCI session, Phonemic Verbal Fluency average time was 6.42±2.76seconds and Semantic Verbal Fluency average time was 4.08±1.94seconds.Regarding the ET session, we used the Fluency Index, as above described.Phonemic Verbal Fluency Index was 4.28±5.84seconds and the Semantic Verbal Fluency Index was 3.37±2.58seconds.These data will be used to compare future patients' performances.

Behavioral Measures
In order to assess if BCI and ET generated negative affective states, anxiety level has been first measured via the STAI-Y1 questionnaire for the two different conditions on BCI (pre-and post-BCI), and the two different conditions on ET (pre-and post-ET).Measures of pre-post ET and BCI were also collected for the three scales of the SAM questionnaire, namely pleasure, arousal, and dominance.Wilcoxon Signed Ranks Tests indicated no statistical differences for both the pre-post STAI-Y1 and pre-post SAM scales, indicating that no negative affective state or anxiety have been generated by the performance with BCI or ET.However, a small increase in anxiety was detected after the use of BCI.

Usability
As it is showed in Table 1, subjects recognized both systems as enough usable, but ET was generally perceived as more usable than BCI (statistical significance is calculated with Wilcoxon Signed Ranks Tests).To analyze the answers to items which emphasize a positive impact of the instruments (item: 1, 2, 4, 5, 6, 10, 11, 12, 14, 18, and 19), a model of internal consistency, based on the average inter-item correlation, showed Cronbach's Alphas of 0.792 and 0.784, respectively for BCI and ET.The average value of the "positive item" consequently created allowed us to compare BCI and ET (5.17±0.96 vs. 6.06±0.53;Z = -2.66,p = .008),and to confirm the highest perceived usability for the ET system, also considering only positive items.

Copying Text using Different Keyboards with Eye-tracking
The mean time to copy a list of words with the standard virtual keyboard (in alphabetic order, since QWERTY layout wasn't familiar to all participants) was 46.09±6.55,while the average time to copy the same text with the scrambled keyboard was 60.17±21.23.Wilcoxon Signed Ranks Tests indicated statistical significant differences (Z = -2.201,p = .028).

BCI Calibration
BCI calibration is a critical issues.Above all, it is crucial to verify if such process affects BCI results, in terms of errors made during the experimental test.In the healthy sample, the mean calibration accuracy was 89.73%,,while the accuracy of BCI system during the test was 81.72% (i.e.18.28% of errors).However, since 6 subjects obtained 100% of correct calibration and did very few errors, we were interested in understanding if such two processes (calibration and final results) was correlated.Results showed a negative Spearman's correlation (ρ = -0.862,p = .001),indicating that good calibration seems to lead to fewer errors.

Case study
In order to preliminarily evaluate the feasibility of BCI and ET systems for cognitive assessment and communication purposes also in a clinical population, we collected data from one ALS patient (M.Z.).Main results are shown in Tables 2 and 3.With regard to BCI, data suggested a level of accuracy in calibration and testing, which is comparable to that obtained by healthy subjects.
Concerning ET, the overall performance supported a good rate of accuracy.Moreover, no relevant differences in STAI and SAM pre and post BCI and ET scores were recorded.

Conclusions and Future Work
No studies have been performed so far to evaluate the combined use of BCI and ET system for AAC and cognitive assessment in ALS.
As previously mentioned, BCI is a system that enables the generation of a control signal from brain responses such as sensorimotor rhythms and evoked potentials; it bypasses motor output and conveys messages directly from the brain to a computer.Therefore, it constitutes a novel communication option for people with severe motor disabilities, such as ALS patients.Another available technology for communication purposes is the ET system.
Starting from these considerations, a comprehensive neuropsychological battery based on the use of P300based BCI and Eye-tracker could be created.In particular, some traditional neuropsychological tests could be modified to create computerized short versions of several tests, that could be adapted for BCI and ET administration.
Our pilot study provided evidences for the effectiveness and usability of these techniques.Specifically, the BCI computerized assessment could provide new insights into the understanding of cognitive deficits, when administered to ALS patients, through the integration of multidisciplinary data: neurophysiological, neuropsychological, behavioural and psychological.
The proposed study is characterized by at least two innovative aspects: (1) the comparison between two interesting technologies, one already extensively investigated (ET), the other being a very promising candidate (P300-based BCI), (2) the adaptation to BCI and ET of the Verbal Fluency task for the neuropsychological assessment of higher order cognitive functions in ALS patients.As we described above, some preliminary attempts have been made in this direction [33][34][35].However, the known adopted BCI paradigms require non trivial adjustments of the original neuropsychological test procedures and a training phase before tests administration.Differently, in P300-based BCIs here employed the learning of self-regulation of the brain response and feedback is not necessary and the short latency of the P300 allows a selection of items faster than any other BCI systems.
Results showed a good usability of both instruments, better for ET, but promising for BCI too.Furthermore, the strong negative correlation between trait anxiety and perceived usability clearly showed that the higher the subject is anxious, the more the instruments will be perceived as demanding, tiring, and difficult to use.
Finally, no emotional effects on cognitive performances were revealed by the administered psychological measures.
As expected, BCI calibration was a critical issue.These data suggested that it is crucial to extend the calibration phase in order to reach very high correct ratio (close to 100%).Moreover, the kind of virtual keyboard used in the task clearly influences the observed performances.This resuls suggest a potential use of the scrambled keyboard as a novel cognitive test.
Finally, these preliminary results may have interesting implications for both clinical practice (the availability of an effective tool for neuropsychological evaluation of ALS patients) and ethical issues, the last one arising from a proper assessment of cognitive ability preservation, in particular regarding relevant decisions about medical treatments, economical and end-life issues.

Acknowledgements.
This study has been made possible partially due to funds from the Lombardy Region project "eBrain: BCI-ET for ALS (eBrain: BCI-ET nella SLA)." ).A preliminary version of this work has been presented at MindCare Workshop and published in [32].This new version contains a renewed "materials and methods" section to better explain the complex technologies used in the study.More, results have been enriched by adding new analysis and a renewed discussion has been written accordingly.

Figure 2 .
Figure 2. Experimental procedure timeline for both BCI and ET sessions All participants were required to sign an informed consent form.During the systems set-up, they were informed about the broad functions of electrodes.Since the experiment was constituted by two parts (BCI and ET calibration and testing), the design was balanced between the participants: 4 subjects first performed the ET part, while the remaining 4 subjects first underwent BCI.Before and after each session, the participants were asked to fulfill the described questionnaires.Both instruments required a calibration phase,.After the calibration, the neuropsychologist started the Phonemic and Semantic Fluency tests with BCI ET.At the end of ET session, the subjects performed the ET words-copy test with the scrambled and standard keyboards.The total experiment duration was about two hours.

Figure 3 .
Figure 3.The used ET Keyboards and the Saccades plotted over them: (a) Standard Virtual Keyboard, (b) Scrambled Keyboard, (c) Saccades on Standard Virtual Keyboard, (d) Saccades on Scrambled Keyboard.

Table 1 .
Average values of 7-point Likert scale items of the usability questionnaire

Table 2 .
ALS patient performances on behavioral and cognitive testing on ET and BCI assessment