Testing Brain–Computer Interfaces with Airplane Pilots under New Motor Imagery Tasks

German Rodriguez-Bermudez; Alejandro Lopez-Belchi; Arnaud Girault

doi:10.2991/ijcis.d.190806.001

<Previous Article In Issue

Download article (PDF)

Next Article In Issue>

Volume 12, Issue 2, 2019, Pages 937 - 946

Testing Brain–Computer Interfaces with Airplane Pilots under New Motor Imagery Tasks

Authors

German Rodriguez-Bermudez¹^{, *}, Alejandro Lopez-Belchi¹, Arnaud Girault²

¹University Center of Defence at the Spanish Air Force Academy, UPCT-MDE Calle Coronel Lopez Peña, s/n, Santiago de la Ribera, Murcia, 30720, Spain

²French Air Force Academy, Base Aérienne 701, Route de Saint Jean, Salon-de-Provence, 13661, France

^*Corresponding author. Email: german.rodriguez@cud.upct.es

Corresponding Author

German Rodriguez-Bermudez

Received 3 April 2019, Accepted 31 July 2019, Available Online 23 August 2019.

DOI: 10.2991/ijcis.d.190806.001 How to use a DOI?
Keywords: Brain computer interface; Motor imagery; Aircraft pilots
Abstract: The purpose of a brain–computer interface (BCI) is the recording of brain signals to translate them into commands. This work proposes new naturalistic and intuitive motor imagery (MI) BCI task for aircraft Pilots for a BCI System, and explore if they take advantage of their previous motor experience in MI–BCI experiments. Results show that Pilots get better performance than a control group and that it is possible to get good classification accuracy results with new naturalistic MI tasks.
Copyright: © 2019 The Authors. Published by Atlantis Press SARL.
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

Understanding how our brain works and obtaining the capability to analyze our thinking processes are important fields of research which have recently advanced considerably. In the beginning, this field of research was anecdotal or considered as pure fiction, but the development made since the first human electroencephalography (EEG) in 1924 made by Hans Berger [1] have created possibility of an infinite number of applications for neurosciences [2]. The purpose of a brain–computer interface (BCI) is to record brain signals and translate them into commands to operate a device. At the same time, it provides feedback to the user about how the intentions are transformed into actions [3]. This definition explains how a BCI system works. Based on the previous definition, a BCI based device transforms mental activity into a physical effect while the user does not have to make any physical effort.

The great number of applications make this research extremely interesting [4].

A typical BCI system based on motor imagery (MI) can be seen as a pattern recognition system, see Fig. 1.

The system has five consecutive stages: i) EEG acquisition; ii) Preprocessing; iii) Feature extraction (FE); iv) Classification, and finally, v) Feedback.

Everything begins with an EEG signal generation of the user, in this case, MI.

After the signal acquisition, in the next step, it is necessary to apply some preprocessing to the signal. Preprocessing consists of cleaning and suppressing noising input data called artifacts to enhance the relevant information contained in the raw signals [5,6].

Once the signal is clean, the data reaches the feature extraction (FE) stage. The interesting data of the EEG signals are reflected in the features extracted by different techniques. Then, the Features go to the Classification stage to be classified. The purpose of the classification is to determine the intentions based on the extracted features. Finally, if the feedback is enabled in the system the user can see the classification results and it helps to modulate their Brain signals in a right way in the next iteration.

In a binary MI–BCI experiment, an accuracy of between 80% and 90% is expected after 6–9 training sessions of 20 minutes each [7]. Nevertheless, and according to state of the art, certain subjects may face difficulties in using MI based BCI systems and, in these cases, the classification performances are quite poor even after using multiple training sessions [8]. This is called BCI illiteracy [9]. Therefore, a previous selection of subjects with good classification performances in the experiments is expected in order to obtain high classification results. Starting with untrained users presents difficulties because of the high risk that their learning may be limited by frustration [10]. A variety of methods for optimizing training were described in the literature [10]. User-centered training approaches may be employed in control tasks, and signal processing algorithms which are highly customized to individual users [11]. It is important to note that motivation is one of the variables to take into account to get good results [10].

In MI–BCI experiments it is usual to imagine peculiar movements of the hands, the feet or tongue [12], to be discriminated versus other parts of the body [13]. Also, it is very common to use dataset available on the Internet such as BCI competitions or Physionet. There are several mechanisms to discriminate left/right hand in BCI experiments using datasets from BCI competitions. For example, recently Bashar et al. [14], Chatterjee et al. [15,16], or Seoane et al. [17] published works using data from BCI competition II, and there are many works with data from other BCI competitions [17–20]. There are also papers with the same MI task, published with data collected by the researchers [8,21]. Other examples of binary MI tasks available in the literature are discrimination left/right hand and tongue [22,23]; left/right hand and word generation [24]; left, right hand, foot and tongue [25]; left/right hand, foot and tongue [26]; left, right hand, both hands and both feet [27]; extension/flexion of wrist, open/close fingers [28].

In contrast with the previous part of the introduction, this work wants to explore the control of a MI–BCI system by aeroplane pilots performing new naturalistic and intuitive MI tasks. Conceivably, naturalistic MI would be more intuitive for users and thus require less training time [12]. Another objective of this work is to test if a specific professional group takes advantage of their previous motor experience in a MI–BCI experiment, and also, the authors want to explore the evolution of results of these users in a new MI task.

Two experiments were developed to answer these questions. The first one contains two groups of users: A control group (CG) with 7 users without any experience as aircraft pilots and a group with 7 pilots (PG) of the Spanish Airforce Academy. All of them have to imagine the left and right movement of a plane doing the same movements that that they would do in an aeroplane. It entails that they have to imagine a complex movement that combines the right hand and the feet. Although all the participants in this experiment are motivated (all are volunteers), it is expected that PG would get better results than the CG because they have done it many times before in a real environment [12]. Since, in this work, the authors are planning new MI tasks, it is interesting to check the influence of the previous flight experience in the obtained results, therefore, in this experiment CG act as a reference.

In order to explore more MI commands for BCI systems, a second experiment was developed. Eight pilots were instructed to discriminate the right movement of the aircraft, that combines the right hand and the feet movements, and to press the radio button asking for permission to land. That requires the movement of the left hand and the tongue. It is important to note that none of the users of both experiments have used a BCI system before. Therefore, there is no previous selection of the users. Thus, they are all novices, and it is expected to get low classification accuracy results for all the participants [8], especially in the first and second session.

The system worked with three FE methods to tune the system to each user: Power Spectrum Density (PSD); Adaptive Autoregressive Coefficients (AARs), and a feature reduction method, Principal Component Analysis (PCA). In the classification stage, four techniques: Linear Discriminant Analysis (LDA); Support Vector Machine (SVM); K-Nearest Neighbor (K-NN), and Decision Tree (DT), were implemented. Finally, all combinations between then have been tested to get the BCI system adapted to each user.

2. RELATED WORK

Literature shows papers related to the experiments developed in this work. There are some works to control Unnamed Aerial Vehicle (UAV's) with BCI systems and aircrafts. In 2010, Akce et al. [29] presents preliminary results of an interface that allows a human pilot to remotely teleoperate an unmanned aircraft flying at a fixed altitude using 8 electrodes to distinguish between left and right hand MI. They claimed the feasibility of this approach. At the same time, researchers of University of Minnesota were very active in this area [30]. In 2010, they developed an experiment to control a virtual helicopter in 3-dimensional (3D) space [31]. They used non-invasive 64 electrode cap and four subjects to do the experiment. All of them had previous experience in BCI systems. They were trained in BCI and completed 33, 31, 24, and 20 training sessions with two control commands (left- and right-hand movements). Researchers reported that they were able to control the quadcopter in 3D virtual environment. In 2013 LaFleur et al. [32] presents results with 5 users trained as in the previous experiment [31]. Individual subjects were able to accurately acquire up to 90.5% of all valid targets with a Quadcopter presented while traveling at an average straight-line speed of 0.69 m/s. The MI implemented was the movement of the hands. There are also articles that presents results to control aircrafts. In 2014 Fricke et al. [33] present the work entitled “First Pilot-in-the-Loop simulator experiments on Brain Control of horizontal aircraft motion.” 7 subjects complete the experiment in a realistic cockpit environment and, 6 of them were experienced aircraft pilots. The MI performed was two different classes, hand or feet MI. The results show that brain control of one degree of freedom of the aircraft motion is possible, in particular 2 users achieved high reliability.

Recently, in 2017 Kryger et al. [34] published an experiment to control a flight simulator by one subject diagnosed with a variant of spinocerebellar degeneration resulting in complete quadriplegia. She was implanted with two 96-microelectrode arrays in her left motor cortex. The experiment took 15 weeks. The participant was able to achieve controlled changes in both pitch and roll fairly well. Her strategy combined to imagine visualizing desired aircraft movements and the movement of the hands.

3. MATERIALS AND METHODS

3.1. EEG Signal Generation and Acquisition

For this research, the authors generated a dataset at the University Cenetr of Defence at the Spanish Air Force Academy, Spain. The dataset is composed of EEG signals from 22 male volunteers all of them with normal vision (according with the values established for the Spanish Air Forces); all the characteristics are presented in Table 1. It is remarkable that none of these subjects had used a BCI system before the current experiment. Thus, there has not been a previous selection of the users.

First Experiment

Name	Left/Right	Age	Group	Flight Hours
S1	Right	26	1	-
S2	Right	21	1	-
S3	Right	21	1	-
S4	Right	25	1	-
S5	Right	31	1	-
S6	Right	31	1	-
S7	Left	25	1	-
S8	Right	21	1	50h
S9	Right	23	2	30h
S10	Right	21	2	51h
S11	Right	21	2	50h
S12	Right	22	2	11h
S13	Left	21	2	94h
S14	Right	20	2	11h

Second Experiment

S15	Right	21	3	45h
S16	Right	23	3	48h
S17	Right	21	3	45h
S18	Right	21	3	50h
S19	Right	22	3	50h
S20	Left	21	3	50h
S21	Right	20	3	150h
S22	Right	26	3	150h

Table 1

E list of subjects of the first and second experiments, and their characteristics, right/left-handed, age, group, and number of flight hours.

As explained before, the subjects were divided into three groups: 7 Subjects for the first and second group and 8 subjects for the third. Airforce training pilots composed the second and third groups. These subjects had already performed the movements with a real center stick, pedals, and radio equipment in an aircraft. Consequently, they had to imagine themselves making these movements as they do in the plane. However, CG was instructed how to make the movements before the first session. They had time to practise making the real movements in the Laboratory guided by a supervisor who resolved any uncertainties. After that, they imagined the movement as in the BCI experiment and finally they started the session once they indicated that they were ready. It is important to note that all the participants of the CG have been seated in the pilot cabin of a real airplane touching the controls before the experiment. Table 1 shows all the users, if they are right or left-handed and, the group and the number of flight hours. The table shows that all are young (21–29 years old) and only three of them are left-handed. The flight experience is from 11 hours to 150 hours. Thus, this is not a new movement for them. However, the first group (CG) have never piloted a plane. Therefore, these subjects had to imagine an unnatural movement, because they had never performed it before.

The EEG signals were captured with a StarStim 8 (Neuroelectrics), and 8 monopolar channels located at F3, F4, T7, C3, Cz, C4, T8, and Pz (see Fig. 2) reference and Ground (GND) located at the right mastoid. StarStim is a wireless hybrid EEG/TCS 8-channel neurostimulator system. In this case, dry EEG electrodes were used. The amplifier was configured to acquired the data at 500 Hz with 16 bits. A bandpass filter was also applied to delete power line noise at 50 Hz.

The users were comfortably seated in front of a screen to do the experiments. Figure 3 shows the timing in seconds of the experiments. Each trial starts with a black screen at second 0; then, a cross appears in second 2. At second 4 the system presents an arrow pointing to the left or the right for 1.25 seconds, and the subject must imagine the movement associated with each arrow for a period of 4 seconds. Throughout the first 40 trials the system does not provide feedback to the users, then, after adjusting the CSP parameters and train the LDA classifier, the system shows the feedback as a blue line. At second 8 the screen turns black again. Finally, it is important to note that there is a random time between trials from 0.5 to 2.5 s to avoid adaptation.

As we previously said, in this work, we have introduced novel naturalistic MI movements for aircraft pilots, See Fig. 3. In the first experiment, if the arrow points to the right, the user must imagine the same movements that move the aircraft to the right. It involves the movement of a joystick to the right by the right hand, and also, the movement of the right foot to press a pedal, and to return back the left foot to avoid undesired yaw. If the arrow points left, the user must imagine the same movements that move the aircraft to the left. It involves the movement of a joystick to the left by the right hand, and also, the movement of the left foot to press a pedal and the right foot to return the other pedal to avoid undesired yaw. It requires that the user must imagine the movement of the right hand and feet at the same time depending on the turn direction. Each Subject performs 3 sessions, with one week of separation between sessions. The first session was composed by 2 runs of 40 trials (20 left and 20 right). The first 40 trials were used to train the classifier and to adjust the CSP parameters. Then, for the other 40 trials there was a bar feedback in real time to help the user to improve their BCI response. In the second and the third sessions, there were three runs of 40 trials. This has been used in the first 40 trials to adjust the system and through the other 80 trials the users viewed feedback. Thus, are 80 trials for the first session, 120 trials for the second, and 120 trials for the third session respectively.

The second experiment had a different MI task. If the arrow points to the right, the user must imagine the same movements that move the aircraft to the right as in the first experiment. However, if the arrow points to the left, the user must imagine pressing the radio button with the left hand and then asking for permission to land. This is also shown in Fig. 3. In this experiment subjects also undertook three sessions separated by one week. All sessions were composed by 3 runs of 40 trials (20 left and 20 right). Thus, there are 120 trials per session. In all this sessions the first 40 trials were used to train the system and the others showed a bar feedback to the user.

3.2. Feature Extraction

In BCI it is usual to apply one method to extract features. However, it is possible to implement two or more methods in parallel. In this way, the system could combine and exploit the corresponding information from each procedure and this could improve the classifications results [35]. In this work, the software of the FE module, which was specifically implemented for these experiments, can compute in parallel three well-known FE methods for MI tasks: the band power in two different frequency bands of the EEG data, Common Spatial Pattern (CSP), and the AARs. This section introduces the basic notions of these representative FE methods that, according to Bashashati et al. [36] and Fabian Lotte [37], have been widely applied in many BCI systems for MI tasks. Their software implementations are available in the BioSig toolbox (http://biosig.sourceforge.net/) which is an opensource reference software for biomedical signal processing and, in Lottes's web page (https://sites.google.com/site/fabienlotte/). Note that detailed descriptions of these FE methods are available in [37,38].

PSD features [39]. Four frequency bands are identified for interpretation of the EEG signals [40,41] but, as it is usual in BCI systems, only the most reactive frequency bands for MI have computed [39]: the alpha (8–12 Hz) and beta range (16–24 Hz). Then, for each EEG signal, two BP features are computed as the energies of the alpha and beta bands.
CSP parameters. CSP is a spatial filtering. The purpose of spatial filtering is to combine the signal from several sensors, usually by using a linear combination, to reduce the high number of channels to a small number of virtual channels. The CSP algorithm optimizes coefficients of the filter by making the variance of the EEG signal maximum for one class and minimal for another, which is equivalent to highlighting the differences and minimizing the similarities [37]. Usually, several CSP filters are used as it is possible to determine the coefficients rapidly by using the Generalized Eigen Value Decomposition. In this case for each EEG signal two features are added corresponding to the two largest and lowest eigenvalues of the CSP optimization problem [3]. The spatial filtering is commonly used as the CSP algorithm because it offers great results [37].
Adaptive Autoregressive (AAR) coefficients [42]. The AAR modeling is an evolution of the AR modeling. In AR modeling, each input sample is predicted by a weighted linear combination of the previous p samples, where p denotes the model order. These coefficients are computed using all the samples and get the coefficient vector to predict the next sample. However, in AAR modeling, the coefficient values are continuously updated to get a more accurate estimation. In this work, the AAR coefficients have been estimated using the Kalman filter algorithm. After consulting the literature and based on our previous experimental works [18–20], p was set to 6.

3.3. Dimensionality Reduction of EEG Features

Once features were extracted from the EEG signals, Dimensionality Reduction can be made by Feature Transformation (FT) to a low-dimensional data space [43–46]. In FT approaches, all input variables are required to obtain the reduced feature space. The different FT techniques can be categorized as either unsupervised or supervised depending on whether or not they use class-membership information while computing the r-dimensional space [45]. Unsupervised approaches may not be necessarily useful in classification problems since they do not take the target information into account. The best known and most widely used feature mapping methods is PCA. The following section describes this approach:

3.3.1. Principal component analysis (PCA)

PCA is an unsupervised method that computes a linear mapping to achieve a low-dimensional representation of the original data in which the amount of variance is maximal [46,47]. In general, PCA has been successfully applied in many scientific fields, and it also receives much attention in BCI experiments [20,36,48]. It is widely known that this linear transformation is made by an orthogonal basis from the top eigenvectors (i.e., principal components) of the data covariance matrix, i.e., the eigenvectors corresponding to the largest eigenvalues retained.

3.4. Classifiers

Several classification approaches were tested with EEG signals such as Linear classifiers, SVM, KNN classifiers, or DTs. Linear Classifiers were applied with good results and could be considered as a standard classifier. In this work, four of them have been tested. A description of these classifiers is presented in the following lines:

3.4.1. Linear discriminant analysis

LDA is a very simple classifier that provides acceptable accuracy without high computation requirements. LDA is very common in the BCI community and is a good choice for designing online BCI systems with a rapid response, but limited computational resources. Nevertheless, it can lead to completely erroneous classifications in the presence of outliers or strong noise [2]. For a two-class problem, LDA assumes that the two classes are linearly separable. According to this assumption, LDA defines a linear discrimination function which represents a hyperplane in the feature space to distinguish the classes. The class to which the feature vector belongs will depend on the side of the plane where the vector is found [2].

3.4.2. Support vector machine

SVM is a well-known classification algorithm which can find a decision hyperplane with the maximum distance to the nearest data points of each class. SVM has been used to classify feature vectors for binary and multiclass problems. As a result, this method has a high generalization power. The decision function of SVM is fully specified by a subset of the training data points, which leads to a sparse solution for SVM [49]. The cost function of SVM is a convex function that leads to an optimal solution for the optimization task. The mathematical formulation of SVM gives us the ability to use the kernel trick to map the original finite dimensional space into a destination space with much higher dimensions. Such a classifier is regarded as a linear classifier since it uses one or several hyperplanes. Nevertheless, it is also possible to create an SVM with non-linear decision boundary using a kernel function [2]. In this work, we have used a Linear Kernel.

3.4.3. Decision tree

The DT is used to predict the response by the recursive partition of the instance space. The appealing point of DT are the rules that assist in reaching the final mode. DT's structure is like a tree along with some nodes. Each node can have two possibilities either a leaf or a decision node. It begins from the root just like the tree and reaches the endpoint that is a leaf node, which delivers the classification of the data into a class. In our case, we use Gini's diversity index (GDI) as an optimization criterion [42].

3.4.4. KNN

KNN K-nearest neighbor classifiers (k-NNC) are based on the principle that the features corresponding to the different classes will usually form separated clusters in the feature space, while the close neighbors belong to the same class. This classifier takes k metric distances into account between the test samples features and those of the nearest classes to classify a test feature vector. The metric distances are a measure of the similarities between the features of the test vector and the features of each class [2].

4. EXPERIMENTAL WORK

The experiments aimed to test new MI–BCI task for BCI systems by novice subjects. In this study, EEG signals have been collected as described in Section 2.1. In this work, two strategies to classify the signals have been developed. The first one is to compute the results with standard methods PSD + LDA and CSP + LDA. The second is to adapt the FE and classification stage to the user in each session. Figure 4 shows the Methodology of this work.

In order to adapt the system to the user, three different techniques of features extraction were used: PSD, AAR, and CSP, and also a combination of these. To get the features, and to take in to account the variability of the EEG signals, in the first step, it have been computed from 4 windows of 1 second each trial as in other previous research works [8,17,19,50]. Subsequently, each average feature was placed in a single vector. Thus, there are 10 features from each channel and 80 in each trial. Then, a well-known Feature Reduction approach, PCA, was implemented. Regarding the classifier, the authors used MATLAB Statistics and Machine Learning. MATLAB offers a variety of algorithms to classify the features. After several trials, it was decided to keep the four most relevant classifiers (the others showed poor results) to perform the offline analysis. In this work, LDA, SVM, k-NN, and DT Classifier were used. To carry out fair comparisons with the proposed approach, we used a Leave One Out-Cross Validation procedure for performance evaluation [51], this avoids undesirable shifts from the random selection of training and test sets. It is appropriate since the amount of available data is limited. In this validation technique, the system is trained using all the trials except one and predicts the class of the last remaining trial. 14 subjects (7 Pilots and 7 non-pilots) were used for the first experiment and 8 pilots for the second one and, each subject have done 3 sessions. For each one, the analysis was automatized to allow the computation of all the combinations of features (with or without feature reduction) and classifiers and select the best option for each subject, taking into account the mean value of the three sessions. Thus, 44 combinations were tested for each one to get an adaptation of the system to the user. Therefore, the system is customized to individual users.

4.1. Experimental Results

In both experiments, 44 combinations of “Feature + Feature reduction + Classifier” for all the subjects have been computed. Also, the results using standard methods have been computed. The detailed results are presented in the following section:

4.1.1. First experiment

For the sake of clarity and due to space restrictions Table 2 shows the results of the three sessions of a first experiment for the standard classifier LDA and two of the most used FE methods, CSP, and LDA. These results are a good reference to show the evolution of the users. As was explained before, 80 trials have been evaluated for session 1, and 120 trials for sessions 2 and 3.

	Session 1	Session 1	Session 2	Session 2	Session 3	Session 3
Subject	PSD	CSP	PSD	CSP	PSD	CSP
S1	0.65	0.55	0.51	0.53	0.53	0.61
S2	0.40	0.56	0.45	0.51	0.40	0.62
S3	0.52	0.67	0.51	0.66	0.58	0.60
S4	0.54	0.55	0.37	0.56	0.47	0.62
S5	0.55	0.59	0.54	0.60	0.49	0.54
S6	0.50	0.59	0.55	0.63	0.52	0.69
S7	0.44	0.57	0.53	0.69	0.57	0.64

Average control	0.51	0.58	0.49	0.60	0.51	0.62

S8	0.44	0.61	0.51	0.59	0.64	0.70
S9	0.45	0.59	0.52	0.63	0.77	0.79
S10	0.44	0.50	0.70	0.73	0.82	0.86
S11	0.56	0.62	0.49	0.60	0.55	0.63
S12	0.55	0.50	0.53	0.62	0.61	0.55
S13	0.49	0.46	0.55	0.70	0.47	0.62
S14	0.61	0.65	0.55	0.57	0.62	0.71

Average pilots	0.50	0.58	0.55	0.60	0.64	0.69

Average All	0.51	0.57	0.52	0.62	0.58	0.66

LDA, Linear Discriminant Analysis; FE, Feature Extraction; PSD, Power Spectrum Density; CSP, Common Spatial Pattern.

Table 2

Performance classification accuracy results in % using a LDA classifier and two classical FE methods, PSD, and CSP for the first experiment.

Different articles have presented different thresholds for BCI “efficiency,” but a reference value for acceptable EEG-BCI results is 70% [52–55]. Taking in to account this reference value, results are not high. In Session 1 there are no values higher than 70%, and S3 (67%) gets the best clasification accuracy results using LDA as a classifier and CSP as an FE. In the second session, only S10 surpass 70% using both FE methods. However, in Session 3, S8, S9, S10, and S14 get values over 70%. Therefore, it is evident that there is a learning process of the users.

Then, we performed an automatic system that executed a complete analysis using many features extraction techniques and classification techniques, and we can extract the best results out of these combinations and select the best method for each subject.

Table 3 shows Best Combination of Feature + Feature Reduction + Classifier for each subject. This approach shows results slightly better than using CSP filtering and LDA classification. It is common in BCI systems that algorithms perform differently depending on the subject. Therefore, using these combinations of techniques, we tune the system to the user to get adaptation and to improve the results. With this approach, in the first Session S4 (BP + AAR + PCA + DT), S9 (CSP + AAR + DT), and S13 (BP + PCA + LDA) pass over 70% of classification accuracy with 70%, 70%, and 72%, respectively. In Session 2 S7 (AAR + CSP + PCA + SVM), S10 (AAR + CSP + PCS + SVM), S11 (BP + AAR + CSP), and S13 get 71%, 79%, 70%, and 82%, respectively. Finally in Session 3 S10, S13, and S14 (BP + CSP) get 90%, 75%, and 71%.

Name	FE	FR	Classifier	Session 1	Session 2	Session 3
S1	AAR + CSP	No	SVM	0.55	0.63	0.62
S2	AAR + CSP	PCA	SVM	0.57	0.60	0.61
S3	AAR	No	LDA	0.67	0.66	0.60
S4	PSD + AAR	PCA	Tree	0.70	0.51	0.56
S5	PSD + AAR + CSP	No	LDA	0.60	0.65	0.57
S6	CSP	No	LDA	0.59	0.63	0.69
S7	AAR + CSP	PCA	SVM	0.56	0.71	0.64

Average control	Best M.	Best M.	Best M.	0.61	0.63	0.61

S8	CSP	No	SVM	0.65	0.58	0.67
S9	CSP + AAR	No	Tree	0.70	0.51	0.56
S10	AAR + CSP	PCA	SVM	0.55	0.79	0.90
S11	PSD + AAR + CSP	PCA	SVM	0.69	0.70	0.61
S12	PSD + AAR	No	KNN	0.61	0.54	0.67
S13	PSD	LFDA	LDA	0.72	0.82	0.75
S14	PSD + CSP	No	SVM	0.65	0.58	0.71

Average pilots	Best M.	Best M.	Best M.	0.66	0.66	0.73

All	Best M.	Best M.	Best M.	0.63	0.64	0.67

FE, Feature Extraction; AAR, Adaptive Autoregressive Coefficient; LDA, Linear Discriminant Analysis; KNN, K-Nearest Neighbor; CSP, Common Spatial Pattern; PCA, Principal Component Analysis; SVM, Support Vector Machine; LFDA, Local Fisher Discriminant Analisys; FR, Feature Reduction; PSD, Power Spectrum Density.

Table 3

Classification accuracy results in % of the first experiment in each session, using the best method as a Feature Extraction (FE), the best Feature Reduction (FR) technique, if it improves results, and the best method as a classifier for each subject.

Figure 5 shows a bar diagram, with average values of the First Experiment obtained with LDA as classifier and PSD or CSP as FE approach for CG and PG and obtained with the Best Combination. LDA + PSD shows the lower values for the first and second session for both groups. However, in the third session, LDA + CSP shows the lower level and LDA + PSD the intermediate value for PG. LDA + CSP combination achieves intermediate values for the first and second session, and BC always gains the best values for both groups. The results are quite similar for pilots and non-pilots for the first and seconds sessions. The difference does not look significant, however, for the third session, pilots have improved their results whereas non-pilots results are similar to first and second sessions. To conclude properly on the significance of this difference, the MATLAB function “t-test” was used. The tested hypothesis is that the means of the two samples are equal. The significance level is 5%. Results show that there is no significant difference between PG and CG groups computing the results with LDA + PSD, LDA + CSP, or BC for each subject on the first and second session. In Session 3, the difference in the results is significant for all cases.

4.1.2. Second experiment

For this experiment, the offline analysis was similar as the one performed for the first experiment and the results of the two experiments could be compared. Therefore, we can extract the same two tables as for the first experiment: the first using the PSD and CSP Features, and the LDA classifier, see Table 4 and the second using the best method for each subject, see Table 5.

	Session 1	Session 1	Session 2	Session 2	Session 3	Session 3
Subject	PSD	CSP	PSD	CSP	PSD	CSP
S15	0.55	0.71	0.64	0.69	0.66	0.69
S16	0.63	0.81	0.77	0.83	0.60	0.87
S17	0.37	0.51	0.53	0.54	0.77	0.87
S18	0.42	0.62	0.53	0.54	0.63	0.64
S19	0.65	0.67	0.55	0.65	0.39	0.74
S20	0.71	0.73	0.80	0.86	0.91	0.95
S21	0.45	0.60	0.31	0.47	0.57	0.57
S22	0.50	0.58	0.57	0.74	0.47	0.73

Average control	0.53	0.65	0.58	0.67	0.62	0.76

LDA, Linear Discriminant Analysis; FE, Feature Extraction; PSD, Power Spectrum Density; CSP, Common Spatial Pattern.

Table 4

Peformance classification accuracy results in % using a LDA classifier and two clasical FE methods, PSD, and CSP for the second experiment.

Name	FE	FR	Classifier	Session 1	Session 2	Session 3
S15	CSP	No	SVM	0.75	0.69	0.69
S16	CSP	No	LDA	0.81	0.83	0.87
S17	PSD + CSP	No	Tree	0.64	0.62	0.91
S18	PSD + CSP	No	SVM	0.60	0.57	0.67
S19	CSP	No	Ensemble	0.72	0.74	0.78
S20	CSP	No	Tree	0.89	0.87	0.95
S21	AAR + CSP	PCA	LDA	0.56	0.57	0.55
S22	CSP	No	LDA	0.58	0.74	0.73

Average	Best M.	Best M.	Best M.	0.69	0.71	0.77

LDA, Linear Discriminant Analysis; FE, Feature Extraction; PSD, Power Spectrum Density; CSP, Common Spatial Pattern; AAR, Adaptive Autoregressive Coefficient; SVM, Support Vector Machine.

Table 5

Classification accuracy results in % of the second experiment in each session, using the best method as a FE, a feature reduction, if it improves results, and the best method as a classifier for each subject.

In this experiment results appear higher. Looking at Table 4 In Session 1 S15 (71%) S16 (81%), S20 (71% and 73%) achieves values higher than 70%. In Session 2 S16(87%), S20 (77% and 83%), and S22 (74%) gets overpass 0.70%. Finally, in Session 3 S16(87%), S17(77% and 87%), S19 (74%), S20 (91% and 95%), and S22 (73%) surpasses 70%. Then, the system is tuned and it extracts the best results selecting the BC for each subject.

Table 5 shows BC results for the each subject. This approach shows results slightly better than using CSP filtering and LDA classification. With this approach, in the first Session S15, S16, S19, S20 exceeds 70% of classification accuracy with 75%, 81%, 72%, and 89%, respectively. In Session 2 S16, S19, S20, and S22 reach 83%, 74%, 87%, and 74%, respectively. Finally in Session 3 S16, S17, S19, S20, and S22 achieve 87%, 91%, 78%, 95%, and 73%, respectively.

Figure 6 presents a graph with average results using LDA + PSD, LDA + CSP, and BC of the second experiment. As in the first experiment: BandPower provides low results whereas CSP based results are closer to the result using the BC for each subject. Also, we observe that the results improve in the second and third sessions.

Figure 7 is a graph bar picture to observe the differences of results between the two experiments. It shows that the results of the second experiment are higher than those in the first experiment. On the third session, the average result is above 77%. Statistical comparison of the results between first and second experiment shows that for all methods the test concludes that the results are significantly higher for the second experiment. Therefore, a BCI system based on the movements from the second experiment is easier for pilots to control.

5. CONCLUSIONS

To explore new MI tasks for Aircraft Pilots, in this work, two experiments were developed with novice users in BCI experiments. In these experiments, standard methods have been used and to tune the system to the user, 3 FE methods (PSD, AAR, and CSP) a feature reduction technique, PCA and, 4 classifiers (LDA, SVM, K-NN and DT) and all combinations of these have been applied. The first experiment collected signals from 14 novice BCI users, 7 of them without any experience as aircraft pilots (as CG), following new MI commands to make a left or right movement of a plane. Results show that users with previous flight experience are better fitted on the third session than non-pilots to control a BCI system based on these imagery tasks. However, this result was obtained on the third session. For the first and second sessions, it was not the case, and there is no significant difference between groups. Therefore, it appears that pilots can control the system faster, but to do that, they need to be trained to use the system properly.

The second experiment collected signals from 8 aircraft pilots doing MI of the right movement and asking permission to land. It shows that pilots controlled the system using speech on the left arrow and the turn to the right on the right arrow better than with turns on both sides. We observe that several subjects achieve very high scores (S16 87%, S17 91%, and S20 95%): these subjects could control a system based on this MI paradigm. Thus, these are more discriminative tasks.

Finally, these experiments showed that for now, to control a system using precise and complicated movements is a difficult task. It can be noted that our experiment only considered discriminating two different movements, however controlling a plane involves more movements than simply 2: the pilot must control the stick to move it from left to right, front to back and also the pedals, the thrust and has to manage the radio and many other systems. Nevertheless, these experiments have shown that we could imagine a new interface providing other methods for conducting experiments or providing an output to control a system using MI tasks detected through EEG recorded signals.

In particular as a future work, we would like to do more training sessions to evaluate the evolution of the results. Also, as a more ambitious experiment, we would like to develop a system that allows airplane pilots to control drones using naturalistic movements with optimized MI paradigms.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

AUTHORS' CONTRIBUTIONS

This statement is to certify that all Authors have seen and approved the manuscript being submitted. We warrant that the article is the Authors' original work. We warrant that the article has not received prior publication and is not under consideration for publication elsewhere. On behalf of all Co-Authors, the corresponding Author shall bear full responsibility for the submission.

This research has not been submitted for publication nor has it been published elsewhere. We attest to the fact that all Authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission to the International Journal of computational Intelligence Systems.

Funding Statement

This work proposes new naturalistic and intuitive motor imagery (MI) BCI task for aircraft Pilots for a BCI System, and explore if they take advantage of their previous motor experience in MI-BCI experiments. Results show that Pilots get better performance than a control group and that it is possible to get good classification accuracy results with new naturalistic MI tasks.

This work was partially supported by the Seneca Foundation [20783/PI/18] and by Spanish Ministry of Science, Innovation and Universities [PGC2018-097198-B-I00] and we declare that.

ACKNOWLEDGMENTS

This work was partially supported by the Seneca Foundation [20783/PI/18] and by Spanish Ministry of Science, Innovation and Universities [PGC2018-097198-B-I00].

REFERENCES

1.H. Berger, Über das Elektroenkephalogramm des Menschen, Arch. Psychiatr. Nervenkr., Vol. 87, 1929, pp. 527-570.

2.L. Nicolas-Alonso and J. Gomez-Gil, Brain computer interfaces, a review, Sensors, Vol. 12, 2012, pp. 1211-1279.

3.L. Bougrain and F. Lotte, Brain-Computer Interfaces 1: Methods and Perspectives, Wiley, 2016.

4.M.N. Teninson and J.D. Moreno, Neuroscience, ethics, and national security: the state of the art, PLoS Biol., Vol. 10, 2012, pp. e1001289.

5.J.R. Wolpaw, N. Birbaumer, D.J. McFarland, G. Pfurtscheller, and T.M. Vaughan, Brain computer interfaces for communication and control, Clin. Neurophysiol., Vol. 113, 2002, pp. 767-791.

6.J.R. Wolpaw et al., BCI meeting 2005–workshop on signals and recording methods, IEEE Trans. Neural Syst. Rehabil. Eng., Vol. 14, 2006, pp. 138-141.

7.C. Guger, G. Edlinger, W. Harkam, I. Niedermayer, and G.A. Pfurtscheller, How many people are able to operate an EEG-based Brain-Computer Interface (BCI)?, IEEE Trans. Neural Syst. Rehabil. Eng., Vol. 11, 2003, pp. 145-147.

8.G. Rodríguez-Bermúdez, A. Bueno-Crespo, and F. Martinez-Albaladejo, Classifying BCI signals from novice users with extreme learning machine, Open Phys., Vol. 15, 2017, pp. 494-500.

9.C. Vidaurre and B. Blankertz, Towards a cure for BCI illiteracy, Brain Topogr., Vol. 23, 2010, pp. 194-198.

10.F. Lotte, F. Larrue, and C. Muhl, Flaws in current human training protocols for spontaneous brain-computer interfaces: lessons learned from instructional design, Front. Hum. Neurosci., Vol. 7, 2013, pp. 568.

11.E.V.C. Friedrich, C. Neuper, and R. Scherer, Whatever works: a systematic user-centered training protocol to optimize brain-computer interfacing individually, PLoS ONE, Vol. 8, 2013, pp. e76214.

12.B. He, B. Baxter, B.J. Edelman, C.C. Cline, and W.W. Ye, Noninvasive brain-computer interfaces based on sensorimotor rhythms, Proc. IEEE, Vol. 103, 2015, pp. 907-925.

13.R. Chaudhari and H. Galiyawala, A review on motor imagery signal classification for BCI, Signal Process Int. J., Vol. 11, 2017, pp. 16-34.

14.S.K. Bashar, A. Rashik Hassan, and M. Imamul Hassan Bhuiyan, Motor imagery movements classification using multivariate EMD and short time Fourier transform, in Annual IEEE India Conference (INDICON) (New Delhi), 2015, pp. 1-6.

15.R. Chatterjee and T. Bandyopadhyay, EEG based motor imagery classification using SVM and MLP, in IEEE International Conference on Computational Intelligence and Networks (CINE) (Bhubaneswar), 2016, pp. 84-89.

16.R. Chatterjee, T. Bandyopadhyay, and D. Sanyal, Effects of wavelets on quality of features in motor imagery EEG signal classification, in IEEE International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET) (Chennai), 2016, pp. 1346-1350.

17.M.S. Santos, P.H. Abreu, G. Rodriguez-Bermudez, and P. García-Laencina, Improving the classifier performance in motor imagery task classification: what are the steps in the classification process that we should worry about?, Int. J. Comput. Intell. Syst., Vol. 11, 2018, pp. 1278-1293.

18.P.J. Garca-Laencina, G. Rodriguez-Bermudez, and J. Roca-Dorda, Exploring dimensionality reduction of EEG features in motor imagery task classication, Expert Syst. Appl., Vol. 41, 2014, pp. 5285-5295.

19.G. Rodríguez-Bermúdez, J. Roca-González, and J. Roca-Dorda, Efficient feature selection and linear discrimination of EEG signals, Neurocomputing, Vol. 115, 2013, pp. 161-165.

20.G. Rodríguez-Bermúdez, P.J. García Laencina, and J. Roca-Dorda, Efficient automatic selection and combination of EEG features in least square classifiers for motor imagery brain–computer interfaces, Int. J. Neural Syst., Vol. 23, 2013, pp. 1350015.

21.C.-Y. Chen, C.-W. Wu, C.-T. Lin, and S.-A. Chen, A novel classification method for motor imagery based on brain-computer interface, in IEEE International Joint Conference on Neural Networks (IJCNN) (Beijing), 2014, pp. 4099-4102.

22.N. Prakaksita, C.-Y. Kuo, and C.-H. Kuo, Development of a motor imagery based brain-computer interface for humanoid robot control applications, in IEEE International Conference on Industrial Technology (ICIT) (Taipei), 2016, pp. 1607-1613.

23.S. Agarwal, S. Shah, and R. Kumar, Classification of mental tasks from EEG data using back tracking search optimization based neural classifier, Neurocomputing, Vol. 166, 2015, pp. 397-403.

24.M. Hamedi, S.-H. Salleh, A.M. Noor, and I. Mohammad-Rezazadeh, Neural network-based three-class motor imagery classification using time domain features for BCI applications, in IEEE Region 10 Symposium (Kuala Lumpur), 2014, pp. 204-207.

25.L.Q. Thang and C. Temiyasathit, Increase performance of four-class classification for motor-imagery based brain-computer interface, in IEEE International Conference on Computer, Information and Telecommunication System (CITS) (Jeju), 2014, pp. 1-5.

26.D. Xiao, Z. Mu, and J. Hu, Classification of motor imagery EEG signals based on energy entropy, in IEEE International Symposium on Intelligent Ubiquitous Computing and Education (Chengdu), 2009, pp. 61-64.

27.H.S. Kim, M.H. Chang, H.J. Lee, and K.S. Park, A comparison of classification performance among the various combinations of motor imagery tasks for brain-computer interface, in 6th IEEE Annual International EMBS Conference of Neural Engineering (San Diego), 2013, pp. 435-438.

28.S. Bhattacharyya, M. Pal, A. Konar, and D.N. Tibarewala, An interval type-2 fuzzy approach for real-time EEG-based control of wrist and finger movement, J. Biomed. Signal Process. Control, Vol. 21, 2015, pp. 90-98.

29.A. Akce, M. Johnson, and T. Bretl, Remote teleoperation of an unmanned aircraft with a brain-machine interface: theory and preliminary results, in IEEE International Conference on Robotics and Automation (Anchorage), 2010, pp. 5322-5327.

30.A. Nourmohammadi, M. Jafari, and T.O. Zander, A survey on unmanned aerial vehicle remote control using brain–computer interface, IEEE Trans. Hum. Mach. Syst., Vol. 48, 2018, pp. 337-348.

31.A.S. Royer, A.J. Doud, M.L. Rose, and B. He, EEG control of a virtual helicopter in 3-dimensional space using intelligent control strategies, IEEE Trans. Neural Syst. Rehabil. Eng., Vol. 18, 2010, pp. 581-589.

32.K. LaFleur, K. Cassady, A. Doud, K. Shades, E. Rogin, and B. He, Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain-computer interface, J. Neural Eng., Vol. 10, 2013, pp. 046003.

33.T. Fricke, T.O. Zander, K. Gramann, and F. Holzapfel, First pilot-in-the-loop simulator experiments on brain control of horizontal aircraft motion, in Deutscher Luft- und Raumfahrtkongress, 2014 DGLR e.V. (Bonn), 2014.

34.M. Kryger, B. Wester, E.A. Pohlmeyer, M. Richb, B. John, J. Beaty, M. McLoughlinb, M. Boninger, and C.T. Tyler-Kabara, Flight simulation using a brain-computer interface: a pilot, pilot study, Exp. Neurol., Vol. 287, 2017, pp. 473-478.

35.G. Rodríguez-Bermúdez and P. García-Laencina, Analysis of EEG signals using nonlinear dynamics and chaos: a review, Appl. Math. Inf. Sci., Vol. 9, 2015, pp. 2309-2321.

36.A. Bashashati, M. Fatourechi, R.K. Ward, and G.E. Birch, A survey of signal processing algorithms in brain-computer interfaces based on electrical brain signals, J. Neural Eng., Vol. 4, 2007, pp. 32-57.

37.F. Lotte and C. Guan, Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms, IEEE Trans. Biomed. Eng., Vol. 58, 2011, pp. 355-362.

38.S. Tong and N.V. Thankor, Quantitative EEG Analysis Methods and Clinical Applications, Artech House, Boston and London, 2009.

39.G. Pfurtscheller, C. Brunner, A. Schlögl, and F.L. da Silva, Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks, Neuroimage, Vol. 31, 2006, pp. 153-159.

40.D. Sanei and J. Chambers, EEG Signal Processing, John Wiley & Sons, Inc., Hoboken, 2008.

41.M. Sabeti, R. Boostani, S. Katebi, and G. Price, Selection of relevant features for EEG signal classification of schizophrenic patients, Biomed. Signal Process. Control, Vol. 2, 2007, pp. 122-134.

42.A. Ishfaque, A.J. Awan, N. Rashid, and J. Iqbal, Evaluation of ANN, LDA and decision trees for EEG based brain computer interface, in 2013 IEEE 9th International Conference on Emerging Technologies (ICET) (Islamabad), 2013, pp. 1-6.

43.K. Fukunaga, Introduction to Statistical Pattern Recognition, Computer Science and Scientific Computing, Academic Press, 1990.

44.I. Guyon and A. Elisseeff, An introduction to variable and feature selection, J. Mach. Learn. Res., Vol. 3, 2013, pp. 1532-4435.

45.L. Van der Maaten, E. Postma, and J. Van den Herik, Dimensionality reduction: a comparative review, Tilburg University, 2009. Technical Report

46.E. Alpaydin, Introduction to Machine Learning, MIT Press, Cambridge, 2010.

47.R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification, Wiley Interscience, 2000. https://www.wiley.com/en-es/globallocations

48.J.A. Lee and M. Verleysen, Nonlinear Dimensionality Reduction, Springer, New York, 2007.

49.H. Bashashati, R.K. Ward, G.E. Birch, and A. Bashashati, Comparing different classifiers in sensory motor brain computer interfaces, PLoS ONE, Vol. 10, 2015, pp. e0129435.

50.A. Cabrera, D. Farina, and K. Dremstrup, Comparison of feature selection and classification methods for a brain-computer interface driven by non-motor imagery, Med. Biol. Eng. Comput., Vol. 48, 2010, pp. 123-132.

51.G. Rodríguez-Bermúdez and P.J. García-Laencina, Automatic and adaptive classification of electroencephalographic signals for brain computer interfaces, J. Med. Syst., Vol. 36, 2012, pp. 51-63.

52.A. Minkyu and J. Sung Chan, Performance variation in motor imagery brain–computer interface: a brief review, J. Neurosci. Methods, Vol. 243, 2015, pp. 103-110.

53.A. Kubler, N. Neumann, B. Wilhelm, T. Hinterberger, and N. Birbaumer, Predictability of brain–computer communication, J. Psychophysiol., Vol. 18, 2004, pp. 121-129.

54.B.Z. Allison, S. Dunne, R. Leeb, J.R. Del Millan, and A. Nijholt, Towards Practical Brain-Computer Interfaces: Bridging the Gap from Research to Real-World Applications, Springer, Berlin, Heidelberg, 2013.

55.Z. Qiu, B.Z. Allison, J. Jin, Y. Zhan, X. Wang, W. Li, and A. Cichoki, Optimized motor imagery paradigm based on imagining Chinese characters writing movement, IEEE Trans. Neural Syst. Rehabil. Eng., Vol. 25, 2017, pp. 1009-1017.

<Previous Article In Issue

Download article (PDF)

Next Article In Issue>

Journal: International Journal of Computational Intelligence Systems
Volume-Issue: 12 - 2
Pages: 937 - 946
Publication Date: 2019/08/23
ISSN (Online): 1875-6883
ISSN (Print): 1875-6891
DOI: 10.2991/ijcis.d.190806.001 How to use a DOI?
Open Access: This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

Cite this article

ris enw bib

TY  - JOUR
AU  - German Rodriguez-Bermudez
AU  - Alejandro Lopez-Belchi
AU  - Arnaud Girault
PY  - 2019
DA  - 2019/08/23
TI  - Testing Brain–Computer Interfaces with Airplane Pilots under New Motor Imagery Tasks
JO  - International Journal of Computational Intelligence Systems
SP  - 937
EP  - 946
VL  - 12
IS  - 2
SN  - 1875-6883
UR  - https://doi.org/10.2991/ijcis.d.190806.001
DO  - 10.2991/ijcis.d.190806.001
ID  - Rodriguez-Bermudez2019
ER  -

download .riscopy to clipboard