IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE 1
Electroencephalogram Based Reaction Time
Prediction With Differential Phase Synchrony
Representations Using Co-Operative
Multi-Task Deep Neural Networks
Tharun Kumar Reddy, Vipul Arora , Satyam Kumar, Laxmidhar Behera , Senior Member, IEEE,
Yu-Kai Wang, Member, IEEE, and Chin-Teng Lin, Fellow, IEEE
Abstract—Driver drowsiness is receiving a lot of deliberation
as it is a major cause of traffic accidents. This paper proposes
a method which utilizes the fuzzy common spatial pattern op-
timized differential phase synchrony representations to inspect
electroencephalogram (EEG) synchronization changes from the
alert state to the drowsy state. EEG-based reaction time predic-
tion and drowsiness detection are formulated as primary and
ancillary problems in the context of multi-task learning. Statis-
tical analysis results suggest that our method can be used to dis-
tinguish between alert and drowsy state of mind. The proposed
Multi-Task DeepNet (MTDNN) performs superior to the baseline
regression schemes, like support vector regression (SVR), least ab-
solute shrinkage and selection operator, ridge regression, K-nearest
neighbors, and adaptive neuro fuzzy inference scheme (ANFIS), in
terms of root mean squared error (RMSE), mean absolute per-
centage error (MAPE), and correlation coefficient (CC) metrics.
In particular, the best performing multi-task network MTDNN
5
recorded a 15.49% smaller RMSE, a 27.15% smaller MAPE, and
a 10.13% larger CC value than SVR.
Index Terms—Brain Computer Interface (BCI), Deep Neural
Network (DNN), One Versus Rest (OVR), Reaction Time (RT),
Multi-Task Learning, Root Mean Squared Error (RMSE), Mean
Absolute Percentage Error (MAPE) and Correlation Coefficient
(CC).
Manuscript received July 15, 2018; revised September 3, 2018 and Septem-
ber 24, 2018; accepted October 5, 2018. This work was supported in part by
the Australian Research Council (ARC) under discovery Grants DP180100670
and DP180100656, in part by the Ministry of Human Resource Development
(MHRD), Government of India, under Project MHRD-EE-2016150, and in part
by the Army Research Laboratory and was accomplished under Cooperative
Agreement Numbers W911NF-10-2-0022 and W911NF-10-D-0002/TO 0023.
The work of T. K. Reddy was supported under Project TCS/CS/2011191A.
(Corresponding author: Laxmidhar Behera.)
T. K. Reddy, V. Arora, S. Kumar, and L. Behera are with the Department of
Electrical Engineering, Indian Institute of Technology Kanpur, Kanpur 208016,
India (e-mail:, tharun@iitk.ac.in; vipular.iitk@gmail.com; satyamk@iitk.ac.in;
lbehera@iitk.ac.in).
Y.-K. Wang and C.-T. Lin are with the Centre for Artificial Intelligence,
Faculty of Engineering and Information Technology, University of Technol-
ogy Sydney, Ultimo, NSW 2007, Australia (e-mail:, yukai.wang@uts.edu.au;
chintenglin@gmail.com).
This paper has supplementary downloadable material available at
http://ieeexplore.ieee.org, provided by the authors. This includes two PDFs
that show a more detailed performance analysis of results and implementation
details comparing performance of the features and a computational complexity
analysis of the proposed approach. This material is 467 KB in size.
Digital Object Identifier 10.1109/TETCI.2018.2881229
I. INTRODUCTION
D
ROWSINESS (fatigue), also mentioned as sleepiness, sig-
nifies “the tendency to fall asleep”. A major shift leading
to the deployment of drowsiness detection systems in vehicles
is largely attributed to the WHO report in 2013, which stated
that almost almost 6% of world’s road accidents are caused
by drivers in drowsy state [1]. Some recent developments in
drowsiness detection research are discussed below.
Among various physiological signals, EEG is one of the most
reliable indicators because it is in close conjunction with mental
and physical activities [2]. Several methods proposed in lit-
erature for detecting fatigue by EEG can be categorized into
amplitude and phase based approaches. At first, we summarize
the amplitude based approaches. Budi et al. [3] used EEG spec-
tral feature segments to analyze four algorithms for detecting fa-
tigue. They demonstrated that the r atio of the total spectra power
in theta and alpha bands to the power in beta band witnessed
a greater upsurge over time with drowsiness phenomenon. Wei
et al. [4] defined a term, the Level of Session Generalizability
(LSG) through a novel Transfer Learning (TL) based method.
Their method utilizes a subjects pilot data to select ancillary
data from other subjects to enhance the performance of an EEG
based BCI for drowsiness detection. Authors in [5] propose
regression with Random Forest on multiband power features
providing a highly accurate fatigue index using only three elec-
trode positions. [6] contains two novel driving fatigue prediction
metrics. First integrated fatigue metric is based on power spec-
trum density analysis with subject specific channel selection and
second metric is based on sample entropy analysis from ‘O1h’
and ‘O2h’ electrodes. A. Saha et al. [7] used motor planning
phase to detect cognitive failure in driving using type-2 fuzzy
classifiers.
The phase based approaches for detecting drowsiness are dis-
cussed next. Phase based analysis has also been demonstrated
to be the cynosure of functional neural connectivity inference
[8]. Phase Synchronization (PS) or phase based analysis can de-
tect the spatial lateralization of drowsiness phenomenon. This
approach studies the interplay between signal pairs through the
functional relationships of instantaneous phase among the sig-
nals independent of their amplitudes. Thus, the notion of PS [9]
2471-285X © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications
standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Technology Sydney. Downloaded on March 20,2020 at 00:39:55 UTC from IEEE Xplore. Restrictions apply.
2 IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE
is fundamental for neuronal information processing within the
brain region as well as communication between different brain
regions. Lachaux et al. [10] proposed Phase Locking Value
(PLV) statistic to quantify the frequency specific synchroniza-
tion between the neural signals. However, since PLV is a tem-
poral measure of synchrony across the trials, it is not suitable
for trial wise analysis. To measure trial wise phase synchrony
information, a new statistical single trial PLV was formulated
[11]. Caramia et al. [12] proposed a modified single trial PLV
but it did not account for lead lag behavior between EEG signals.
Kumar et al. [13] proposed to extract features based on Instan-
taneous Phase Difference (IPD) for trial wise analysis. Further,
[14] explored Mean Phase Coherence (MPC) with large and
local scale synchrony for fatigue detection. In this work, au-
thors adapt the DPS representations for regression which are
an outcome of integration of IPD with fuzzy CSPR-OVR [15]
framework. Authors further validate the proposed DPS features
on the reaction time dataset from an EEG based lane-keeping
task.
Thus, a large amount of literature already deals with signal
processing for BCIs based on EEG classification problems, but
research on EEG regression problems is mostly neglected till
now. Some of the prominent EEG based regression problems
are: estimation of continuous workload levels [16], Reaction
Time (RT) for the EEG-tracked SVIPT [17] and RT prediction
for the EEG based PVT and lane-keeping tasks [ 18].
After the EEG signal is obtained, the regression process in-
volves several steps: 1) Signal pre-processing to enhance the
Signal to noise ratio (SNR). Filters in frequency realm such as
low-pass filters, band-pass filters, band-stop filters, and spatial
filters such as independent component analysis (ICA), SPoC
[19], CSP and fuzzy CSSP [20] are frequently used here. 2)
Feature representations to shape relevant predictors, e.g., Rie-
mannian Geometry features [18] and EEG power band features
[15]. 3) Regressors to estimate the continuous valued variable,
e.g., Ordinary Linear Regression (OLR) [21], Ridge Regression
[22], LASSO [15], [18], K-nearest neighbors (kNN) [15], fuzzy
neural networks [23], Transfer Learning [4], [24], [25], active
learning [26], domain adaptation [27], multiview learning [28],
multitask learning [29] etc.
In the multitask learning setup, there are multiple tasks, each
of which is a general task such as supervised task, unsupervised
task, semi-supervised task etc. A handful of these tasks, or a
portion of them are related to each other. Cooperative training
among these tasks can lead to a greater performance enhance-
ment compared to training the tasks one at a time [30]. In BCIs,
ataskis usually considered a unique recording session, ei-
ther for an individual or multiple subjects [ 31]. In [32], authors
demonstrated the effectiveness of multitask learning for classifi-
cation of motor imagery trials. The multitask learning approach
proved to be robust against misclassification in different exper-
imental conditions. Furthermore, multitask learning technique
efficiently estimates spatial filters for classification of motor
imagery in subjects with no prior training data.
RT denotes the time period between the onset of the lane de-
viation and the onset of the response and is used as an objective
measure of the drowsiness (DS) level during each lane departure
event [33]. Since EEG drowsiness estimation and EEG reaction
time prediction are two related problems, where approximate
solution to either of them helps to solve the other, Hence, mul-
titask learning can be applied in this scenario.
The primary objectives of this study are:
r
To demonstrate the utility of phase based feature represen-
tations for EEG based drowsiness detection.
r
To model the drivers’ drowsiness detection as a multi-task
learning problem based on DPS-FCSPR-OVR feature rep-
resentations and train intelligent models for the proposed
tasks.
The major contributions of this work are:
r
Novel DPS-FCSPR-OVR representations are proposed to
demonstrate the utility of phase based EEG representations
for EEG based RT prediction.
r
A novel MTDNN framework with a supervised pre-
training and fine tuning steps is proposed.
r
Extensive experiments (including comparison with ad-
vanced regression models) are carried out to confirm the
effectiveness of the proposed method in EEG based RT
prediction.
The novelty of the proposed method is highlighted as follows:
r
The utility of phase based feature representations is very
scarcely studied in literature for EEG based drowsiness
detection problem. For addressing this issue, the DPS-
FCSPR-OVR representations are adopted to train intelli-
gent models.
r
In the multitask BCI literature, the concept of task is usu-
ally limited to either a subject, session etc. For the first
time, we extend the notion of BCI task to address two BCI
problems: drowsiness detection and RT prediction.
This paper is organized as follows: Section II presents the
formulation of proposed DPS-FCSPR-OVR representations.
Section III evaluates their performance on the EEG lane keep-
ing task. Section IV leads us to the proposed co-operative DNN
based multitask approach (MTDNN). Section V evaluates the
proposed MTDNN approach with several baseline DeepNets
and other regression schemes. Finally, discussion and conclud-
ing remarks are provided in Section VI and VII respectively.
II. D
IFFERENTIAL PHASE SYNCHRONY (DPS)
R
EPRESENTATIONS
A. Fuzzy CSPR-OVR
Let X
n
∈ R
C ×T
n ∈{1, 2,...,N} denote the nth EEG trial,
where C denotes number of channels and T denotes number of
time samples per trial. Trial X
n
constitutes a band pass filtered
signal whose mean is removed from each of the channels. Using
the concept of fuzzy sets, we define fuzzy classes (assume M
fuzzy classes) to generalize to regression problems. Next, the
interval range [0, 100] is used to generate M +1regions and
let us denote the boundary partition points by {y
p
m
},m=
{1,...,M}. To mention
P
m
=
100 · m
M +1
,m=1,...,M (1)
Each y
p
m
is the P
m
percentile value of the training set of
RTs. Next, we define M fuzzy classes and categorize training RT
values into one of the M fuzzy classes, in a manner analogous to
Authorized licensed use limited to: University of Technology Sydney. Downloaded on March 20,2020 at 00:39:55 UTC from IEEE Xplore. Restrictions apply.
REDDY et al.: EEG BASED RT PREDICTION WITH DIFFERENTIAL PHASE SYNCHRONY REPRESENTATIONS 3
Fig. 1. ‘M’ fuzzy classes for training RT values using triangular fuzzy mem-
bership.
M classes in a multiclass classification scenario. For each fuzzy
class, a y
n
can belong to it at a membership degree ∈ [0, 1].
Further, we compute an average covariance matrix for each
fuzzy class as:
˜
Σ
m
=
N
n=1
μ
m
(y
n
)X
n
X
n
N
n=1
μ
m
(y
n
)
,m=1,...,M (2)
where μ
m
(y
n
) is the membership degree of y
n
in fuzzy class m.
We now mention the One-Versus-Rest (OVR) CSP to extend
the CSP from binary classification to M classes. For class m,
OVR-CSP finds a matrix W
∗
m
∈ R
C ×F
, where F is number of
spatial filters to maximize variance of class m against rest.
W
∗
m
= arg max
W
Tr
W
˜
Σ
m
W
Tr
W
j=m
˜
Σ
j
W
(3)
˜
Σ
m
is the mean covariance matrix of trials in class m. W
∗
m
is
the concatenation of the F eigenvectors associated with the F
largest eigenvalues of the matrix (
i=m
˜
Σ
i
)
−1
˜
Σ
m
. We con-
catenate the obtained F filters for each of the M classes to ob-
tain W
∗
=[W
∗
1
,...W
∗
M
] ∈ R
C ×MF
. Then, one can compute
a spatially transformed trial by X
n
= W
∗
X
n
,n=1,...,N.
B. Phase Locking Value
PLV is a statistic used to investigate synchronization of neural
activity from EEG data and expresses a transient measure of
connectivity. Any cognitive task results due to combination of
various functional areas distributed over different regions of
the brain. The task induced coupling between these areas is
interpreted as synchronization of neural activity. PLV and its
variants are common measures of phase synchronization.
Consider two signals x
1
(t) and x
2
(t) whose instantaneous
phases are ψ
1
(t) and ψ
2
(t). In accordance with [12] the Single
trial PLV (sPLV) for a given trial can be defined as follows:
sPLV =
1
N
s
N
s
t=1
e
|ψ
1
(t)−ψ
2
(t)|
(4)
where N
s
is the number of samples in the trial. The instantaneous
phase ψ(t) can be obtained using the analytic signal calculated
from Hilbert transform. For any arbitrary signal x(t) the analytic
signal z(t) is given as
z(t)=x(t)+i˜x(t) (5)
˜x(t)=
1
π
∞
−∞
x(τ)
t − τ
dτ (6)
where ˜x(t) is the Hilbert transform of x(t). The instantaneous
phase ψ(t) is then calculated using
ψ(t) = arctan
˜x(t)
x(t)
(7)
The sPLV holds a value between 0 and 1 with extremas cor-
responding to the cases of signal being unsynchronized and
completely synchronized respectively.
C. Differential-Phase Synchrony Representations
Instantaneous Phase Difference (IPD) sequence Δψ(t) be-
tween a pair of distinct signals s
1
(t) and s
2
(t) is defined as
Δψ(t)=|ψ
1
(t) − ψ
2
(t)| (8)
Authors in [13] coupled the notion of variance of instanta-
neous phasors with sPLV. Furthermore, they formulated a frame-
work to estimate a linear transform that maximizes the variance
of instantaneous phasors across one class while simultaneously
minimizing it across the other class. The proposed framework
is similar to CSP but, in contrary, it explicitly uses phase in-
formation for binary classification. Thus, drawing an analogy
from the regression CSP algorithm (Fuzzy CSPR-OVR), a novel
framework is formulated to find a linear transform on the IPD
sequence in such a way that it maximizes the variance across
the temporal phase differences of each fuzzy class and simulta-
neously minimizes the variance of phases across the other fuzzy
class. As the IPD sequence is used to measure synchrony be-
tween EEG signals, we refer to the extracted features as DPS
representations.
1) Fuzzy Spatial Filter Optimization: Fuzzy CSPR-OVR
[15] extends multiclass CSP to regression problems using fuzzy
sets. Indeed, as mentioned before, the EEG signals from indi-
vidual channels are prone to have a low SNR, due to spatial
blurring and smearing effects. In order to obtain more discrim-
inative DPS features for EEG classification of fuzzy classes, it
thus seems prudent to compute spatial filters which maximize
the variance of instantaneous phase across a particular fuzzy
class and minimize across rest of them. We therefore propose
an algorithm to optimize the spatial filters in order to maximize
the resulting DPS feature discriminative power. Mathematically,
it can be written as
W
i
∗
= arg max
W
Tr(W
Σ
Δψ
i
W)
Tr
W
j= i
Σ
Δψ
j
W
(9)
where Σ
Δψ
i
and Σ
Δψ
j
are the covariance matrices of the IPD
sequence for the fuzzy classes i and j. The column vectors of W
are the spatial synchrony filters. A scheme similar to fuzzy CSP
algorithm is employed for feature extraction from IPD sequence
of an EEG trial in a given frequency range. The obtained features
are coined as ‘DPS’ repersentations.
III. E
VA L UAT I O N O F DIFFERENTIAL-PHASE SYNCHRONY
REPRESENTATIONS FOR REACTION TIME PREDICTION
The proposed feature representations are evaluated and com-
pared to other phase based baseline methods on an EEG based
reaction time (RT) prediction in a lane-keeping task.
Authorized licensed use limited to: University of Technology Sydney. Downloaded on March 20,2020 at 00:39:55 UTC from IEEE Xplore. Restrictions apply.
4 IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE
TABLE I
N
UMBER OF TRIALS (NTRIALS) AND MEAN RT IN THE DATAS E T
A. Lane-Keeping Task
The EEG signals were recorded from 30 active electrode
sites which were placed according to modified international
10–20 electrode montage system.
The Institutional Review Board of the Veterans General Hos-
pital, Taipei, Taiwan, approved the study. A total of 12 university
students (average age 22.4, standard deviation 1.6) from the Na-
tional Chiao Tung University (NCTU) in Taiwan volunteered to
support the data-collection efforts over a five-month period to
study EEG correlates of attention and performance changes un-
der specific conditions of real-world drowsiness [34].
Simulated driving experiments were conducted on a vir-
tual reality (VR)-based dynamic driving simulator. A real car
frame was mounted on a six degree-of-freedom Stewart motion
platform which moved in sync with the driving scene during
“motion” sessions. The motion platform was inactive during
“motionless” sessions. The VR driving scene simulated night-
time cruising (100 km/h) on a straight highway (two lanes in
each direction) without other traffic. The computer program
generated a random perturbation (deviation onset), and the car
started to drift to the left of the right of the cruising lane with
equal probability. Following each deviation, subjects were re-
quired to steer the car back to the cruising lane as quickly as
possible using the steering wheel (response onset), and hold
on the wheel after the car returned to the approximate center
of the cruising lane (response offset). A lane departure trial is
defined as consisting of three events, deviation onset, response
onset, and response offset. The next lane-departure trial ran-
domly occurs about 5 to 10 sec after response offset in the
current trial. The subjects reaction time (RT) to each lane depar-
ture trial is defined as the interval between deviation onset and
response onset. If the subject does not respond promptly within
2.5 (1.5) sec, the vehicle will hit the left (right) roadside without
a crash and continue to move forward against the curb event
the subject completely ceases to respond. No intervention was
made when the subject fell asleep and stopped responding. After
reaching the lapse period, subjects resumed the task voluntarily
and steered the car back to the cruising position at the earliest.
The goal is to predict RT using a 5-s EEG trial immediately
before it.
B. EEG Pre-Processing
r
At first, raw EEG data was passed through standard pre-
processing pipeline (PREP) of EEGLAB to increase the
signal to noise ratio, it comprises mainly of three operations
[35] [36].
Fig. 2. Distribution of RT values.
– Removing line noise.
– Determining and removing robust reference signal.
– Interpolating the bad channels.
r
Further, the data was downsampled to 250 Hz.
r
Then, the data was epoched to 5 sec trials, i.e. if the lane
deviation is starting at time ‘t’ then the EEG data from
[t − 5,t] is used to predict the RT. Each EEG trial is of
size 30 × 1250.
r
Outliers in the RT values are removed by ignoring the EEG
trials with RT values greater than sum of mean and three
times the standard deviation.
r
Thus, the obtained trials are filtered by a [1, 20] Hz finite
impulse response band-pass filter.
r
The obtained data is then fed through the appropriate spa-
tial filters.
C. RT Pre-Processing
The RT values for 12 subjects are pre-processed in a way
similar to that of the paper (cf. section IV D of [15]). The
data collected from subject 12 is erroneous with data recording
anomalies and is removed from further analysis. This is because,
a large number of response times were longer than 5 seconds,
which are highly absurd in practice. The final distribution of
RTs obtained after pre-processing are shown in Fig. 2.
D. Feature Evaluation
8-fold cross-validation is used to compute the regression per-
formance for each possible fusion of feature set and regression
method. Following feature sets are extracted for each EEG trial.
1) Theta and Alpha powerband features are extracted from
the band-pass filtered EEG trials. We computed the av-
erage power spectral density in the Theta band (4–8 Hz)
and Alpha band (8–13 Hz) for each channel using Welchs
method, and converted these 30 × 2=60band powers to
dBs as our features (denoted as ‘FS1’).
2) Differential Phase Synchrony features (DPS) are extracted
from the band-pass filtered EEG trials. We used 3 fuzzy
sets (K =3) for the RTs, and 21 spatial filters (F =21)
for each fuzzy class. A vector of size (63 × 1) = 63 con-
stitutes the feature vector (DPS). It is denoted as ‘FS2’.
3) Theta and Alpha powerband features extracted from EEG
trials filtered by fuzzy CSPR-OVR. This procedure was
Authorized licensed use limited to: University of Technology Sydney. Downloaded on March 20,2020 at 00:39:55 UTC from IEEE Xplore. Restrictions apply.
REDDY et al.: EEG BASED RT PREDICTION WITH DIFFERENTIAL PHASE SYNCHRONY REPRESENTATIONS 5
TABLE II
R
EGRESSION PERFORMANCE OF FS1, FS2 AND FS3
almost identical to that of ‘FS1’, except that the band-
pass filtered EEG trials were also spatially filtered by
fuzzy CSPR-OVR before the powerband features were
computed. We used 3 fuzzy sets for the RTs, and 10
1
spatial filters for each fuzzy class, so that the spatially
filtered EEG trials is of the dimension 30 × 1250, and the
resultant feature ‘FS3’ has 60 × 1 dimensions.
4) Differential Phase Synchrony features (DPS) extracted
from the EEG trials pre-filtered by fuzzy CSPR-OVR. We
used 3 fuzzy sets (K =3) for the RTs, and 10 spatial filters
(F =10) for each fuzzy class again for a fair comparison.
A vector of size (63 × 1) = 63 constitutes the feature
vector (DPS-CSPR-OVR). It is denoted as ‘FS4’.
All the feature sets obtained above are passed through LASSO
regressor to obtain final reaction time value.
E. Performance Metrics
RMSE, CC and MAPE are the metrics in use for judging the
regression performance. Assume, there are N training points,
y
d
i
represents the true reaction time value of the ith data point
and y
i
represents the predicted reaction time value.
F. Regression Results
The average RMSEs, CCs and MAPEs of LASSO using the
four feature sets (explained in Section III-D) are shown in the
Table II. For each subject, 8-fold cross validation has been used
to partition the feature set into training and validation sets. The
performance is averaged across all the 8-folds. Also, the average
performance across all the subjects is reported. Here, i n general
‘FS4’ recorded the best performance, and both FS4 and FS2
achieved much smaller RMSEs, MAPEs and much larger CCs
than FS3 and FS1, suggesting that our extension of FS4 from
supervised classification to supervised regression can indeed
improve the regression performance.
In conclusion, FS4 had better regression performance than
FS2, FS3 and FS1.
More detailed performance analysis of results and imple-
mentation details comparing performance of the features in
Section III-D are presented in a supplementary file (DPS-
fuzzyCSPR-OVR.pdf).
IV. M
ULTI-TASK DEEP NEURAL NETWORKS
Wei et al., [37] treated drowsiness detection as a classifica-
tion problem by formulating set of thresholds on reaction time
1
We used 10 spatial filters here so that the filtered signals had almost the
same dimensionality as the original signals, which ensured fair performance
comparison.
Fig. 3. Proposed Multitask-DeepNet (MTDNN) approach.
values. The ancillary task in the proposed multitask method ad-
dresses drowsiness detection as a classification problem. Two
tasks in the name of primary and ancillary are in use for the
experiment. EEG based RT prediction is the primary task and
drowsy state classification problem is considered as an ancillary
problem. As far as the ancillary task is concerned, EEG trials
with RT shorter than 1.5 × (alertRT ) are categorized as ‘Alert’
trials, whereas those with RT longer than 2.5 × (alertRT ) are
taken to be as ‘Lapse’ trials indicating ‘drowsy’ phase. In addi-
tion, those EEG trials with RT shorter than 2.5 × (alertRT ) but
longer than 1.5 × (alertRT ) are categorized as ‘Semi-alert’ tri-
als. The alertRT was individually estimated for each subject as
suggested in [4]. Primary task is accomplished using DeepNet-2
and 3, while the ancillary task is accomplished using DeepNet-1
and 3.
A. Pre-Training
Pre-training is used to avoid the learning algorithm to get
stuck in a local optimum. This is especially true while training
a deep model in the situation of a scarce training data. In the
present work, we propose a supervised pre-training approach.
Fig. 3 shows how the proposed Deep network-based method in
DeepNet-1 and DeepNet-2 incorporates the label information.
The supervised pre-training consists of two steps. Firstly, the
DeepNet-1 was trained to predict the three levels of drowsy
states, namely drowsy, transition and awake. Keeping the hid-
den layers of DeepNet-1 intact (with the pre-trained weights),
an output layer consisting of a single output node is added to
construct DeepNet-2, which is then further pre-trained to pre-
dict the reaction time values. Note that the regression layer of
DeepNet-2 was initialized with random weights because this top
layer is different from DeepNet-1. It is easier for Deep network
to learn the three class problem than an infinite class classi-
fication problem (i.e. regression). Nonetheless, the weights of
DeepNet-2 are tuned-up based on DeepNet-1. It follows the
acumen of a meaningful human learning process: simple to
complex tasks. The understanding of learning simpler tasks be
able to benefit the learning for complex tasks.
B. Multi-Task Objective Function
DeepNet-1 uses the cross entropy loss-function. DeepNet-2
uses the mean-squared error loss-function. In DeepNet-1, soft-
max layer is the output layer, while in DeepNet-2, a sigmoid unit
is present in the output. Further, the output layer of DeepNet-3
Authorized licensed use limited to: University of Technology Sydney. Downloaded on March 20,2020 at 00:39:55 UTC from IEEE Xplore. Restrictions apply.
