A framework for mobile activity recognition

TLDR: A hybrid method that integrates Latent Dirichlet Allocation with conventional classifiers for learning a generic activity model with minimum annotated data is proposed and a framework for low-level activity recognition with dynamically available sensors is proposed.

Abstract: Activity recognition is being applied in an increasing number of applications. They include health monitoring of the elderly, discovery of frequent behavioural patterns, monitoring of daily life activities (e.g. eating, tooth brushing, sleeping), and analysis of exercise activities (e.g. swimming, running). Current approaches for activity recognition usually use the process of data preprocessing, feature extraction, activity model learning and activity recognition. Most of the previous research pipeline these steps and create static models for processing activity data and recognizing activities. The static models have predefined data sources that are tightly coupled with the models and never change once the models are created. However, the static models are unable to deal with sensor failures and sensor replacements that are quite common in real scenarios. Moreover, additional information provided by newly available data sources from dynamically discovered new sensors may potentially refine the activity model if this information can discriminatively characterize a specific activity class. However, the static models cannot leverage this additional information for self-refinement due to the static assumption of data sources. The primary goal of our research is to design and develop frameworks for activity recognition with dynamically available data sources, and propose and develop algorithms for activity model adaptation with the additional information provided by those data sources. In this thesis, we first provide a critical literature review in the areas of contexts modelling, context management, sensor modelling and sensors in mobile devices, activity recognition, activity model retraining and adaptation, and sensor dynamics in activity recognition. We then present the research on our activity recognition framework that makes the following key contributions. First, we propose a hybrid method that integrates Latent Dirichlet Allocation with conventional classifiers for learning a generic activity model with minimum annotated data. The hybrid method is able to alleviate the problem of data sparsity and requires a little amount of labelled activity data. Furthermore, it can deal with different variants of activity patterns since it is created with activity data of multiple users. The generic activity modelling serves as the starting point of our activity model adaptation with dynamically available sensor data. However, it can also serve as an independent component for other applications such as activity personalization. Second, based on the generic model, we propose a framework for low-level activity (e.g. running, walking) recognition with dynamically available sensors. The components of the framework include a basic classifier, instance selection and smoothing. Firstly, we use AdaBoost as our basic classifier as it is flexible with feature dimensionality and it can automatically select the discriminative features during the learning process. Secondly, we propose to select the most informative instances for activity model adaptation in an unsupervised manner. The instances contain features of the new sensor data, and the information of new sensors are incorporated seamlessly through the adaptation process. Finally, we design smoothing methods by integrating the graphical models such as Hidden Markov Model and Conditional Random Field with the basic classifier AdaBoost. Finally, we propose a framework for high-level activity (e.g, making coffee) recognition with dynamically available contexts. We propose sensor and activity models to address sensor heterogeneity and populating contextual information. Knowledge-driven and data-driven methods are proposed for incorporating the new contexts. The knowledge-driven method specifies the parameters of the new contexts with external knowledge in an unsupervised manner, and the data-driven method learns the parameters of the new contexts with the users' data using the proposed learning-to-rank technique and temporal regularization. Extensive experiments and comprehensive comparisons demonstrate the effectiveness of the proposed frameworks.

Figures

Figure 2.4: Sensor context model.

Figure 5.17: F1-score before and after adaptation across the datasets

Figure 4.11: F1-score corresponding to the number of iterations during inference process for BoostCRF.

Table 5.6: Parameters for baselines.

Table 5.7: Comparison with hybrid classifiers.

Figure 2.6: General architecture of mobile context sensing system.

Full text

A Framework for Mobile Activity Recognition
Jiahui Wen
B.E. (Computer science & Technology), M.Tech(Computer science & Technology)
A thesis submitted for the degree of Doctor of Philosophy at
The University of Queensland in 2017
School of Information Technology & Electrical Engineering

Abstract
Activity recognition is being applied in an increasing number of applications. They include
health monitoring of the elderly, discovery of frequent behavioural patterns, monitoring
of daily life activities (e.g. eating, tooth brushing, sleeping), and analysis of exercise a c-
tivities (e.g. swimming, running). Current app roaches for activity recognition usually use
the process of data preprocessing, feature extraction, activity model learning and activity
recognition. Most of the previous research pipeline these steps a nd create static models
for processing activity data and recognizing activities. The static models have predefined
data sources that are tightly coupled with the models and never change once the models
are created. However, the static models are unable to deal with sensor failures and sensor
replacements that are quite common in real scenarios. Moreover, additional information
provided by newly available data sources from dynamically discovered new sensors may
potentially refine the activity model if this in formation can discriminatively characterize a
specific activity class. However, the static models cannot leverage this additional informa-
tion for self-refinement due to the static assumption of data sources.
The primary goal of our research is to design and develop frameworks for activity recog-
nition with dynamically available data sources, and propose and develop algorithms for
activity model adaptation with the additional information provided by those data sources.
In this thesis, we first provide a critical literature review in the areas of contexts modelling,
context management, sensor modelling and sensors in mobile devices, activity recognition,
activity model retraining and adaptation, and sensor dynamics in activity recognition. We
then present the research on our activity recognition framework that makes the following
key contributions.
First, we propose a hybrid method that integrates Latent Dirichlet Allocation with conven-
tional classifiers for learning a gene ric activity model with minimum annotated data. The
hybrid method is able to alleviate the problem of data sparsity and requires a little amount
of labelled activity data. Furthermore, it can deal with different variants of activity patterns
since it is created with activity data of multiple users. The generic activity modelling serves
as the starting point of our activity model adaptation with dynamically available sensor
data. However, it can also serve as an independent component for other applications such
as activity personalization.
Second, based on the generic model, we propose a framework for low-level activity (e.g.
running, walking) recognition with dynamically available sensors. The components of the
i

framework i nclude a basic classifier, instance sele ction and smoothing. Firstly, we use Ad-
aBoost as our basic classifier as it is flexible with feature dimensionality and it can auto-
matically select the discriminative features during the learning process. Secondly, we pro-
pose to select the most informative instances for activity model adaptation in an unsuper-
vised manner. The instances contain features of the new sensor data, and the information of
new sensors are incorporated seamlessly through the adaptation process. Finally, we design
smoothing methods by integrating the graphical models such as Hidden Markov Model and
Conditional Random Field with the basic classifier AdaBoost.
Finally, we propose a framework for high-level activity (e. g, making coffee) recognition with
dynamically available contexts. We propose sensor and activity models to address sensor
heterogeneity and populating contextual information. Knowledge-driven and data-driven
methods are proposed for incorporating the new contexts. The knowledge-driven m ethod
specifies the parameters of the new contexts with external knowledge in an unsupervised
manner, and the data-driven method learns the parameters of the new contexts with the
users’ data using the proposed learning-to-rank technique and temporal regularization. Ex-
tensive experiments and comprehensive comparisons demonstrate the effectiveness of the
proposed frameworks.
ii

Declaration by author
This thesis is composed of my original work, a nd contains no material previously published
or written by another person except where due reference has been made in the text. I have
clearly stated the contribution by others to jointly-authored works that I have included in
my thesis.
I have clearly stated the contribution of others to my thesis as a whole, including statisti-
cal assistance, survey design, data analysis, significant technical procedures, professional
editorial advice, and any other original research work used or reported in my thesis. The
content of my thesis is the result of work I have carried out since the commencement of my
research higher degree candidature and does not include a substantial part of work that has
been submitted to qualify for the awa rd of any other degree or diploma in any university
or other tertiary institution. I have clearly stated which parts of m y thesis, if any, have been
submitted to qualify for another award.
I acknowledge that an electronic copy of my thesis must be lodged with the University Li-
brary and, subject to the policy and p rocedures of The University of Queensland, the thesis
be made available for research and study in accordance with the Copyright Act 1968 unless
a period of embargo has been approved by the Dean of the Graduate School.
I acknowledge that copyright of all material contained in my thesis resides with the copy-
right holder(s) of that material. Where appropriate I have obtained copyright permission
from the copyright holder to reproduce material in this thesis.
iii

Publications during candidature
[1] Jiahui Wen, Jadwiga Indulska, Mingyang Zhong, Adaptive Activity Learning with Dy-
namically Available Context, In Proc. of the IEEE International Conference on Pervasive
Computing and Communications (PerCom), Sydney, Australia, March 2016.
[2] Mingyang Zhong, Jiahui Wen, Peizhao Hu, Jadwiga Indulska, Advancing Android Ac-
tivity Recognition Service with Markov Smoother, In Proc. of the IE E E PerCom workshop
on Context and Activity Modelling and R ecognition (CoMoRea), St. Louis, USA, March,
2015.
[3] Jiahui Wen, Mingyang Zhong, Jadwiga Indulska, Creating General Model f or Activity
Recognition with M inimum Labelled Data, In Proc. of the AC M International Symposium
on Wearable Computers (ISWC), Osaka, Japan, September, 2015.
[4] Jiahui Wen, Seng Loke, Jadwiga Indulska, Mingyang Zhong, Sensor-based Activity
Recognition with Dynam ically Added Context, In Proc. of the 12th EAI International Con-
ference on Mobile and Ubiquitous Systems: Computing, Networking and Services (Mobiq-
uitous), Coimbra, Portugal, July 2015.
[5] Jiahui Wen, Jadwiga Indulska, Zhiying Wang, Discovering Latent Structures for Activ-
ity Recognition in Smar t Environments, In Proc. of the IEEE International Conference on
Ubiquitous Intelligence and Computing (UIC), Bali, Indonesia, December 2014.
iv