Image Reconstruction of Tablet Front Camera Recordings in Educational Settings.

TL;DR: The applicability of the setting and processing pipeline on affective state prediction based on front camera recordings during math-solving tasks and emotional stimuli from pictures shown on a tablet are demonstrated.

Abstract: Front camera data from tablets used in educational settings offer valuable clues to student behavior, attention, and affective state. Due to the camera’s angle of view, the face of the student is partially occluded and skewed. This hinders the ability of experts to adequately capture the learning process and student states. In this paper, we present a pipeline and techniques for image reconstruction of front camera recordings. Our setting consists of a cheap and unobtrusive mirror construction to improve the visibility of the face. We then process the image and use neural inpainting to reconstruct missing data in the recordings. We demonstrate the applicability of our setting and processing pipeline on affective state prediction based on front camera recordings (i.e., action units, eye gaze, eye blinks, and movement) during math-solving tasks (active) and emotional stimuli from pictures (passive) shown on a tablet. We show that our setup provides comparable performance for affective state prediction to recordings taken with an external and more obtrusive GoPro camera.