We present an algorithm for simultaneous face detection, landmarks localization, pose estimation and gender recognition using deep convolutional neural networks (CNN). The proposed method called, HyperFace, fuses the intermediate layers of a deep CNN using a separate CNN followed by a multi-task learning algorithm that operates on the fused features. It exploits the synergy among the tasks which boosts up their individual performances. Additionally, we propose two variants of HyperFace: (1) HyperFace-ResNet that builds on the ResNet-101 model and achieves significant improvement in performance, and (2) Fast-HyperFace that uses a high recall fast face detector for generating region proposals to improve the speed of the algorithm. Extensive experiments show that the proposed models are able to capture both global and local information in faces and performs significantly better than many competitive algorithms for each of these four tasks.

HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition

The present disclosure generally relates to generating and modifying virtual avatars. An electronic device having a camera and a display apparatus displays a virtual avatar that changes appearance in response to changes in a face in a field of view of the camera. In response to detecting changes in one or more physical features of the face in the field of view of the camera, the electronic device modifies one or more features of the virtual avatar.

Emoji recording and sending

Detecting liveness of a speaker comprises: generating an ultrasound signal; receiving an audio signal comprising a reflection of the ultrasound signal; using the received audio signal comprising the reflection of the ultrasound signal to detect the liveness of a speaker; monitoring ambient ultrasound noise; and adjusting the operation of a system receiving the audio signal, based on a level of the reflected ultrasound and the monitored ambient ultrasound noise. The method can be used in a voice biometrics system, in which case detecting the liveness of a speaker comprises determining whether a received speech signal may be a product of a replay attack. The operation of the voice biometrics system may be adjusted based on a level of the reflected ultrasound and the monitored ambient ultrasound noise.

Detection of liveness

Detecting a replay attack on a voice biometrics system comprises receiving a speech signal; forming an autocorrelation of at least a part of the speech signal; and identifying that the received speech signal may result from a replay attack based on said autocorrelation. Identifying that the received speech signal may result from a replay attack may be achieved by: comparing the autocorrelation with a reference value; and identifying that the received speech signal may result from a replay attack based on a result of the comparison of the autocorrelation with the reference value, or by: supplying the autocorrelation to a neural network trained to distinguish autocorrelations formed from speech signals resulting from replay attacks from autocorrelations formed from speech signals not resulting from replay attacks.

Detection of replay attack

A method of detecting a replay attack on a voice biometrics system comprises: receiving an audio signal representing speech; detecting a magnetic field; determining if there is a correlation between the audio signal and the magnetic field; and if there is a correlation between the audio signal and the magnetic field, determining that the audio signal may result from a replay attack.

Magnetic detection of replay attack

Audio analysis learning is performed using video data. Video data is obtained, on a first computing device, wherein the video data includes images of one or more people. Audio data is obtained, on a second computing device, which corresponds to the video data. A face is identified within the video data. A first voice, from the audio data, is associated with the face within the video data. The face within the video data is analyzed for cognitive content. Audio features are extracted corresponding to the cognitive content of the video data. The audio data is segmented to correspond to an analyzed cognitive state. An audio classifier is learned, on a third computing device, based on the analyzing of the face within the video data. Further audio data is analyzed using the audio classifier.

Audio analysis learning using video data

Techniques are described for image generation for avatar image animation using translation vectors. An avatar image is obtained for representation on a first computing device. An autoencoder is trained, on a second computing device comprising an artificial neural network, to generate synthetic emotive faces. A plurality of translation vectors is identified corresponding to a plurality of emotion metrics, based on the training. A bottleneck layer within the autoencoder is used to identify the plurality of translation vectors. A subset of the plurality of translation vectors is applied to the avatar image, wherein the subset represents an emotion metric input. The emotion metric input is obtained from facial analysis of an individual. An animated avatar image is generated for the first computing device, based on the applying, wherein the animated avatar image is reflective of the emotion metric input and the avatar image includes vocalizations.

Avatar image animation using translation vectors

Techniques are described for machine-trained analysis for multimodal machine learning. A computing device captures a plurality of information channels, wherein the plurality of information channels includes contemporaneous audio information and video information from an individual. A multilayered convolutional computing system learns trained weights using the audio information and the video information from the plurality of information channels, wherein the trained weights cover both the audio information and the video information and are trained simultaneously, and wherein the learning facilitates emotional analysis of the audio information and the video information. A second computing device captures further information and analyzes the further information using trained weights to provide an emotion metric based on the further information. Additional information is collected with the plurality of information channels from a second individual and learning the trained weights factors in the additional information. The further information can include only video data or audio data.

Multimodal machine learning for emotion metrics

Techniques are described for speech analysis for cross-language mental state identification. A first group of utterances in a first language is collected, on a computing device, with an associated first set of mental states. The first group of utterances and the associated first set of mental states are stored on an electronic storage device. A machine learning system is trained using the first group of utterances and the associated first set of mental states that were stored. A second group of utterances from a second language is processed, on the machine learning system that was trained, wherein the processing determines a second set of mental states corresponding to the second group of utterances. The second set of mental states is output. A series of heuristics is output, based on the correspondence between the first group of utterances and the associated first set of mental states.

Speech analysis for cross-language mental state identification

Vehicle manipulation uses cognitive state engineering. Images of a vehicle occupant are obtained using imaging devices within a vehicle. The one or more images include facial data of the vehicle occupant. A computing device is used to analyze the images to determine a cognitive state. Audio information from the occupant is obtained and the analyzing is augmented based on the audio information. The cognitive state is mapped to a loading curve, where the loading curve represents a continuous spectrum of cognitive state loading variation. The vehicle is manipulated, based on the mapping to the loading curve, where the manipulating uses cognitive state alteration engineering. The manipulating includes changing vehicle occupant sensory stimulation. Additional images of additional occupants of the vehicle are obtained and analyzed to determine additional cognitive states. Additional cognitive states are used to adjust the mapping. A cognitive load is estimated based on eye gaze tracking.

Taniya Mishra

Papers

Audio analysis learning using video data

Avatar image animation using translation vectors

Multimodal machine learning for emotion metrics

Speech analysis for cross-language mental state identification

Vehicle manipulation using cognitive state engineering