Voice-user interface (VUI) has become an integral component in modern personal devices (\textite.g., smartphones, voice assistant) by fundamentally evolving the information sharing between the user and device. Acoustic sensing for VUI is designed to sense all acoustic objects; however, the existing VUI mechanism can only offer low-quality speech sensing. This is due to the audible and inaudible interference from complex ambient noise that limits the performance of VUI by causing denial-of-service (DoS) of user requests. Therefore, it is of paramount importance to enable noise-resistant speech sensing in VUI for executing critical tasks with superior efficiency and precision in robust environments. To this end, we investigate the feasibility of employing radio-frequency signals, such as millimeter wave (mmWave) for sensing the noise-resistant voice of an individual. We first perform an in-depth study behind the rationale of voice generation and resulting vocal vibrations. From the obtained insights, we presentWaveEar, an end-to-end noise-resistant speech sensing system.WaveEar comprises a low-cost mmWave probe to localize the position of the speaker among multiple people and direct the mmWave signals towards the near-throat region of the speaker for sensing his/her vocal vibrations. The received signal, containing the speech information, is fed to our novel deep neural network for recovering the voice through exhaustive extraction. Our experimental evaluation under real-world scenarios with 21 participants shows the effectiveness ofWaveEar to precisely infer the noise-resistant voice and enable a pervasive VUI in modern electronic devices.

https://dl.acm.org/doi/pdf/10.1145/3307334.3326073

WaveEar: Exploring a mmWave-based Noise-resistant Speech Sensing for Voice-User Interface

Motion sensors on current smartphones have been exploited for audio eavesdropping due to their sensitivity to vibrations. However, this threat is considered low-risk because of two widely acknowledged limitations: First, unlike microphones, motion sensors can only pick up speech signals traveling through a solid medium. Thus, the only feasible setup reported previously is to use a smartphone gyroscope to eavesdrop on a loudspeaker placed on the same table. The second limitation comes from a common sense that these sensors can only pick up a narrow band (85-100Hz) of speech signals due to a sampling ceiling of 200Hz. In this paper, we revisit the threat of motion sensors to speech privacy and propose AccelEve, a new side-channel attack that employs a smartphone’s accelerometer to eavesdrop on the speaker in the same smartphone. Specifically, it utilizes the accelerometer measurements to recognize the speech emitted by the speaker and to reconstruct the corresponding audio signals. In contrast to previous works, our setup allows the speech signals to always produce strong responses in accelerometer measurements through the shared motherboard, which successfully addresses the first limitation and allows this kind of attacks to penetrate into real-life scenarios. Regarding the sampling rate limitation, contrary to the widely-held belief, we observe up to 500Hz sampling rates in recent smartphones, which almost covers the entire fundamental frequency band (85-255Hz) of adult speech. On top of these pivotal observations, we propose a novel deep learning based system that learns to recognize and reconstruct speech information from the spectrogram representation of acceleration signals. This system employs adaptive optimization on deep neural networks with skip connections using robust and generalizable losses to achieve robust recognition and reconstruction performance. Extensive evaluations demonstrate the effectiveness and high accuracy of our attack under various settings.

/pdf/learning-based-practical-smartphone-eavesdropping-with-built-mn4za4xmgl.pdf

Learning-based Practical Smartphone Eavesdropping with Built-in Accelerometer.

Speech and speaker recognition systems are employed in a variety of applications, from personal assistants to telephony surveillance and biometric authentication. The wide deployment of these systems has been made possible by the improved accuracy in neural networks. Like other systems based on neural networks, recent research has demonstrated that speech and speaker recognition systems are vulnerable to attacks using manipulated inputs. However, as we demonstrate in this paper, the end-to-end architecture of speech and speaker systems and the nature of their inputs make attacks and defenses against them substantially different than those in the image space. We demonstrate this first by systematizing existing research in this space and providing a taxonomy through which the community can evaluate future work. We then demonstrate experimentally that attacks against these models almost universally fail to transfer. In so doing, we argue that substantial additional work is required to provide adequate mitigations in this space.

SoK: The Faults in our ASRs: An Overview of Attacks against Automatic Speech Recognition and Speaker Identification Systems

Internet of Thing platforms prosper home automation applications (apps). Prior research concerns intra-app security. Our work reveals that automation apps, even secured individually, still cause a family of threats when they interplay, termed as Cross-App Interference (CAI) threats. We systematically categorize such threats and encode them using satisfiability modulo theories (SMT). We present HomeGuard, a system for detecting and handling CAI threats in real deployments. A symbolic executor is built to extract rule semantics, and instrumentation is utilized to capture configuration during app installation. Rules and configuration are checked against SMT models, the solutions of which indicate the existence of corresponding CAI threats. We further combine app functionalities, device attributes and CAI types to label the risk level of CAI instances. In our evaluation, HomeGuard discovers 663 CAI instances from 146 SmartThings market apps, imposing minor latency upon app installation and no runtime overhead.

Cross-App Interference Threats in Smart Homes: Categorization, Detection and Handling

Modern electronic devices have become “smart” as well as omnipresent in our day-to-day lives. From small household devices to large industrial machines, smart devices have become very popular in every possible application domain. Smart devices in our homes, offices, buildings, and cities can connect with other devices as well as with the physical world around them. This increasing popularity has also placed smart devices as the center of attention among attackers. Already, several types of malicious activities exist that attempt to compromise the security and privacy of smart devices. One interesting and noteworthy emerging threat vector is the attacks that abuse the use of sensors on smart devices. Smart devices are vulnerable to sensor-based threats and attacks due to the lack of proper security mechanisms available to control the use of sensors by installed apps. By exploiting the sensors (e.g., accelerometer, gyroscope, microphone, light sensor, etc.) on a smart device, attackers can extract information from the device, transfer malware to a device, or trigger a malicious activity to compromise the device. In this paper, we explore various threats and attacks abusing sensors of smart devices for malicious purposes. Specifically, we present a detailed survey about existing sensor-based threats and attacks to smart devices and countermeasures that have been developed to secure smart devices from sensor-based threats. Furthermore, we discuss security and privacy issues of smart devices in the context of sensor-based threats and attacks and conclude with future research directions.

A Survey on Sensor-Based Threats and Attacks to Smart Devices and Applications

According to recent research, motion sensors available on current smartphone platforms may be sensitive to speech signals. From a security and privacy perspective, this raises a serious concern regarding sensitive speech reconstruction, and speaker or gender identification by a malicious application having unrestricted access to motion sensor readings, without using the microphone. In this paper, we revisit this important line of research and closely inspect the effect of speech on smartphone motion sensors, in particular, gyroscope and accelerometer. First, we revisit the previously studied scenario (Michalevsky et al.; USENIX Security 2014), where the smartphone shares a common surface with a loudspeaker (with subwoofer) generating speech signals. We observe some effect on the motion sensor signals, which may indeed allow speaker and gender recognition to an extent. However, we also argue that the recorded effect on the sensor readings is possibly from conductive vibrations through the shared surface instead of direct acoustic vibrations due to speech as perceived in previous work. Second, we further extend the previous work by analyzing the effect of speech produced by (1) other less powerful speakers like the in-built laptop and smartphone speakers, and (2) live humans. Our experiments show that in-built laptop speakers were only able to affect the accelerometer when the laptop and the motion sensor shared a surface. Smartphone speakers were not found to be powerful enough to invoke a response in the motion sensors through aerial vibrations. We also report that in the presence of live human speech, we did not notice any effect on the motion sensor readings. Our results have two-fold implications. First, human-rendered speech seems potentially incapacitated to trigger smartphone motion sensors within the limited sampling rates imposed by the smartphone operating systems. Second, it seems that even machine-rendered speech may not be powerful enough to affect smartphone motion sensors through the aerial medium, although it may induce vibrations through a conductive surface that these sensors, especially accelerometer, could pick up if a relatively powerful speaker is used. Overall, our results suggest that smartphone motion sensors may pose a threat to speech privacy only in some limited scenarios.

/pdf/speechless-analyzing-the-threat-to-speech-privacy-from-5ah9oymlya.pdf

Speechless: Analyzing the Threat to Speech Privacy from Smartphone Motion Sensors

Voice access technologies are widely adopted in mobile devices and voice assistant systems as a convenient way of user interaction. Recent studies have demonstrated a potentially serious vulnerability of the existing voice interfaces on these systems to "hidden voice commands". This attack uses synthetically rendered adversarial sounds embedded within a voice command to trick the speech recognition process into executing malicious commands, without being noticed by legitimate users. In this paper, we employ low-cost motion sensors, in a novel way, to detect these hidden voice commands. In particular, our proposed system extracts and examines the unique audio signatures of the issued voice commands in the vibration domain. We show that such signatures of normal commands vs. synthetic hidden voice commands are distinctive, leading to the detection of the attacks. The proposed system, which benefits from a speaker-motion sensor setup, can be easily deployed on smartphones by reusing existing on-board motion sensors or utilizing a cloud service that provides the relevant setup environment. The system is based on the premise that while the crafted audio features of the hidden voice commands may fool an authentication system in the audio domain, their unique audio-induced surface vibrations captured by the motion sensor are hard to forge. Our proposed system creates a harder challenge for the attacker as now it has to forge the acoustic features in both the audio and vibration domains, simultaneously. We extract the time and frequency domain statistical features, and the acoustic features (e.g., chroma vectors and MFCCs) from the motion sensor data and use learning-based methods for uniquely determining both normal commands and hidden voice commands. The results show that our system can detect hidden voice commands vs. normal commands with 99.9% accuracy by simply using the low-cost motion sensors that have very low sampling frequencies.

Defeating hidden audio channel attacks on voice assistants via audio-induced surface vibrations

Pairing between wireless devices may be secured by the use of an auxiliary channel such as audio, visuals or vibrations. A simple approach to pairing involves one of the devices initiating the transmission of a key, or keying material like a short password, over the auxiliary channel to the other device. A successful pairing is achieved when the receiving device is able to decode the key without any errors while the attacker is unable to eavesdrop the key.In this paper, we focus on the security of the vibration channel when used for the key transmission. As shown in some recent work, sending the keying material over a clear vibrational channel poses a significant risk of an acoustic side channel attack. Specifically, an adversary can listen onto the acoustic sounds generated by the vibration motor of the sending device and infer the keying material with a high accuracy. To counteract this threat, we propose a novel pairing scheme, called Vibreaker (a ``Vibrating speaker''), that involves active injection of acoustic noise in order to mask the key signal. In this scheme, the sending device artificially injects noise in the otherwise clear audio channel while transmitting the keying material via vibrations. We experiment with several choices for the noise signal and demonstrate that the security of the audio channel is significantly enhanced with Vibreaker when appropriate noise is used. The scheme requires no additional effort by the user, and imposes minimum hardware requirement and hence can be applied to many different contexts, such as pairing of IoT and implanted devices, wearables and other commodity gadgets.

Vibreaker: Securing Vibrational Pairing with Deliberate Acoustic Noise

In this paper, we build a speech privacy attack that exploits speech reverberations from a smartphone's inbuilt loudspeaker captured via a zero-permission motion sensor (accelerometer). We design our attack Spearphone, and demonstrate that speech reverberations from inbuilt loudspeakers, at an appropriate loudness, can impact the accelerometer, leaking sensitive information about the speech. In particular, we show that by exploiting the affected accelerometer readings and carefully selecting feature sets along with off-the-shelf machine learning techniques, Spearphone can perform gender classification (accuracy over 90%) and speaker identification (accuracy over 80%) for the audio/video playback on the smartphone for our recorded dataset. We use lightweight classifiers and an off-the-shelf machine learning tool so that the attacking effort is minimized, making our attack practical. Our results with testing the attack on a voice call and voice assistant response were also encouraging, showcasing the impact of the proposed attack. In addition, we perform speech recognition and speech reconstruction to extract more information about the eavesdropped speech to an extent. Our work brings to light a fundamental design vulnerability in many currently-deployed smartphones, which may put people's speech privacy at risk while using the smartphone in the loudspeaker mode during phone calls, media playback or voice assistant interactions.

Spearphone: a lightweight speech privacy exploit via accelerometer-sensed reverberations from smartphone loudspeakers

In this paper, we build a speech privacy attack that exploits speech reverberations generated from a smartphone's in-built loudspeaker captured via a zero-permission motion sensor (accelerometer). We design our attack Spearphone2, and demonstrate that speech reverberations from inbuilt loudspeakers, at an appropriate loudness, can impact the accelerometer, leaking sensitive information about the speech. In particular, we show that by exploiting the affected accelerometer readings and carefully selecting feature sets along with off-the-shelf machine learning techniques, Spearphone can successfully perform gender classification (accuracy over 90%) and speaker identification (accuracy over 80%) for any audio/video playback on the smartphone. Our results with testing the attack on a voice call and voice assistant response were also encouraging, showcasing the impact of the proposed attack. In addition, we perform speech recognition and speech reconstruction to extract more information about the eavesdropped speech to an extent. Our work brings to light a fundamental design vulnerability in many currently-deployed smartphones, which may put people's speech privacy at risk while using the smartphone in the loudspeaker mode during phone calls, media playback or voice assistant interactions.

S Abhishek Anand

Papers

Speechless: Analyzing the Threat to Speech Privacy from Smartphone Motion Sensors

Defeating hidden audio channel attacks on voice assistants via audio-induced surface vibrations

Vibreaker: Securing Vibrational Pairing with Deliberate Acoustic Noise

Spearphone: a lightweight speech privacy exploit via accelerometer-sensed reverberations from smartphone loudspeakers

Spearphone: A Speech Privacy Exploit via Accelerometer-Sensed Reverberations from Smartphone Loudspeakers.