An electronic device and a method for controlling the electronic device are disclosed. The electronic device includes a memory and a processor electrically connected with the memory. The memory stores instructions to be executed by the processor. The processor is configured to obtain a user voice, determine a response level of the user voice, when the response level of the user voice is a first response level, perform an operation corresponding to the user voice based on a time point of obtaining the user voice, and when the response level of the user voice is a second response level, perform the operation corresponding to the user voice based on a time point of detecting an additional event.

Electronic device and method for controlling the same

Touchscreens continue to advance including progress towards sensing fingers proximal to the display. We explore this emerging pre-touch modality via a self-capacitance touchscreen that can sense multiple fingers above a mobile device, as well as grip around the screen's edges. This capability opens up many possibilities for mobile interaction. For example, using pre-touch in an anticipatory role affords an "ad-lib interface" that fades in a different UI--appropriate to the context--as the user approaches one-handed with a thumb, two-handed with an index finger, or even with a pinch or two thumbs. Or we can interpret pre-touch in a retroactive manner that leverages the approach trajectory to discern whether the user made contact with a ballistic vs. a finely-targeted motion. Pre-touch also enables hybrid touch + hover gestures, such as selecting an icon with the thumb while bringing a second finger into range to invoke a context menu at a convenient location. Collectively these techniques illustrate how pre-touch sensing offers an intriguing new back-channel for mobile interaction.

/pdf/pre-touch-sensing-for-mobile-interaction-33h9h0ooju.pdf

Pre-Touch Sensing for Mobile Interaction

For our Spring Edition of The Reaction Times, we asked the department to share their experiences involving “Chemistry at Home.” With many of us spending more time than usual in our living often-turned-working space, we wanted to hear about how chemistry was continuing, happening, and evolving for our faculty, students, and staff. We were thrilled at the rich and varied responses we received. For some, “chemistry at home” meant the actual doing of chemistry at home, with makeshift labs cropping up in basements and kitchens! We are excited to share Ed Voigtman’s piece on “exercising curiosity” via constructing an echelle spectrograph , as well as Julian Tyson’s update on his work developing methods of determining arsenic compounds in rice—from his home!

On reaction times.

We explore how gaze can support touch interaction on tablets. When holding the device, the free thumb is normally limited in reach, but can provide an opportunity for indirect touch input. Here we propose gaze and touch input, where touches redirect to the gaze target. This provides whole-screen reachability while only using a single hand for both holding and input. We present a user study comparing this technique to direct-touch, showing that users are slightly slower but can utilise one-handed use with less physical effort. To enable interaction with small targets, we introduce CursorShift, a method that uses gaze to provide users temporal control over cursors during direct-touch interactions. Taken together, users can employ three techniques on tablets: direct-touch, gaze and touch, and cursor input. In three applications, we explore how these techniques can coexist in the same UI and demonstrate how tablet tasks can be performed with thumb-only input of the holding hand, and with it describe novel interaction techniques for gaze based tablet interaction.

Gaze and Touch Interaction on Tablets

Selection is a canonical task in user interfaces, commonly supported by presenting objects for acquisition by pointing. In this article, we consider motion correlation as an alternative for selection. The principle is to represent available objects by motion in the interface, have users identify a target by mimicking its specific motion, and use the correlation between the system’s output with the user’s input to determine the selection. The resulting interaction has compelling properties, as users are guided by motion feedback, and only need to copy a presented motion. Motion correlation has been explored in earlier work but only recently begun to feature in holistic interface designs. We provide a first comprehensive review of the principle, and present an analysis of five previously published works, in which motion correlation underpinned the design of novel gaze and gesture interfaces for diverse application contexts. We derive guidelines for motion correlation algorithms, motion feedback, choice of modalities, overall design of motion correlation interfaces, and identify opportunities and challenges identified for future research and design.

/pdf/motion-correlation-selecting-objects-by-matching-their-58d24pzitz.pdf

Motion Correlation: Selecting Objects by Matching Their Movement

Smartphones are frequently used in environments where the user is distracted by another task, for example by walking or by driving. While the typical interface for smartphones involves hardware and software buttons and surface gestures, researchers have recently posited that, for distracted environments, benefits may exist in using motion gestures to execute commands. In this paper, we examine the relative cognitive demands of motion gestures and surface taps and gestures in two specific distracted scenarios: a walking scenario, and an eyes-free seated scenario. We show, first, that there is no significant difference in reaction time for motion gestures, taps, or surface gestures on smartphones. We further show that motion gestures result in significantly less time looking at the smartphone during walking than does tapping on the screen, even with interfaces optimized for eyes-free input. Taken together, these results show that, despite somewhat lower throughput, there may be benefits to making use of motion gestures as a modality for distracted input on smartphones.

/pdf/tap-swipe-or-move-attentional-demands-for-distracted-1f7d2qkofg.pdf

Tap, swipe, or move: attentional demands for distracted smartphone input

In order to reach targets with one hand on common large mobile touch displays, users tilt and shift the device in their hand. In this work, we use this grip change as a continuous information stream for detecting where the user will touch while their finger is still en-route. We refer to this as in the air prediction. We show that grip change detected using standard mobile motion sensors produces similar in the air touch point predictions to techniques that use auxiliary sensor arrays, even in varying physical scenarios such as interacting in a moving vehicle. Finally, our model that combines grip change and the resulting touch point predicted where users intended to land, lowering error rates by 41%.

Grip Change as an Information Side Channel for Mobile Touch Interaction

Designers of motion gestures for mobile devices face the difficult challenge of building a recognizer that can separate gestural input from motion noise. A threshold value is often used to classify motion and effectively balances the rates of false positives and false negatives. We present a bi-level threshold recognition technique designed to lower the rate of recognition failures by accepting either a tightly thresholded gesture or two consecutive possible gestures recognized by a relaxed model. Evaluation of the technique demonstrates that the technique can aid in recognition for users who have trouble performing motion gestures. Lastly, we suggest the use of bi-level thresholding to scaffold the learning of gestures.

https://cs.uwaterloo.ca/research/tr/2011/CS-2011-25.pdf

A recognition safety net: bi-level threshold recognition for mobile motion gestures

Certain implementations of the disclosed technology may include systems, methods, and computer-readable media for transferring images and information from a mobile computing device to a computer monitor for display. In one example implementation, a method is provided that includes receiving, from a remote client, an initiation request, wherein the remote client is associated with a remote display. The method further includes sending a representation of a unique code to the remote client, and receiving, from a mobile device, an indication that the mobile device captured the representation of the unique code. The method further includes receiving, from the mobile device, a display image for presentation on the remote display, and sending the display image to the remote client for presentation on the remote display.

Systems and methods for transferring images and information from a mobile computing device to a computer monitor for display

In this work, we explore the design of multi-resolution input on multi-touch devices. We devised a refined zooming technique named Offset, where the target is set at a location offset from the non-dominant hand while the dominant hand controls the direction and magnitude of the expansion. Additionally, we explored the use of non-persistent transformations of the view in our design. A think-aloud study that compared our design to a bimanual widget interaction and the classic pinch-based interaction with a freeform drawing task suggests that Offset offers benefits in terms of performance and degree of control. As well, for the drawing tasks, the transient nature of view transformations appears to impact not only performance, but workflow, focus of interaction, and subjective quality of results by providing a constant overview of the user's task.

Matei Negulescu

Papers

Tap, swipe, or move: attentional demands for distracted smartphone input

Grip Change as an Information Side Channel for Mobile Touch Interaction

A recognition safety net: bi-level threshold recognition for mobile motion gestures

Systems and methods for transferring images and information from a mobile computing device to a computer monitor for display

ZoomPointing revisited: supporting mixed-resolution gesturing on interactive surfaces