Dr. Edward Colgate gave a keynote on morning of the second day of the conference. He spoke about haptics and their vital role in everyday life. He showed that, on contrary, haptics are decidedly less important when interfacing to the digital world. Our touch screens treat the finger as a pointer and more or less ignore the thousands of mechanoreceptors within, not to mention the somatosensory cortex. This is an odd state of affairs. Dr. Colgate explained that part of the reason is technological: how do you go beyond simple vibrations and create versatile, programmable haptic experiences? Research in his lab has focused on solving this question for better than a decade now. In his talk, he explained the rapidly expanding capabilities of “surface haptic” technologies. Dr. Colgate also showed how limited are the data on haptics used to create great experiences, be they for education, entertainment, or elsewhere. That turns out to be a much more difficult problem. He described some incipient efforts and offered some thoughts for the future. This chapter provides a lightly edited transcription of that keynote.

Touching with Feeling

According to amputees, sensory feedback is amongst the most important features lacking from commercial prostheses. Although restoration of touch by means of implantable neural interfaces has been achieved, these approaches require surgical interventions, and their long-term usability still needs to be fully investigated. Here, we developed a non-invasive alternative which maintains some of the advantages of invasive approaches, such as a somatotopic sensory restitution scheme. We used transcutaneous electrical nerve stimulation (TENS) to induce referred sensations to the phantom hand of amputees. These sensations were characterized in four amputees over two weeks. Although the induced sensation was often paresthesia, the location corresponded to parts of the innervation regions of the median and ulnar nerves, and electroencephalographic (EEG) recordings confirmed the presence of appropriate responses in relevant cortical areas. Using these sensations as feedback during bidirectional prosthesis control, the patients were able to perform several functional tasks that would not be possible otherwise, such as applying one of three levels of force on an external sensor. Performance during these tasks was high, suggesting that this approach could be a viable alternative to the more invasive solutions, offering a trade-off between the quality of the sensation, and the invasiveness of the intervention.

A somatotopic bidirectional hand prosthesis with transcutaneous electrical nerve stimulation based sensory feedback.

Any implant or prosthesis replacing a function or functions of an organ or group of organs should be biologically and sensorily integrated with the human body in order to increase their acceptance with their user. If this replacement is for a human hand, which is an important interface between humans and their environment, the acceptance issue and developing sensory-motor embodiment will be more challenging. Despite progress in prosthesis technologies, 50–60% of hand amputees wear a prosthetic device. One primary reason for the rejection of the prosthetic hands is that there is no or negligibly small feedback or tactile sensation from the hand to the user, making the hands less functional. In fact, the loss of a hand means interrupting the closed-loop sensory feedback between the brain (motor control) and the hand (sensory feedback through the nerves). The lack of feedback requires significant cognitive efforts from the user in order to do basic gestures and daily activities. To this aim, recently, there has been significant development in the provision of sensory feedback from transradial prosthetic hands, to enable the user take part in the control loop and improve user embodiment. Sensory feedback to the hand users can be provided via invasive and non-invasive methods. The latter includes the use of temperature, vibration, mechanical pressure and skin stretching, electrotactile stimulation, phantom limb stimulation, audio feedback, and augmented reality. This paper provides a comprehensive review of the non-invasive methods, performs their critical evaluation, and presents challenges and opportunities associated with the non-invasive sensory feedback methods.

/pdf/a-review-of-non-invasive-sensory-feedback-methods-for-59bvxitgc0.pdf

A Review of Non-Invasive Sensory Feedback Methods for Transradial Prosthetic Hands

Introduction: The constant challenge to restore sensory feedback in prosthetic hands has provided several research solutions, but virtually none has reached clinical fruition. A prosthetic ...

A review of invasive and non-invasive sensory feedback in upper limb prostheses.

OBJECTIVE Hand amputation is a highly disabling event, which significantly affects quality of life. An effective hand replacement can be achieved if the user, in addition to motor functions, is provided with the sensations that are naturally perceived while grasping and moving. Intraneural peripheral electrodes have shown promising results toward the restoration of the sense of touch. However, the long-term usability and clinical relevance of intraneural sensory feedback have not yet been clearly demonstrated. METHODS To this aim, we performed a 6-month clinical study with 3 transradial amputees who received implants of transverse intrafascicular multichannel electrodes (TIMEs) in their median and ulnar nerves. After calibration, electrical stimulation was delivered through the TIMEs connected to artificial sensors in the digits of a prosthesis to generate sensory feedback, which was then used by the subjects while performing different grasping tasks. RESULTS All subjects, notwithstanding their important clinical differences, reported stimulation-induced sensations from the phantom hand for the whole duration of the trial. They also successfully integrated the sensory feedback into their motor control strategies while performing experimental tests simulating tasks of real life (with and without the support of vision). Finally, they reported a decrement of their phantom limb pain and a general improvement in mood state. INTERPRETATION The promising results achieved with all subjects show the feasibility of the use of intraneural stimulation in clinical settings. ANN NEUROL 2019;85:137-154.

/pdf/six-month-assessment-of-a-hand-prosthesis-with-intraneural-1pxqxjg9il.pdf

Six-Month Assessment of a Hand Prosthesis with Intraneural Tactile Feedback.

Transcutaneous electrical stimulation can provide amputees with tactile feedback for better manipulating an advanced prosthesis. In general, there are two ways to transfer the stimulus to the skin: somatotopical feedback (SF) that stimulates the phantom digit somatotopy on the stump and non-somatotopical feedback (NF) that stimulates other positions on the human body. To investigate the difference between SF and NF, electrotactile experiments were conducted on seven amputees. Electrical stimulation was applied via a complete phantom map to the residual limb (SF) and to the upper arm (NF) separately. The behavior results of discrimination accuracy and response time were used to examine: 1) performance differences between SF and NF for discriminating position, type and strength of tactile feedback; 2) performance differences between SF and NF for one channel (1C), three channels (3C), and five channels (5C). NASA-TLX standardized testing was used to determine differences in mental workload between SF and NF. The grand-averaged discrimination accuracy for SF was 6% higher than NF, and the average response time for SF was 600 ms faster than NF. SF is better than NF for position, type, strength, and the overall modality regarding both accuracy and response time except for 1C modality (p<0.001). Among the six modalities of stimulation channels, performance of 1C/SF was the best, which was similar to that of 1C/NF and 3C/SF; performance of 3C/NF was similar to that of 5C/SF; performance of 5C/NF was the worst. NASA-TLX scores indicated that mental workload increased as the number of stimulation channels increased. We quantified the difference between SF and NF, and the influence of different number of stimulation channels. SF was better than NF in general, but the practical issues such as the limited area of stumps could constrain the use of SF. We found that more channels increased the amount and richness of information to the amputee while fewer channels resulted in higher performance, and thus the 3C/SF modality was a good compromise. Based on this study, we provide possible solutions to the practical problems involving the implementation of tactile feedback for amputees. These results are expected to promote the application of SF and NF tactile feedback for amputees in the future.

/pdf/somatotopical-feedback-versus-non-somatotopical-feedback-for-3bm55d8yfd.pdf

Somatotopical feedback versus non-somatotopical feedback for phantom digit sensation on amputees using electrotactile stimulation

Closed-loop control is important for amputees to manipulate myoelectric prostheses intuitively and dexterously. Tactile feedback can help amputees improve myoelectric control performance for grasping objects. To investigate the effects of different tactile feedback, we performed experiments on six amputees and six able-bodied subjects via electrotactile stimulation. Using a virtual environment, six kinds of objects with different weights and stiffnesses were used for grasping tasks. Five feedback conditions (no feedback, pressure feedback, slip feedback, pressure ${+}$ slip feedback, and vision feedback) were considered. Nine evaluation indexes and three control objectives (rapidity, economy, and stability) were proposed. Under the five feedback conditions, our study investigated four issues: 1) three types of grasping-related failures; 2) four types of grasping-related time measures; 3) average grasping force; 4) standard deviation of the grasping force. Results indicate that: 1) slip feedback is better than pressure feedback; 2) pressure ${+}$ slip feedback can improve grasping rapidity; 3) slip feedback significantly contributes to grasping economy and stability; and 4) pressure ${+}$ slip feedback can perform as well as vision feedback.

Effects of Different Tactile Feedback on Myoelectric Closed-Loop Control for Grasping Based on Electrotactile Stimulation

Psychophysical tests and standardized questionnaires are often used to analyze tactile sensation based on subjective judgment in conventional studies. In contrast with the subjective evaluation, a novel method based on electroencephalography (EEG) is proposed to explore the possibility of quantifying tactile sensation in an objective way. The proposed experiments adopt cutaneous electrical stimulation to generate two kinds of sensations (vibration and pressure) with three grades (low/medium/strong) on eight subjects. Event-related potentials (ERPs) and event-related synchronization/desynchronization (ERS/ERD) are extracted from EEG, which are used as evaluation indexes to distinguish between vibration and pressure, and also to discriminate sensation grades. Results show that five-phase P1–N1–P2–N2–P3 deflection is induced in EEG. Using amplitudes of latter ERP components (N2 and P3), vibration and pressure sensations can be discriminated on both individual and grand-averaged ERP (p < 0.05). The grand-average ERPs can distinguish the three sensations grades, but there is no significant difference on individuals. In addition, ERS/ERD features of mu rhythm (8–13 Hz) are adopted. Vibration and pressure sensations can be discriminated on grand-average ERS/ERD (p < 0.05), but only some individuals show significant difference. The grand-averaged results show that most sensation grades can be differentiated, and most pairwise comparisons show significant difference on individuals (p < 0.05). The work suggests that ERP- and ERS/ERD-based EEG features may have potential to quantify tactile sensations for medical diagnosis or engineering applications.

Quantifying Different Tactile Sensations Evoked by Cutaneous Electrical Stimulation Using Electroencephalography Features.

We propose a new lateral force feedback device, the UltraShiver, which employs a combination of in-plane ultrasonic oscillation (around 30 kHz) and out-of-plane electroadhesion. It can achieve a strong active lateral force (400 mN) on the bare fingertip while operating silently. The lateral force is a function of pressing force, lateral vibration velocity, and electroadhesive voltage, as well as the relative phase between the velocity and voltage. In this paper, we perform experiments to investigate characteristics of the UltraShiver and their influence on lateral force.

UltraShiver: Lateral force feedback on a bare fingertip via ultrasonic oscillation and electroadhesion

We propose a new lateral force feedback device, the UltraShiver, which employs a combination of in-plane ultrasonic oscillation (around 30 kHz) and out-of-plane electroadhesion. It can achieve a strong active lateral force (400 mN) on the bare fingertip while operating silently. The lateral force is a function of pressing force, lateral vibration velocity, and electroadhesive voltage, as well as the relative phase between the velocity and voltage. In this paper, we perform experiments to investigate characteristics of the UltraShiver and their influence on lateral force. A lumped-parameter model is developed to understand the physical underpinnings of these influences. The model with frequency-weighted electroadhesion forces shows good agreement with experimental results. In addition, a Gaussian-like potential well is rendered as an application of the UltraShiver.

Heng Xu

Papers

Somatotopical feedback versus non-somatotopical feedback for phantom digit sensation on amputees using electrotactile stimulation

Effects of Different Tactile Feedback on Myoelectric Closed-Loop Control for Grasping Based on Electrotactile Stimulation

Quantifying Different Tactile Sensations Evoked by Cutaneous Electrical Stimulation Using Electroencephalography Features.

UltraShiver: Lateral force feedback on a bare fingertip via ultrasonic oscillation and electroadhesion

UltraShiver: Lateral Force Feedback on a Bare Fingertip via Ultrasonic Oscillation and Electroadhesion