We introduce FlyCam—a novel framework—for gimbal drone camera photography. Our approach abstracts the camera and the drone into a single flying camera object so that the user does not need to think about the drone movement and camera control as two separate actions. The camera is controlled from a single mobile device with six simple touch gestures such as rotate, move forward, yaw, and pitch. The gestures are implemented as seamless commands that combine the gimbal motion with the drone movement. Moreover, we add a sigmoidal motion response that compensates for abrupt drone swinging when moving horizontally. The smooth and simple camera movement has been evaluated by user study, where we asked 20 human subjects to mimic a photograph taken from a certain location. The users used both the default two joystick control and our new touch commands. Our results show that the new interaction performed better in both intuitiveness and easiness of navigation. The users spent less time on task, and the System Usability Scale index of our FlyCam method was 75.13, which is higher than the traditional dual joystick method that scored at 67.38. Moreover, the NASA task load index also showed that our method had lower workload than the traditional method.

FlyCam: Multitouch Gesture Controlled Drone Gimbal Photography

This paper describes a model of the multimodal speech and visual gesture (mSVG) control for aerobots operating at higher nCA autonomy levels, within the context of a patrol, search, and rescue application. The developed mSVG control architecture, its mathematical navigation model, and some high level command operation models were discussed. This was successfully tested using both MATLAB simulation and python based ROS Gazebo UAV simulations. Some limitations were identified, which formed the basis for the further works presented.

The multimodal speech and visual gesture (mSVG) control model for a practical patrol, search, and rescue aerobot

This chapter discusses HCI interfaces used in controlling aerial robotic systems (otherwise known as aerobots). The autonomy control level of aerobot is also discussed. However, due to the limitations of existing models, a novel classification model of autonomy, specifically designed for multirotor aerial robots, called the navigation control autonomy (nCA) model is also developed. Unlike the existing models such as the AFRL and ONR, this model is presented in tiers and has a two-dimensional pyramidal structure. This model is able to identify the control void existing beyond tier-one autonomy components modes and to map the upper and lower limits of control interfaces. Two solutions are suggested for dealing with the existing control void and the limitations of the RC joystick controller – the multimodal HHI-like interface and the unimodal BCI interface. In addition to these, some human factors based performance measurement is recommended, and the plans for further works presented.

Multimodal Human Aerobotic Interaction

Flight testing in a realistic three-dimensional virtual environment is increasingly being considered a safe and cost-effective way of evaluating aircraft models and their control systems. The paper starts by reviewing and comparing the most popular personal computer-based flight simulators that have been successfully interfaced to date with the MathWorks software. This co-simulation approach allows combining the strengths of Matlab toolboxes for functions including navigation, control, and sensor modeling with the advanced simulation and scene rendering capabilities of dedicated flight simulation software. This approach can then be used to validate aircraft models, control algorithms, handle flight characteristics, or perform model identification from flight data. There is, however, a lack of sufficiently detailed step-by-step flight co-simulation tutorials, and there have also been few attempts to evaluate more than one flight co-simulation approach at a time. We, therefore, demonstrate our own step-by-step co-simulation implementations using Simulink with three different flight simulators: Xplane, FlightGear, and Alphalink’s virtual flight test environment (VFTE). All three co-simulations employ a real-time user datagram protocol (UDP) for data communication, and each approach has advantages depending on the aircraft type. In the case of a Cessna-172 general aviation aircraft, a Simulink co-simulation with Xplane demonstrates successful virtual flight tests with accurate simultaneous tracking of altitude and speed reference changes while maintaining roll stability under arbitrary wind conditions that present challenges in the single propeller Cessna. For a medium endurance Rascal-110 unmanned aerial vehicle (UAV), Simulink is interfaced with FlightGear and with QGroundControl using the MAVlink protocol, which allows to accurately follow the lateral UAV path on a map, and this setup is used to evaluate the validity of Matlab-based six degrees of freedom UAV models. For a smaller ZOHD Nano Talon miniature aerial vehicle (MAV), Simulink is interfaced with the VFTE, which was specifically designed for this MAV, and with QGroundControl for the testing of advanced H-infinity observer-based autopilots using a software-in-the-loop (SIL) simulation to achieve robust low altitude flight under windy conditions. This is then finally extended to hardware-in-the-loop (HIL) implementation on the Nano Talon MAV using a controller area network (CAN) databus and a Pixhawk-4 mini autopilot with simulated sensor models.

/pdf/a-tutorial-and-review-on-flight-control-co-simulation-using-2keqrbm0.pdf

A Tutorial and Review on Flight Control Co-Simulation Using Matlab/Simulink and Flight Simulators

Fish and fishery products are among the most important sources of nutritional components for human health, including high-quality proteins, essential vitamins, minerals, and healthy polyunsaturated fatty acids. Fish farming and processing technologies are continuously evolving to improve and enhance the appearance, yield, and quality of fish and fish products from farm to fork throughout the fish supply chain, including growth, postharvest, treatment, storage, transportation, and distribution. Processing of fish involves a period of food withdrawal, collection and transportation, the process of stunning, bleeding, chilling, cutting, packaging, and byproduct recycling. Cutting is a set of crucial operations in fish processing to divide the whole fish into smaller pieces for producing fish products (e.g., fish fillets, steaks, etc.). Various techniques and machinery have been introduced in the field to advance and automate cutting operations. This review aims to provide a comprehensive review of fish cutting techniques, machine vision and artificial intelligence applications, and future directions in fish industries. This paper is expected to stimulate research on enhancing fish cutting yield, product diversity, safety and quality, as well as providing advanced solutions for engineering problems encountered in the fish industry.

/pdf/cutting-techniques-in-the-fish-industry-a-critical-review-5f9uh4yk.pdf

Cutting Techniques in the Fish Industry: A Critical Review

This paper presents the idea of a multimodal human aerobotic interaction. An overview of the aerobotic system and its application is given. The joystick-based controller interface and its limitations is discussed. Two techniques are suggested as emerging alternatives to the joystick-based controller interface used in human aerobotic interaction. The first technique is a multimodal combination of speech, gaze, gesture, and other non-verbal cues already used in regular human-human interaction. The second is telepathic interaction via brain computer interfaces. The potential limitations of these alternatives is highlighted, and the considerations for further works are presented.

The multimodal edge of human aerobotic interaction

This paper discusses the research work conducted to quantify the effective range of lighting levels and ambient noise levels in order to inform the design and development of a multimodal speech and visual gesture (mSVG) control interface for the control of a UAV. Noise level variation from 55 dB to 85 dB is observed under control lab conditions to determine where speech commands for a UAV fails, and to consider why, and possibly suggest a solution around this. Similarly, lighting levels are varied within the control lab condition to determine a range of effective visibility levels. The limitation of this work and some further work from this were also presented.

/pdf/quantifying-the-effects-of-varying-light-visibility-and-594zjvdz82.pdf

Quantifying the effects of varying light-visibility and noise-sound levels in practical multimodal speech and visual gesture (mSVG) interaction with aerobots

This paper investigated the effects of varying noise levels and varying lighting levels on speech and gesture control command interfaces for aerobots. The aim was to determine the practical suitability of the multimodal combination of speech and visual gesture in human aerobotic interaction, by investigating the limits and feasibility of use of the individual components. In order to determine this, a custom multimodal speech and visual gesture interface was developed using CMU (Carnegie Mellon University) sphinx and OpenCV (Open source Computer Vision) libraries, respectively. An experiment study was designed to measure the individual effects of each of the two main components of speech and gesture, and 37 participants were recruited to participate in the experiment. The ambient noise level was varied from 55 dB to 85 dB. The ambient lighting level was varied from 10 Lux to 1400 Lux, under different lighting colour temperature mixtures of yellow (3500 K) and white (5500 K), and different background for capturing the finger gestures. The results of the experiment, which consisted of around 3108 speech utterance and 999 gesture quality observations, were presented and discussed. It was observed that speech recognition accuracy/success rate falls as noise levels rise, with 75 dB noise level being the aerobot’s practical application limit, as the speech control interaction becomes very unreliable due to poor recognition beyond this. It was concluded that multi-word speech commands were considered more reliable and effective than single-word speech commands. In addition, some speech command words (e.g., land) were more noise resistant than others (e.g., hover) at higher noise levels, due to their articulation. From the results of the gesture-lighting experiment, the effects of both lighting conditions and the environment background on the quality of gesture recognition, was almost insignificant, less than 0.5%. The implication of this is that other factors such as the gesture capture system design and technology (camera and computer hardware), type of gesture being captured (upper body, whole body, hand, fingers, or facial gestures), and the image processing technique (gesture classification algorithms), are more important in developing a successful gesture recognition system. Some further works were suggested based on the conclusions drawn from this findings which included using alternative ASR (Automatic Speech Recognition) speech models and developing more robust gesture recognition algorithm.

Peter Saddington

Papers

The multimodal speech and visual gesture (mSVG) control model for a practical patrol, search, and rescue aerobot

Multimodal Human Aerobotic Interaction

The multimodal edge of human aerobotic interaction

Quantifying the effects of varying light-visibility and noise-sound levels in practical multimodal speech and visual gesture (mSVG) interaction with aerobots

Effects of Varying Noise Levels and Lighting Levels on Multimodal Speech and Visual Gesture Interaction with Aerobots