Showing papers on "Situation awareness published in 2022"

Sensors and AI Techniques for Situational Awareness in Autonomous Ships: A Review

Abstract: Autonomous ships are expected to improve the level of safety and efficiency in future maritime navigation. Such vessels need perception for two purposes: to perform autonomous situational awareness and to monitor the integrity of the sensor system itself. In order to meet these needs, the perception system must fuse data from novel and traditional perception sensors using Artificial Intelligence (AI) techniques. This article overviews the recognized operational requirements that are imposed on regular and autonomous seafaring vessels, and then proceeds to consider suitable sensors and relevant AI techniques for an operational sensor system. The integration of four sensors families is considered: sensors for precise absolute positioning (Global Navigation Satellite System (GNSS) receivers and Inertial Measurement Unit (IMU)), visual sensors (monocular and stereo cameras), audio sensors (microphones), and sensors for remote-sensing (RADAR and LiDAR). Additionally, sources of auxiliary data, such as Automatic Identification System (AIS) and external data archives are discussed. The perception tasks are related to well-defined problems, such as situational abnormality detection, vessel classification, and localization, that are solvable using AI techniques. Machine learning methods, such as deep learning and Gaussian processes, are identified to be especially relevant for these problems. The different sensors and AI techniques are characterized keeping in view the operational requirements, and some example state-of-the-art options are compared based on accuracy, complexity, required resources, compatibility and adaptability to maritime environment, and especially towards practical realization of autonomous systems.

Microgrid Digital Twins: Concepts, Applications, and Future Trends

Abstract: Following the fourth industrial revolution, and with the recent advances in information and communication technologies, the digital twinning concept is attracting the attention of both academia and industry worldwide. A microgrid digital twin (MGDT) refers to the digital representation of a microgrid (MG), which mirrors the behavior of its physical counterpart by using high-fidelity models and simulation platforms as well as real-time bi-directional data exchange with the real twin. With the massive deployment of sensor networks and IoT technologies in MGs, a huge volume of data is continuously generated, which contains valuable information to enhance the performance of MGs. MGDTs provide a powerful tool to manage the huge historical data and real-time data stream in an efficient and secure manner and support MGs’ operation by assisting in their design, operation management, and maintenance. In this paper, the concept of the digital twin (DT) and its key characteristics are introduced. Moreover, a workflow for establishing MGDTs is presented. The goal is to explore different applications of DTs in MGs, namely in design, control, operator training, forecasting, fault diagnosis, expansion planning, and policy-making. Besides, an up-to-date overview of studies that applied the DT concept to power systems and specifically MGs is provided. Considering the significance of situational awareness, security, and resilient operation for MGs, their potential enhancement in light of digital twinning is thoroughly analyzed and a conceptual model for resilient operation management of MGs is presented. Finally, future trends in MGDTs are discussed.

Situational Awareness: Techniques, Challenges, and Prospects

Abstract: Situational awareness (SA) is defined as the perception of entities in the environment, comprehension of their meaning, and projection of their status in near future. From an Air Force perspective, SA refers to the capability to comprehend and project the current and future disposition of red and blue aircraft and surface threats within an airspace. In this article, we propose a model for SA and dynamic decision-making that incorporates artificial intelligence and dynamic data-driven application systems to adapt measurements and resources in accordance with changing situations. We discuss measurement of SA and the challenges associated with quantification of SA. We then elaborate a plethora of techniques and technologies that help improve SA ranging from different modes of intelligence gathering to artificial intelligence to automated vision systems. We then present different application domains of SA including battlefield, gray zone warfare, military- and air-base, homeland security and defense, and critical infrastructure. Finally, we conclude the article with insights into the future of SA.

Situational Assessment for Intelligent Vehicles Based on Stochastic Model and Gaussian Distributions in Typical Traffic Scenarios

Abstract: In intelligent driving, situational assessment (SA) is an important technology, which helps to improve the cognitive ability of intelligent vehicles in the environment. Uncertainty analysis is very significant in situation assessment. This article proposes an SA method based on uncertainty risk analysis. Under uncertain conditions, according to the random environment model and Gaussian distribution model, the collision probability between multiple vehicles is estimated by comprehensive trajectory prediction. The proposed method considers collision probabilities of different prediction points within and outside the prediction range and obtains long-term accurate prediction results. The method is suitable for the situation risk assessment of sensor systems in the presence of unexpected dynamic obstacles, sensor failures or communication losses in traffic, and different environmental sensing accuracy. The experimental results show that in the dynamic traffic environment, the proposed scenario assessment method can not only accurately predict and assess the situation risks within the prediction range, but also provide accurate scenario risk assessment outside the prediction range.

Anomaly Detection in Maritime AIS Tracks: A Review of Recent Approaches

Abstract: The automatic identification system (AIS) was introduced in the maritime domain to increase the safety of sea traffic. AIS messages are transmitted as broadcasts to nearby ships and contain, among others, information about the identification, position, speed, and course of the sending vessels. AIS can thus serve as a tool to avoid collisions and increase onboard situational awareness. In recent years, AIS has been utilized in more and more applications since it enables worldwide surveillance of virtually any larger vessel and has the potential to greatly support vessel traffic services and collision risk assessment. Anomalies in AIS tracks can indicate events that are relevant in terms of safety and also security. With a plethora of accessible AIS data nowadays, there is a growing need for the automatic detection of anomalous AIS data. In this paper, we survey 44 research articles on anomaly detection of maritime AIS tracks. We identify the tackled AIS anomaly types, assess their potential use cases, and closely examine the landscape of recent AIS anomaly research as well as their limitations.

DRONET: Multi-Tasking Framework for Real-Time Industrial Facility Aerial Surveillance and Safety

Abstract: The security of key and critical infrastructures is crucial for uninterrupted industrial process flow needed in strategic management as these facilities are major targets of invaders. The emergence of non-military use of drones especially for logistics comes with the challenge of redefining the anti-drone approach in determining a drone’s harmful status in the airspace based on certain metrics before countering it. In this work, a vision-based multi-tasking anti-drone framework is proposed to detect drones, identifies the airborne objects, determines its harmful status through perceived threat analysis, and checks its proximity in real-time prior to taking an action. The model is validated using manually generated 5460 drone samples from six (6) drone models under sunny, cloudy, and evening scenarios and 1709 airborne objects samples of seven (7) classes under different environments, scenarios (blur, scales, low illumination), and heights. The proposed model was compared with seven (7) other object detection models in terms of accuracy, sensitivity, F1-score, latency, throughput, reliability, and efficiency. The simulation result reveals that, overall, the proposed model achieved superior multi-drone detection accuracy of 99.6%, attached object identification of sensitivity of 99.80%, and F1-score of 99.69%, with minimal error, low latency, and less computational complexity needed for effective industrial facility aerial surveillance. A benchmark dataset is also provided for subsequent performance evaluation of other object detection models.

IoMT-Enabled Fusion-Based Model to Predict Posture for Smart Healthcare Systems

Abstract: Smart healthcare applications depend on data from wearable sensors (WSs) mounted on a patient’s body for frequent monitoring information. Healthcare systems depend on multi-level data for detecting illnesses and consequently delivering correct diagnostic measures. The collection of WS data and integration of that data for diagnostic purposes is a difficult task. This paper proposes an Errorless Data Fusion (EDF) approach to increase posture recognition accuracy. The research is based on a case study in a health organization. With the rise in smart healthcare systems, WS data fusion necessitates careful attention to provide sensitive analysis of the recognized illness. As a result, it is dependent on WS inputs and performs group analysis at a similar rate to improve diagnostic efficiency. Sensor breakdowns, the constant time factor, aggregation, and analysis results all cause errors, resulting in rejected or incorrect suggestions. This paper resolves this problem by using EDF, which is related to patient situational discovery through healthcare surveillance systems. Features of WS data are examined extensively using active and iterative learning to identify errors in specific postures. This technology improves position detection accuracy, analysis duration, and error rate, regardless of user movements. Wearable devices play a critical role in the management and treatment of patients. They can ensure that patients are provided with a unique treatment for their medical needs. This paper discusses the EDF technique for optimizing posture identification accuracy through multi-feature analysis. At first, the patients’ walking patterns are tracked at various time intervals. The characteristics are then evaluated in relation to the stored data using a random forest classifier.

Impact of VR-Based Training on Human-Robot Interaction for Remote Operating Construction Robots

Abstract: Despite the increased interest in automation and the expanded deployment of robots in the construction industry, using robots in a dynamic and unstructured working environment has caused safety concerns in operating construction robots. Improving human–robot interaction (HRI) can increase the adoption of robots on construction sites; for example, increasing trust in robots could help construction workers to accept new technologies. Confidence in operation (or self-efficacy), mental workload, and situational awareness are among other key factors that help such workers to remote operate robots safely. However, construction workers have very few opportunities to practice with robots to build trust, self-efficacy, and situational awareness, as well as resistance against increasing mental workload, before interacting with them on job sites. Virtual reality (VR) could afford a safer place to practice with the robot; thus, we tested if VR-based training could improve these four outcomes during the remote operation of construction robots. We measured trust in the robot, self-efficacy, mental workload, and situational awareness in an experimental study where construction workers remote-operated a demolition robot. Fifty workers were randomly assigned to either VR-based training or traditional in-person training led by an expert trainer. Results show that VR-based training significantly increased trust in the robot, self-efficacy, and situational awareness, compared to traditional in-person training. Our findings suggest that VR-based training can allow for significant increases in beneficial cognitive factors over more traditional methods and has substantial implications for improving HRI using VR, especially in the construction industry.

Smart Grid Cyber-Physical Situational Awareness of Complex Operational Technology Attacks: A Review

Abstract: The smart grid (SG), regarded as the complex cyber-physical ecosystem of infrastructures, orchestrates advanced communication, computation, and control technologies to interact with the physical environment. Due to the high rewards that threats to the grid can realize, adversaries can mount complex cyber-attacks such as advanced persistent threats-based and coordinated attacks to cause operational malfunctions and power outages in the worst scenarios: The latter of which was reflected in the Ukrainian power grid attack. Despite widespread research on smart grid security, the impact of targeted attacks on control and power systems is anecdotal. This article reviews the smart grid security from collaborative factors, emphasizing the situational awareness (SA). Specifically, we propose a threat modeling framework and review the nature of cyber-physical attacks to understand their characteristics and impacts on the smart grid’s control and physical systems. We examine the existing threats detection and defense capabilities, such as intrusion detection systems (IDSs), moving target defense (MTD), and co-simulation techniques, along with discussing the impact of attacks through situational awareness and power system metrics. We discuss the human factor aspects for power system operators in analyzing the impacts of cyber-attacks. Finally, we investigate the research challenges with key research gaps to shed light on future research directions.

Agent Transparency, Situation Awareness, Mental Workload, and Operator Performance: A Systematic Literature Review.

Abstract: OBJECTIVE In this review, we investigate the relationship between agent transparency, Situation Awareness, mental workload, and operator performance for safety critical domains. BACKGROUND The advancement of highly sophisticated automation across safety critical domains poses a challenge for effective human oversight. Automation transparency is a design principle that could support humans by making the automation's inner workings observable (i.e., "seeing-into"). However, experimental support for this has not been systematically documented to date. METHOD Based on the PRISMA method, a broad and systematic search of the literature was performed focusing on identifying empirical research investigating the effect of transparency on central Human Factors variables. RESULTS Our final sample consisted of 17 experimental studies that investigated transparency in a controlled setting. The studies typically employed three human-automation interaction types: responding to agent-generated proposals, supervisory control of agents, and monitoring only. There is an overall trend in the data pointing towards a beneficial effect of transparency. However, the data reveals variations in Situation Awareness, mental workload, and operator performance for specific tasks, agent-types, and level of integration of transparency information in primary task displays. CONCLUSION Our data suggests a promising effect of automation transparency on Situation Awareness and operator performance, without the cost of added mental workload, for instances where humans respond to agent-generated proposals and where humans have a supervisory role. APPLICATION Strategies to improve human performance when interacting with intelligent agents should focus on allowing humans to see into its information processing stages, considering the integration of information in existing Human Machine Interface solutions.

Towards an Interpretable Approach to Classify and Summarize Crisis Events from Microblogs

Abstract: Microblogging platforms like Twitter have been heavily leveraged to report and exchange information about natural disasters. The real-time data on these sites is highly helpful in gaining situational awareness and planning aid efforts. However, disaster-related messages are immersed in a high volume of irrelevant information. The situational data of disaster events also vary greatly in terms of information types ranging from general situational awareness (caution, infrastructure damage, casualties) to individual needs or not related to the crisis. It thus requires efficient methods to handle data overload and prioritize various types of information. This paper proposes an interpretable classification-summarization framework that first classifies tweets into different disaster-related categories and then summarizes those tweets. Unlike existing work, our classification model can provide explanations or rationales for its decisions. In the summarization phase, we employ an Integer Linear Programming (ILP) based optimization technique along with the help of rationales to generate summaries of event categories. Extensive evaluation on large-scale disaster events shows (a). our model can classify tweets into disaster-related categories with an 85% Macro F1 score and high interpretability (b). the summarizer achieves (5-25%) improvement in terms of ROUGE-1 F-score over most state-of-the-art approaches.

Multi-Robot Collaborative Perception With Graph Neural Networks

Abstract: Multi-robot systems such as swarms of aerial robots are naturally suited to offer additional flexibility, resilience, and robustness in several tasks compared to a single robot by enabling cooperation among the agents. To enhance the autonomous robot decision-making process and situational awareness, multi-robot systems have to coordinate their perception capabilities to collect, share, and fuse environment information among the agents in an efficient and meaningful way such to accurately obtain context-appropriate information or gain resilience to sensor noise or failures. In this paper, we propose a general-purpose Graph Neural Network (GNN) with the main goal to increase, in multi-robot perception tasks, single robots' inference perception accuracy as well as resilience to sensor failures and disturbances. We show that the proposed framework can address multi-view visual perception problems such as monocular depth estimation and semantic segmentation. Several experiments both using photo-realistic and real data gathered from multiple aerial robots' viewpoints show the effectiveness of the proposed approach in challenging inference conditions including images corrupted by heavy noise and camera occlusions or failures.

On the (In)Security of 1090ES and UAT978 Mobile Cockpit Information Systems – An Attacker Perspective on the Availability of ADS-B Safety- and Mission-Critical Systems

Abstract: Automatic dependent surveillance-broadcast (ADS-B) is a key air surveillance technology and a critical component of next-generation air transportation systems. It significantly simplifies aircraft surveillance technology and improves airborne traffic situational awareness. Many types of mobile cockpit information systems (MCISs) are based on ADS-B technology. MCIS gives pilots the flight and traffic-related information they need. MCIS has two parts: an ADS-B transceiver and an electronic flight bag (EFB) application. The ADS-B transceivers transmit and receive the ADS-B radio signals while the EFB applications hosted on mobile phones display the data. Because they are cheap, lightweight, and easy to install, MCISs became very popular. However, because it lacks basic security measures, ADS-B technology is vulnerable to cyberattacks, which makes the MCIS inherently exposed to attacks. This is even more likely because they are power, memory, and computationally constrained. This study explores the cybersecurity posture of various MCIS setups for both types of ADS-B technology: 1090ES and UAT978. Total six portable MCIS devices and 21 EFB applications were tested against radio-link- based attacks by transmission-capable software-defined radio (SDR). Packet-level denial of service (DoS) attacks affected approximately 63% and 37% of 1090ES and UAT978 setups, respectively, while many of them experienced a system crash. Our experiments show that DoS attacks on the reception could meaningfully reduce transmission capacity. Our coordinated attack and fuzz tests also reported worrying issues on the MCIS. The consistency of our results on a very broad range of hardware and software configurations indicate the reliability of our proposed methodology as well as the effectiveness and efficiency of our platform.

Threat assessment, sense making, and critical decision-making in police, military, ambulance, and fire services

Abstract: Abstract Military and emergency response remain inherently dangerous occupations that require the ability to accurately assess threats and make critical decisions under significant time pressures. The cognitive processes associated with these abilities are complex and have been the subject of several significant, albeit service specific studies. Here, we present an attempt at finding the commonalities in threat assessment, sense making, and critical decision-making for emergency response across police, military, ambulance, and fire services. Relevant research is identified and critically appraised through a systematic literature review of English-language studies published from January 2000 through July 2020 on threat assessment and critical decision-making theory in dynamic emergency service and military environments. A total of 10,084 titles and abstracts were reviewed, with 94 identified as suitable for inclusion in the study. We then present our findings focused on six lines of enquiry: Bibliometrics, Language, Situation Awareness, Critical Decision Making, Actions, and Evaluation. We then thematically analyse these findings to reveal the commonalities between the four services. Despite existing single or dual service studies in the field, this research is significant in that it is the first examine decision making and threat assessment theory across all four contexts of military, police, fire and ambulance services, but it is also the first to assess the state of knowledge and explore the extent that commonality exists and models or practices can be applied across each discipline. The results demonstrate all military and emergency services personnel apply both intuitive and formal decision-making processes, depending on multiple situational and individual factors. Institutional restriction of decision-making to a single process at the expense of the consideration of others, or the inappropriate training and application of otherwise appropriate decision-making processes in certain circumstances is likely to increase the potential for adverse outcomes, or at the very least restrict peak performance being achieved. The applications of the findings of the study not only extend to facilitating improved practice in each of the individual services examined, but provide a basis to assist future research, and contribute to the literature exploring threat assessment and decision making in dynamic contexts.

Lightweight Challenge-Response Authentication in SDN-Based UAVs Using Elliptic Curve Cryptography

Abstract: Unmanned aerial vehicles (UAVs) (also known as drones) are aircraft that do not require the presence of a human pilot to fly. UAVs can be controlled remotely by a human operator or autonomously by onboard computer systems. UAVs have many military uses, including battlefield surveillance, effective target tracking and engagement in air-to-ground warfare, and situational awareness in challenging circumstances. They also offer a distinct advantage in various applications such as forest fire monitoring and surveillance. Surveillance systems are developed using advanced technologies in the modern era of communications and networks. As a result, UAVs require enhancements to control and manage systems efficiently. Network security is a critical concern with respect to UAVs due to the risk of surveillance information theft and physical misuse. Although several new tools have been introduced to secure networks, attackers can use more advanced methods to get into a UAV network and create problems that pose an organizational threat to the entire system. Security mechanisms also reduce the performance of systems because some restrictive measures prevent users from accessing specific resources, but a few techniques and tools have overcome the problem of performance reduction in various scenarios. There are many types of attacks, i.e., denial of service attacks (DOS), distributed denial of service attacks (DDOS), address resolution protocol (ARP) spoofing, sniffing, etc., that make it challenging to maintain a UAV network. This research paper proposes a lightweight challenge-response authentication that can overcome the previously mentioned problems. As security is provided by utilizing a minimum number of bits in memory, this technique offers the same security features while using fewer network resources, low computing resources, and low power consumption.

Human-centred risk assessment for a land-based control interface for an autonomous vessel

Abstract: Abstract Autonomous ferries are providing new opportunities for urban transport mobility. With this change comes a new risk picture, which is characterised to a large extent by the safe transition from autonomous mode to manual model in critical situations. The paper presents a case study of applying an adapted risk assessment method based on the Scenario Analysis in the Crisis Intervention and Operability study (CRIOP) framework. The paper focuses on the applicability of the Scenario Analysis to address the human-automation interaction. This is done by presenting a case study applying the method on a prototype of a Human–Machine Interface (HMI) in the land-based control centre for an autonomous ferry. Hence, the paper presents findings on two levels: a method study and a case study. A concept of operation (CONOPS) and a preliminary hazard analysis lay the foundation for the scenario development, the analysis, and the discussion in a case study workshop. The case study involved a Scenario Analysis of a handover situation where the autonomous system asked for assistance from the operator in a land-based control centre. The results include a list of identified safety issues such as missing procedures, an alarm philosophy and an emergency preparedness plan, and a need for explainable AI. Findings from the study show that the Scenario Analysis method can be a valuable tool to address the human element in risk assessment by focusing on the operators’ ability to handle critical situations.

Human machine interface design for continuous support of mode awareness during automated driving: An online simulation

Abstract: In the transition towards higher levels of vehicle automation, one of the key concerns with regards to human factors is to avoid mode confusion, when drivers misinterpret the driving mode and therewith misjudge their own tasks and responsibility. To enhance mode awareness, a clear human centered Human Machine Interface (HMI) is essential. The HMI should support the driver tasks of both supervising the driving environment when needed and self-regulating their non-driving related activities (NDRAs). Such support may be provided by either presenting continuous information on automation reliability, from which the driver needs to infer what task is required, or by presenting continuous information on the currently required driving task and allowed NDRA directly. Additionally, it can be valuable to provide continuous information to support anticipation of upcoming changes in the automation mode and its associated reliability or required and allowed driver task(s). Information that could support anticipation includes the available time until a change in mode (i.e. time budget), information on the upcoming mode, and reasons for changing to the upcoming mode. The current work investigates the effects of communicating this potentially valuable information through HMI design. Participants received information from an HMI during simulated drives in a simulated car presented online (using Microsoft Teams) with an experimenter virtually accompanying and guiding each session. The HMI either communicated on automation reliability or on the driver task, and either included information supporting anticipation or did not include such information. Participants were thinking aloud during the simulated drives and reported on their experience and preferences afterwards. Anticipatory information supported understanding about upcoming changes without causing information overload or overreliance. Moreover, anticipatory information and information on automation reliability, and especially a combination of the two, best supported understandability and usability. Recommendations are provided for future work on facilitating supervision and NDRA self-regulation during automated driving through HMI design.

Exploration of Mobile Device Behavior for Mitigating Advanced Persistent Threats (APT): A Systematic Literature Review and Conceptual Framework

Abstract: During the last several years, the Internet of Things (IoT), fog computing, computer security, and cyber-attacks have all grown rapidly on a large scale. Examples of IoT include mobile devices such as tablets and smartphones. Attacks can take place that impact the confidentiality, integrity, and availability (CIA) of the information. One attack that occurs is Advanced Persistent Threat (APT). Attackers can manipulate a device’s behavior, applications, and services. Such manipulations lead to signification of a deviation from a known behavioral baseline for smartphones. In this study, the authors present a Systematic Literature Review (SLR) to provide a survey of the existing literature on APT defense mechanisms, find research gaps, and recommend future directions. The scope of this SLR covers a detailed analysis of most cybersecurity defense mechanisms and cutting-edge solutions. In this research, 112 papers published from 2011 until 2022 were analyzed. This review has explored different approaches used in cybersecurity and their effectiveness in defending against APT attacks. In a conclusion, we recommended a Situational Awareness (SA) model known as Observe–Orient–Decide–Act (OODA) to provide a comprehensive solution to monitor the device’s behavior for APT mitigation.

Cyber-resilience of Critical Cyber Infrastructures: Integrating digital twins in the electric power ecosystem

Abstract: This contribution presents a model for cyber-resilience of critical cyber infrastructures (CCI) based on the implementation of a digital twin. It addresses the risks associated with the integration of computational, communication and physical aspects of CCIs. We focus specifically on cybersecurity in the electric power sector due both to its salience and to the potential risks associated to failures in guaranteeing resilience. Informed by the literature on information security management, situational awareness (SA) and common operational picture (COP), we derive an overarching model to provide CCIs’ actors with increased cyber situational awareness, common understanding of incidents and enhanced response capacity. On the practical side, the model seeks to minimize response time and to reduce the impact of cyber-attacks on the organizations and on society as a whole. We develop a process model and validate three design propositions through a formative evaluation in the context of a digital twin implementation in the EU electrical power sector. We discuss the implications of this model for further research as well as practical applications for the electrical power sector.

User Requirements Analysis on Augmented Reality-Based Maintenance in Manufacturing

Abstract: Maintenance of technical equipment in manufacturing is inevitable for sustained productivity with minimal downtimes. Elimination of unscheduled interruptions as well as real-time monitoring of equipment health can potentially benefit from adopting augmented reality (AR) technology. How best to employ this technology in maintenance demands a fundamental comprehension of user requirements for production planners. Despite augmented reality applications being developed to assist various manufacturing operations, no previous study has examined how these user requirements in maintenance have been fulfilled and the potential opportunities that exist for further development. Reviews on maintenance have been general on all industrial fields rather than focusing on a specific industry. In this regard, a systematic literature review was performed on previous studies on augmented reality applications in the maintenance of manufacturing entities from 2017 to 2021. Specifically, the review examines how user requirements have been addressed by these studies and identifies gaps for future research. The user requirements are drawn from the challenges encountered during AR-based maintenance in manufacturing following a similar approach to usability engineering methodologies. The needs are identified as ergonomics, communication, situational awareness, intelligence sources, feedback, safety, motivation, and performance assessment. Contributing factors to those needs are cross-tabulated with the requirements and their results presented as trends, prior to drawing insights and providing possible future suggestions for the made observations.