When will automated vehicles come onto the market? This question has puzzled the automotive industry and society for years. The technology and its implementation have made rapid progress over the last decade, but the challenge of how to prove the safety of these systems has not yet been solved. Since a market launch without proof of safety would neither be accepted by society nor by legislators, much time and many resources have been invested into safety assessment in recent years in order to develop new approaches for an efficient assessment. This paper therefore provides an overview of various approaches, and gives a comprehensive survey of the so-called scenario-based approach. The scenario-based approach is a promising method, in which individual traffic situations are typically tested by means of virtual simulation. Since an infinite number of different scenarios can theoretically occur in real-world traffic, even the scenario-based approach leaves the question unanswered as to how to break these down into a finite set of scenarios, and find those which are representative in order to render testing more manageable. This paper provides a comprehensive literature review of related safety-assessment publications that deal precisely with this question. Therefore, this paper develops a novel taxonomy for the scenario-based approach, and classifies all literature sources. Based on this, the existing methods will be compared with each other and, as one conclusion, the alternative concept of formal verification will be combined with the scenario-based approach. Finally, future research priorities are derived.

/pdf/survey-on-scenario-based-safety-assessment-of-automated-2u2a6wl7tk.pdf

Survey on Scenario-Based Safety Assessment of Automated Vehicles

A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. The most active research areas in MLS testing address automated scenario/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.

/pdf/testing-machine-learning-based-systems-a-systematic-mapping-4440jrw6yp.pdf

Testing machine learning based systems: a systematic mapping

Context: Ontologies are known as a formal and explicit conceptualization of entities, their interfaces, behaviors, and relationships. They have been applied in various application domains such as autonomous driving where ontologies are used for decision making, traffic description, auto-pilot etc. It has always been a challenge to test the corresponding safety-critical software systems in autonomous driving that have been playing an increasingly important role in our daily routines. Objective: Failures in these systems potentially not only cause great financial loss but also the loss of lives. Therefore, it is vital to obtain and cover as many as critical driving scenarios during auto drive testing to ensure that the system can always reach a fail-safe state under different circumstances. Method: We outline a general framework for testing, verification, and validation for automated and autonomous driving functions. The introduced method makes use of ontologies for describing the environment of autonomous vehicles and convert them to input models for combinatorial testing. The combinatorial test suite comprises abstract test cases that are mapped to concrete test cases that can be executed using simulation environments. Results: We discuss in detail on how to automatically convert ontologies to the corresponding combinatorial testing input models. Specifically, we present two conversion algorithms and compare their applicability using ontologies with different sizes. We also carried out a case study to further demonstrate the practical value of applying ontology-based test generation in industrial settings. Conclusion: The proposed approach for testing autonomous driving takes ontologies describing the environment of autonomous vehicles, and automatically converts it to test cases that are used in a simulation environment to verify automated driving functions. The conversion relies on combinatorial testing. The first experimental results relying on an example from the automotive industry indicates that the approach can be used in practice.

Ontology-based test generation for automated and autonomous driving functions

Automated Driving Safer and More Efficient Future Driving Foreword

Autonomous, or self-driving, cars are emerging as the solution to several problems primarily caused by humans on roads, such as accidents and traffic congestion. However, those benefits come with great challenges in the verification and validation (V&V) for safety assessment. In fact, due to the possibly unpredictable nature of Artificial Intelligence (AI), its use in autonomous cars creates concerns that need to be addressed using appropriate V&V processes that can address trustworthy AI and safe autonomy. In this study, the relevant research literature in recent years has been systematically reviewed and classified in order to investigate the state-of-the-art in the software V&V of autonomous cars. By appropriate criteria, a subset of primary studies has been selected for more in-depth analysis. The first part of the review addresses certification issues against reference standards, challenges in assessing machine learning, as well as general V&V methodologies. The second part investigates more specific approaches, including simulation environments and mutation testing, corner cases and adversarial examples, fault injection, software safety cages, techniques for cyber-physical systems, and formal methods. Relevant approaches and related tools have been discussed and compared in order to highlight open issues and opportunities.

/pdf/software-verification-and-validation-of-safe-autonomous-cars-5n59tyoh6v.pdf

Software Verification and Validation of Safe Autonomous Cars: A Systematic Literature Review

In this paper, we outline a general automated testing approach to be applied for verification and validation of automated and autonomous driving functions. The approach makes use of ontologies of environment the system under test is interacting with. Ontologies are automatically converted into input models for combinatorial testing, which are used to generate test cases. The obtained abstract test cases are used to generate concrete test scenarios that provide the basis for simulation used to verify the functionality of the system under test. We discuss the general approach including its potential for automation in the automotive domain where there is growing need for sophisticated verification based on simulation in case of automated and autonomous vehicles.

Using Ontologies for Test Suites Generation for Automated and Autonomous Driving Functions

The growing importance of automated and autonomous driving systems becomes more and more visible in the industrial domain as well as in research. Validation and verification of autonomous driving functions is one of the grand challenges in autonomous vehicle development. Virtual validation using simulation has been widely discussed and proposed as a method to solve the challenge. Furthermore, scenario-based approaches have been considered as proper methods combined with virtual validation in order to identify critical scenarios. In this paper, we discuss a method for testing automated and autonomous driving functions using ontologies and combinatorial testing that is able to automate test case generation. Moreover, we report on the application of the method at the industrial level. There we depict the comprehensive application process from the construction of the ontology to test suite execution in detail. This case study shows that the proposed approach can be used for testing and validation of autonomous driving functions in practice.

On the Industrial Application of Combinatorial Testing for Autonomous Driving Functions

Scenario-based approaches have been receiving a huge amount of attention in research and engineering of automated driving systems. Due to the complexity and uncertainty of the driving environment, and the complexity of the driving task itself, the number of possible driving scenarios that an Automated Driving System or Advanced Driving-Assistance System may encounter is virtually infinite. Therefore it is essential to be able to reason about the identification of scenarios and in particular critical ones that may impose unacceptable risk if not considered. Critical scenarios are particularly important to support design, verification and validation efforts, and as a basis for a safety case. In this paper, we present the results of a systematic mapping study in the context of autonomous driving. The main contributions are: (i) introducing a comprehensive taxonomy for critical scenario identification methods; (ii) giving an overview of the state-of-the-art research based on the taxonomy encompassing 86 papers between 2017 and 2020; and (iii) identifying open issues and directions for further research. The provided taxonomy comprises three main perspectives encompassing the problem definition (the why), the solution (the methods to derive scenarios), and the assessment of the established scenarios. In addition, we discuss open research issues considering the perspectives of coverage, practicability, and scenario space explosion.

Finding Critical Scenarios for Automated Driving Systems: A Systematic Mapping Study

Assuring the safety of automated and autonomous driving functions is crucial for a safe deployment of self-driving vehicles on public roads. This includes the need for automated virtual testing methods and exhaustive search for critical scenarios that potentially reveal faults in the driving feature under test. In the past, researches have demonstrated the effectiveness of search-based testing to create situations that result in unintended behavior of the driving feature. In this paper, we contribute to this field of research by developing a method for automated generation of diverse critical scenarios based on a search algorithm that iterative optimizes behavior action sequences of the surrounding traffic participants towards critical situations. Utilizing the provided LG SVL Simulator pipeline, our method effectively generated both critical and challenging test scenarios that either revealed faulty behavior of the ego-vehicle (crash or near-crash) or showed extraordinary behavior of the surrounding traffic participants (e.g. approaching traffic on wrong lane).

Critical and Challenging Scenario Generation based on Automatic Action Behavior Sequence Optimization: 2021 IEEE Autonomous Driving AI Test Challenge Group 108

Jianbo Tao

Papers

Ontology-based test generation for automated and autonomous driving functions

Using Ontologies for Test Suites Generation for Automated and Autonomous Driving Functions

On the Industrial Application of Combinatorial Testing for Autonomous Driving Functions

Finding Critical Scenarios for Automated Driving Systems: A Systematic Mapping Study

Critical and Challenging Scenario Generation based on Automatic Action Behavior Sequence Optimization: 2021 IEEE Autonomous Driving AI Test Challenge Group 108