Assuring the quality of an expert system is critical. A poor quality system may make costly errors resulting in considerable damage to the user or owner of the system, such as financial loss or human suffering. Hence verification and validation, methods and techniques aimed at ensuring quality, are fundamentally important. This paper surveys the issues, methods and techniques for verifying and validating expert systems. Approaches to defining the quality of a system are discussed, drawing upon work in both computing and the model building disciplines, which leads to definitions of verification and validation and the associated concepts of credibility, assessment and evaluation. An approach to verification based upon the detection of anomalies is presented, and related to the concepts of consistency, completeness, correctness and redundancy. Automated tools for expert system verification are reviewed. Considerable attention is then given to the issues in structuring the validation process, particularly the establishment of the criteria by which the system is judged, the need to maintain objectivity, and the concept of reliability. This is followed by a review of validation methods for validating both the components of a system and the system as a whole, and includes examples of some useful statistical methods. Management of the verification and validation process is then considered, and it is seen that the location of methods for verification and validation in the development life-cycle is of prime importance.

Expert system verification and validation: a survey and tutorial

Ripple Down Rules (RDR) is a knowledge acquisition method which constrains the interactions between the expert and a shell to acquire only correct knowledge. Although RDR works well, it is only suitable for the problem of providing a single classification for a set of data. Multiple Classification Ripple Down Rules (MCRDR) is an extension of RDR which allows multiple independent classifications. The approach has been evaluated in simulation studies where the human expert is replaced by a simulated expert. MCRDR may provide a basis for building a general problem solver for a range of problems beyond classification.

Multiple Classification Ripple Down Rules : Evaluation and Possibilities

We evaluated whether a knowledge-based system (KBS) connected to a ventilator in pressure support mode could correctly predict the ability of patients to tolerate total withdrawal from ventilatory support. The KBS was designed to continuously adapt ventilatory assistance to the needs of the patient, to manage a strategy of gradually decreasing ventilatory assistance, and to indicate when the patient was able to breathe without assistance. Thirty-eight patients for whom weaning was being considered were evaluated using a conventional battery of parameters, including weaning criteria, tolerance of a T-piece trial, and outcome 48h after permanent withdrawal of ventilation. The results of this evaluation were compared with the suggestions made by the KBS at the end of a period of KBS-driven mechanical ventilation inserted in the conventional weaning procedure. The positive predictive value of the KBS was 89%, versus 77% for the conventional procedure and 81% for the rapid shallow breathing index alone. The KBS correctly predicted the course of five patients who tolerated a T-piece trial but required ventilation within 48 h. We conclude that our KBS ensured appropriate patient management during the weaning period and improved our ability to predict responses to weaning.

/pdf/evaluation-of-a-knowledge-based-system-providing-ventilatory-2qvomb0kjr.pdf

Evaluation of a knowledge-based system providing ventilatory management and decision for extubation.

Many expert systems have moved out of development laboratories into field test and routine use. About sixty such systems are listed. Academic research laboratories are contributing manpower to fuel the commercial development of AI. But the quantity of AI research may decline as a result unless the applied systems are experimented with and analyzed.

/pdf/expert-systems-working-systems-and-the-research-literature-4e2u8p4e17.pdf

Expert systems: working systems and the research literature

Anomalies such as redundant, contradictory, and deficient knowledge in a knowledge base are symptoms of probable errors. Detecting anomalies is a well-established method for verifying knowledge-based systems. Although many tools have been developed to perform anomaly detection, several important issues have been neglected, especially the theoretical foundations and computational limitations of anomaly detection methods, and analyses of the utility of such tools in practical use. This article addresses these issues by presenting a theoretical foundation of anomaly detection methods, and by presenting empirical results obtained in applying one anomaly detection tool to perform verification on five real-world knowledge-based systems. the techniques presented apply specifically to verifying rule-based knowledge bases without numerical certainty measures. © 1994 John Wiley & Sons, Inc.

Foundation and application of knowledge base verification

This paper surveys the verification of expert system knowledge bases by detecting anomalies. Such anomalies are highly indicative of errors in the knowledge base. The paper is in two parts. The first part describes four types of anomaly: redundancy, ambivalence, circularity, and deficiency. We consider rule bases which are based on first-order logic, and explain the anomalies in terms of the syntax and semantics of logic. The second part presents a review of five programs which have been built to detect various subsets of the anomalies. The four anomalies provide a framework for comparing the capabilities of the five tools, and we highlight the strengths and weaknesses of each approach. This paper therefore provides not only a set of underlying principles for performing knowledge base verification through anomaly detection, but also a survey of the state-of-the-art in building practical tools for carrying out such verification. The reader of this paper is expected to be familiar with first-order logic.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S026988890000624X

Principles and practice in verifying rule-based systems

The first step in establishing the reliability of an expert system is to verify that its knowledge base is free from anomalies such as redundant, conflicting or missing knowledge. Such anomalies are suggestive of errors in the knowledge base; empirical evidence suggests that at errors revealed by verification may be hard to detect by conventional system testing. Verification can be performed automatically by a domain-independent anomaly checking tool because the anomalies can be formally defined in terms of logic and detected by syntactic inspection of logic-based knowledge bases. This paper provides not only a formal framework for verification in the form of a set of anomaly definitions expressed in first-order predicate logic, but also detailed descriptions and algorithms for an automatic verification program called COVER. COVER offers a number of advantages over previous anomaly detection tools: firstly, it detects a wider range of anomalies; secondly, it incorporates a number of novel features which afford its users more flexibility to make the checking task more practical for large knowledge bases. A familiarity with first-order predicate logic is assumed for the reader of this paper.

Verifying expert systems: A logical framework and a practical tool

The rapid growth in the use of expert systems has led to widespread interest in techniques for their evaluation, which includes verification and validation. Evaluation techniques in the software engineering community are well established but there are different opinions about the effectiveness of transferring software engineering methodologies to expert systems. This paper outlines some of the issues involved in evaluating expert systems and cites almost 200 significant papers on the topic. We present it with the expectation that it will help both new and established researchers become acquainted with the literature of an important and growing field.

Expert system evaluation techniques: a selected bibliography

The success of numerous expert systems in practical applications warrants a more formal approach to their development and evaluation. Reliability assurance of expert systems requires a methodology for the specification and evaluation of these systems. Expert systems are a new class of software system, but some traditional techniques of software development may be adapted to their construction. However, the specification of an expert system differs from that of a more traditional software program in that parts of the specification are permitted to be only partially described when development starts. Specifications have two important purposes: as contracts between suppliers and clients, and as blueprints for implementation. A specification consists of a problem specification and a solution specification. The problem specification plays the role of contract and states explicitly what the problem to be solved is, and the constraints that the final product must satisfy. The solution specification plays the role of blueprint and has two major aspects: analyzing how a human expert solves the problem, and proposing an equivalent automated solution. We propose an approach to the specification of expert systems that is flexible, yet rigorous enough to cover the important features of a wide range of potential expert system applications. We describe fully each of the components of an expert system specification and we relate specification to the issues of evaluation and maintenance of expert systems.

Specifying an expert system

The authors focus on the problems of specification of an expert system namely, what needs to be specified, what can be specified and how. Two distinct major roles for a software specification are identified: as a contract between parties involved in system development and as a blueprint for the design and implementation of the system. It is shown that these purposes require quite different specifications. The role of specification as a contract is taken by the problem specification, which essentially describes what system is to be built. The blueprint specification is complementary, and describes how the system is to be built, including a description of the knowledge to be used and a description of how to represent and reason with that knowledge. >

Aïda Batarekh

Papers

Principles and practice in verifying rule-based systems

Verifying expert systems: A logical framework and a practical tool

Expert system evaluation techniques: a selected bibliography

Specifying an expert system

Specification of expert systems