Validation of multicriteria analysis models

It is suggested that as rule-based system (RBS) technology gains wider acceptance, the need to create and maintain large knowledge bases will assume greater importance. Demonstrating a rule base to be free from error remains one of the obstacles to the adoption of this technology. An approach to RBS verification in which the system is modeled as a Petri net on which error detection is performed is presented. A set of propositions is formulated to locate errors of redundancy, conflict, circularity, and gaps in domain knowledge. Rigorous proofs of these propositions are provided. Difficulties in implementing a Petri net-based verifier and the potential restrictions of the applicability of this approach are discussed. >

Investigating the applicability of Petri nets for rule-based system verification

As expert system technology spreads, the need for verification of system knowledge assumes greater importance. This paper addresses the issues involved in demonstrating a rule-based system to be free from error. A holistic perspective is adopted, wherein sorces, manifestations, and effects of errors are identified. A general taxonomy is created, and the implications for system performance and development outlined. Existing strategies for knowledge verification are surveyed, their applicability assessed, and some directions for systematic verification suggested.

Issues in the verification of knowledge in rule-based systems

Expert system evaluation comprises verification, validation and user acceptance testing. The nature of expert systems requires that they be evaluated carefully, and that detailed methodologies for their development be devised. This paper attempts to give practical guidance for the evaluation phase of expert system development. Empirical and logical evaluation methods are surveyed, and both types of method are applied in an evaluation case study. This case study provides insights as to the relative effectiveness of the methods, and demonstrates that empirical and logical testing are complementary to each other. The paper concludes that any evaluation methodology should incorporate both types of method, and presents prescriptions and an outline for such a methodology, which combines the methods for maximum effectiveness.

Towards a methodology for evaluating expert systems

The effects of conflict detection and self-separating aircraft resolution on the mental workload and performance of en-route air traffic controllers were examined. An air traffic control simulator was used to manipulate traffic loads and traffic complexity. A mature stage of free flight was simulated by having controllers monitor self-separating aircraft. Four 30-min scenarios were created to combine moderate (11 aircraft) and heavy traffic loads (17 aircraft) in a 50-mile radius sector with the presence or absence of self-separating and conflicting aircraft. Conflicts (defined as a loss of separation of 5 nm laterally and 1,000 ft vertically) were indicated to the controller by the appearance of a red circle around each of the aircraft involved. A self-separation event was defined as an evasive maneuver (either altitude or speed change) made by 1 aircraft to avoid a potential conflict with another aircraft. Performance and workload measurements indicated that controllers had difficulty both in detecting ...

Christopher J. R. Green

Papers

Verification and validation of expert systems