Quantitative expert judgements are used in reliability assessments to inform critically important decisions. Structured elicitation protocols have been advocated to improve expert judgements, yet their application in reliability is challenged by a lack of examples or evidence that they improve judgements. This paper aims to overcome these barriers. We present a case study where two world-leading protocols, the IDEA protocol and the Classical Model, were combined and applied by the Australian Department of Defence for a reliability assessment. We assess the practicality of the methods and the extent to which they improve judgements. The average expert was extremely overconfident, with 90% credible intervals containing the true realisation 36% of the time. However, steps contained in the protocols substantially improved judgements. In particular, an equal weighted aggregation of individual judgements and the inclusion of a discussion phase and revised estimate helped to improve calibration, statistical accuracy, and the Classical Model score. Further improvements in precision and information were made via performance weighted aggregation. This paper provides useful insights into the application of structured elicitation protocols for reliability and the extent to which judgements are improved. The findings raise concerns about existing practices for utilising experts in reliability assessments and suggest greater adoption of structured protocols is warranted. We encourage the reliability community to further develop examples and insights.

/pdf/improving-expert-forecasts-in-reliability-application-and-1rd6tfeesx.pdf

Improving expert forecasts in reliability: Application and evidence for structured elicitation protocols

Systems engineers are routinely tasked with facilitating the delicate balance between cost, schedule, and technical performance in acquisition programs that are continuously subjected to various outside influences. While there are several quantitative methods to estimate acquisition program cost and schedule performance as well as identify their risks e.g., Earned Value Management, the estimation of technical performance and technical risk is generally heuristic in nature. In order to monitor the progress of the technical aspects of an acquisition program, the systems engineering discipline utilizes the process of tracking Technical Measures to gain insight into the design and development, to assess risks and issues, and to evaluate the likelihood of realizing objectives. However, with the diversity of so many technical programs, the estimation and risk analysis of technical performance in technology acquisition programs rely on the opinions of experts because the identification and application of relevant quantitative data for constructive modeling is not practical. The Expert-weighted Technical Risk Index methodology proposed in this article introduces a well-established method for mathematically combining expert judgment into the realm of systems engineering to develop predictive progress plans for technical performance estimation and risk analysis. © 2013 Wiley Periodicals, Inc. Syst Eng 17

The Development of Progress Plans Using a Performance-Based Expert Judgment Model to Assess Technical Performance and Risk

Fragility functions are cumulative distribution functions (cdfs) of strengths at failure. They are needed for reliability analyses of systems such as power generation and transmission systems. Subjective opinions supplement sparse test data for estimating fragility functions. Often the opinions are opinions on the percentiles of the fragility function. Subjective percentiles are likely to be less biased than opinions on parameters of cdfs. Solutions to several problems in the estimation of fragility functions are found for subjective percentiles and test data. How subjective percentiles should be used to estimate subjective fragility functions, how subjective percentiles should be combined with test data, how fragility functions for several failure modes should be combined into a composite fragility function, and how inherent randomness and uncertainty due to lack of knowledge should be represented are considered. Subjective percentiles are treated as independent estimates of percentiles. The following are derived: least-squares parameter estimators for normal and lognormal cdfs, based on subjective percentiles (the method is applicable to any invertible cdf); a composite fragility function for combining several failure modes; estimators of variation within and between groups of experts for nonidentically distributed subjective percentiles; weighted least-squares estimators when subjective percentiles have higher variation at higher percents; andmore » weighted least-squares and Bayes parameter estimators based on combining subjective percentiles and test data. 4 figures, 2 tables.« less

/pdf/usng-subjective-percentiles-and-test-data-for-estimating-12efee3gle.pdf

Usng subjective percentiles and test data for estimating fragility functions

The rapidly changing environment and asymmetric threats currently encountered on the modern battlefield requires the timely delivery of effective weapons systems. Unfortunately in fiscal year 2008, according to the US Government Accountability Office, research and development costs of the United States Department of Defense major weapons acquisition programs increased 42 percent above original estimates, and delays in initial operational capability deliveries slipped to 22 months. While there are several quantitative methods to estimate acquisition program cost and schedule performance and to identify their risks (e.g., Earned Value Management), the estimation of technical performance and technical risk identification is generally heuristic in nature and based on expert judgment because of limited quantitative data for constructive modeling. The proposed research in this paper expands upon the Technical Risk Index Distribution method developed by Lewis, Mazzuchi and Sarkani by incorporating a performance-based method of mathematically combining quantified expert opinion for technical performance estimation and risk analysis.

A Performance‐based Statistical Expert Judgment Model to Assess Technical Performance and Risk

: The development of progress plans for each identified technical performance parameter (TPP) is a critical element of technical performance measurement. The measured values of TPPs are referred to as technical performance measures (TPMs). These terms are used interchangeably; however, TPMs more directly reflect how technical progress and technical risk are measured and evaluated. Progress plans, or planned performance profiles, are crucial to effective risk assessment; however, methods for developing these plans are subjective in nature, have no statistical basis or criteria as a rule, and are not sufficiently addressed in literature. The methodology proposed herein for progress plan development will involve the elicitation of expert judgments to formulate probability distributions that reflect the expected values/estimates used to establish progress plans.

Tiffany L Lewis

Papers

A Statistical Approach to the Development of Progress Plans Utilizing Bayesian Methods and Expert Judgment