On the role of code comparisons in verification and validation.

TLDR: The use of code comparisons for validation is improper and dangerous, and while code comparisons may be argued to provide a beneficial component in code verification activities, there are higher quality code verification tasks that should take precedence.

Abstract: This report presents a perspective on the role of code comparison activities in verification and validation. We formally define the act of code comparison as the Code Comparison Principle (CCP) and investigate its application in both verification and validation. One of our primary conclusions is that the use of code comparisons for validation is improper and dangerous. We also conclude that while code comparisons may be argued to provide a beneficial component in code verification activities, there are higher quality code verification tasks that should take precedence. Finally, we provide a process for application of the CCP that we believe is minimal for achieving benefit in verification processes.

Figures

Figure 5.1 Key elements of a defensible code comparison process.

Figure 5.2 The multiplicity of challenges in using the CCP.

Full text

SANDIA REPORT
SAND2003-2752
Unlimited Release
Printed August 2003
On the Role of Code Comparisons in Verification
and Validation
Timothy G. Trucano, Martin Pilch, and William L. Oberkampf
Prepared by
Sandia National Laboratories
Albuquerque, New Mexico 87185 and Livermore, California 94550
Sandia is a multiprogram laboratory operated by Sandia Corporation,
a Lockheed Martin Company, for the United States Department of
Energy under Contract DE-AC04-94AL85000
Approved for public release; further dissemination unlimited.

Issued by Sandia National Laboratories, operated for the United States
Department of Energy by Sandia Corporation.
NOTICE: This report was prepared as an account of work sponsored by an
agency of the United States Government. Neither the United States Government
nor any agency thereof, nor any of their employees, nor any of their contractors,
subcontractors, or their employees, makes any warranty, express or implied, or
assumes any legal liability or responsibility for the accuracy, completeness, or
usefulness of any information, apparatus, product, or process disclosed, or
represents that its use would not infringe privately owned rights. Reference
herein to any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise, does not necessarily constitute or imply
its endorsement, recommendation, or favoring by the United States Government,
any agency thereof or any of their contractors or subcontractors. The views and
opinions expressed herein do not necessarily state or reflect those of the United
States Government, any agency thereof or any of their contractors.
Printed in the United States of America. This report has been reproduced directly
from the best available copy.
Available to DOE and DOE contractors from
U.S. Department of Energy
Office of Scientific and Technical Information
P.O. Box 62
Oak Ridge, TN 37831
Telephone: (865)576-8401
Facsimile: (865)576-5782
E-Mail: reports@adonis.osti.gov
Online ordering: http://www.doe.gov/bridge
Available to the public from
U.S. Department of Commerce
National Technical Information Service
5285 Port Royal Rd
Springfield, VA 22161
Telephone: (800)553-6847
Facsimile: (703)605-6900
E-Mail: orders@ntis.fedworld.gov
Online order: http://www.ntis.gov/ordering.htm

3
SAND2003-2752
Unlimited Release
Printed August 2003
On the Role of Code Comparisons in
Verification and Validation
Timothy G. Trucano
Optimization and Uncertainty Estimation
Martin Pilch and William L. Oberkampf
Validation and Uncertainty Quantification
Sandia National Laboratories
P. O. Box 5800
Albuquerque, New Mexico 87185-0819
Abstract
This report presents a perspective on the role of code comparison activities in
verification and validation. We formally define the act of code comparison as the
Code Comparison Principle (CCP) and investigate its application in both verification
and validation. One of our primary conclusions is that the use of code comparisons
for validation is improper and dangerous. We also conclude that while code
comparisons may be argued to provide a beneficial component in code verification
activities, there are higher quality code verification tasks that should take precedence.
Finally, we provide a process for application of the CCP that we believe is minimal
for achieving benefit in verification processes.

4
Acknowledgements
The authors thank David Peercy, Steve Lott and Mark Christon of Sandia National
Laboratories for reviewing a draft of this report.

5
Contents
Acknowledgements ............................................................................................................. 4
Section 1 Introduction ........................................................................................................ 9
Section 2 Formality of Code Comparisons ...................................................................... 11
Section 3 Verification Using the Code Comparison Principle........................................ 13
3.1 Code Verification................................................................................................. 14
3.2 Solution Error Estimation and Accuracy Verification ......................................... 16
Section 4 Validation Using the Code Comparison Principle ........................................... 19
Section 5 A Process for the Code Comparison Principle................................................. 21
Section 6 Potentially Useful Code Comparison Activities .............................................. 27
Section 7 Conclusions ...................................................................................................... 31
References ......................................................................................................................... 33

Frequently asked questions

Q1. What are the contributions mentioned in the paper "Sand2003-2752 " ?

This report presents a perspective on the role of code comparison activities in verification and validation. The authors formally define the act of code comparison as the Code Comparison Principle ( CCP ) and investigate its application in both verification and validation. The authors also conclude that while code comparisons may be argued to provide a beneficial component in code verification activities, there are higher quality code verification tasks that should take precedence. Finally, the authors provide a process for application of the CCP that they believe is minimal for achieving benefit in verification processes. 

Q2. What is the chance for understanding numerical error?

Their best chance for understanding numerical error is for test problems that are too simple to convincingly extrapolate to real applications. 

Q3. Why is the CCP considered to have significant power?

Because complex test problems do not have analytic solutions, this is the area where use of the CCP is believed to have significant power. 

Q4. What is the second myth that is ever present?

A second myth that is ever present is the belief that Code2 embodies wide physical modeling experience and understanding, thus allowing Code1 validation to be placed on the same kind of discount plan through operation of the CCP. 

Q5. What is the general position on the issue of code comparisons?

Their general position on the issue of code comparisons is that code comparisons would be performed only as part of a larger program of independent verification tasks, such as application of software quality engineering (SQE) methodologies, algorithm testing procedures, verification test suites, and comparison with analytical solutions (see Oberkampf and Trucano, 2002; Oberkampf, Trucano and Hirsch, 2002). 

Q6. What do you think of the concept of code comparisons?

The authors believe that code comparisons should only be performed as a structured part of a spectrum of verification tasks, so that there is a significant body of evidence for verification in addition to only having code comparisons.