Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences

doi:10.1126/SCIENCE.263.5147.641

Home
/
Papers
/
Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences

Journal Article•DOI•

Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences

Naomi Oreskes¹, Kristin Shrader-Frechette², Kenneth Belitz¹•Institutions (2)

Dartmouth College¹, University of South Florida²

04 Feb 1994-Science (American Association for the Advancement of Science)-Vol. 263, Iss: 5147, pp 641-646

TL;DR: Verification and validation of numerical models of natural systems is impossible because natural systems are never closed and because model results are always nonunique.

read less

Abstract: Verification and validation of numerical models of natural systems is impossible. This is because natural systems are never closed and because model results are always nonunique. Models can be confirmed by the demonstration of agreement between observation and prediction, but confirmation is inherently partial. Complete confirmation is logically precluded by the fallacy of affirming the consequent and by incomplete access to natural phenomena. Models can only be evaluated in relative terms, and their predictive value is always open to question. The primary value of models is heuristic.

...read moreread less

Citations

PDF

Open Access

More filters

Journal Article•DOI•

Predictive habitat distribution models in ecology

[...]

Antoine Guisan, Niklaus E. Zimmermann

05 Dec 2000-Ecological Modelling

TL;DR: A review of predictive habitat distribution modeling is presented, which shows that a wide array of models has been developed to cover aspects as diverse as biogeography, conservation biology, climate change research, and habitat or species management.

...read moreread less

6,748 citations

Cites background or methods from "Verification, Validation, and Confi..."

...verification, calibration, evaluation (= validation), qualification) have been covered in more specific papers, with very special attention given in recent years to evaluation and its usefulness for testing ecological models (e.g. Loehle, 1983; Oreskes et al., 1994; Rykiel, 1996)....
[...]
...…development (e.g. verification, calibration, evaluation (= validation), qualification) have been covered in more specific papers, with very special attention given in recent years to evaluation and its usefulness for testing ecological models (e.g. Loehle, 1983; Oreskes et al., 1994; Rykiel, 1996)....
[...]
...Loehle (1983), Oreskes et al. (1994) and Rykiel (1996) discuss the use of the term 6alidation when measuring the adequacy between model predictions and field observations, what is called accuracy assessment in remote sensing studies....
[...]
...To analyze the predictive success of models, we propose to follow Oreskes et al. (1994) and use the term e6aluation....
[...]
...However, saying that a model is ‘good’ or ‘bad’ is subject to critics, because it is implicit in modeling that perfect truth cannot be attained (Oreskes et al., 1994)....
[...]

Journal Article•DOI•

A bonded-particle model for rock

[...]

D.O. Potyondy¹, Peter Cundall•Institutions (1)

University of Toronto¹

01 Dec 2004-International Journal of Rock Mechanics and Mining Sciences

TL;DR: In this paper, a numerical model for rock is proposed in which the rock is represented by a dense packing of non-uniform-sized circular or spherical particles that are bonded together at their contact points and whose mechanical behavior is simulated by the distinct element method using the two-and three-dimensional discontinuum programs PFC2D and PFC3D.

...read moreread less

3,470 citations

Cites methods from "Verification, Validation, and Confi..."

...No model is complete or fully verifiable [54], but the validity of the BPM is demonstrated by comparing model behavior with measured and observed responses of Lac du Bonnet granite at both laboratory and field scales in this and the next section....
[...]

Journal Article•DOI•

AUC: a misleading measure of the performance of predictive distribution models

[...]

Jorge M. Lobo¹, Alberto Jiménez-Valverde¹, Raimundo Real²•Institutions (2)

Spanish National Research Council¹, University of Málaga²

01 Mar 2008-Global Ecology and Biogeography

TL;DR: The area under the receiver operating characteristic (ROC) curve, known as the AUC, is currently considered to be the standard method to assess the accuracy of predictive distribution models as discussed by the authors.

...read moreread less

Abstract: The area under the receiver operating characteristic (ROC) curve, known as the AUC, is currently considered to be the standard method to assess the accuracy of predictive distribution models. It avoids the supposed subjectivity in the threshold selection process, when continuous probability derived scores are converted to a binary presence‐absence variable, by summarizing overall model performance over all possible thresholds. In this manuscript we review some of the features of this measure and bring into question its reliability as a comparative measure of accuracy between model results. We do not recommend using AUC for five reasons: (1) it ignores the predicted probability values and the goodness-of-fit of the model; (2) it summarises the test performance over regions of the ROC space in which one would rarely operate; (3) it weights omission and commission errors equally; (4) it does not give information about the spatial distribution of model errors; and, most importantly, (5) the total extent to which models are carried out highly influences the rate of well-predicted absences and the AUC scores.

...read moreread less

2,711 citations

Cites background or methods from "Verification, Validation, and Confi..."

...This paper was supported by a Fundación BBVA project (Diseño de una red de reservas para la protección de la Biodiversidad en América Austral), and two MEC Projects (CGL2004-04309 and CGL2006-09567/BOS)....
[...]
...Strictly speaking, only closed models can be validated (Oreskes et al ., 1994)....
[...]

Journal Article•DOI•

Recent developments in Life Cycle Assessment.

[...]

Göran Finnveden¹, Michael Zwicky Hauschild², Tomas Ekvall, Jeroen B. Guinée³, Reinout Heijungs³, Stefanie Hellweg⁴, Annette Koehler⁴, David Pennington, Sangwon Suh - Show less +5 more•Institutions (4)

Royal Institute of Technology¹, Technical University of Denmark², Leiden University³, ETH Zurich⁴

01 Oct 2009-Journal of Environmental Management

TL;DR: A review of recent developments of LCA methods, focusing on some areas where there has been an intense methodological development during the last years, and some of the emerging issues.

...read moreread less

2,683 citations

Cites background from "Verification, Validation, and Confi..."

...To Oreskes et al. (1994) and Heijungs (2001), the only possible validation is the piecemeal one: unit processes, steps in impact pathways, etc., each building block may be validated separately, and as long as the gluing together proceeds according to strict procedures and mathematical rules, we can…...
[...]

Journal Article•DOI•

Catastrophic regime shifts in ecosystems: linking theory to observation

[...]

Marten Scheffer¹, Stephen R. Carpenter²•Institutions (2)

Wageningen University and Research Centre¹, University of Wisconsin-Madison²

01 Dec 2003-Trends in Ecology and Evolution

TL;DR: In this paper, the authors review emerging ways to link theory to observation, and conclude that although, field observations can provide hints of alternative stable states, experiments and models are essential for a good diagnosis.

...read moreread less

Abstract: Occasionally, surprisingly large shifts occur in ecosystems. Theory suggests that such shifts can be attributed to alternative stable states. Verifying this diagnosis is important because it implies a radically different view on management options, and on the potential effects of global change on such ecosystems. For instance, it implies that gradual changes in temperature or other factors might have little effect until a threshold is reached at which a large shift occurs that might be difficult to reverse. Strategies to assess whether alternative stable states are present are now converging in fields as disparate as desertification, limnology, oceanography and climatology. Here, we review emerging ways to link theory to observation, and conclude that although, field observations can provide hints of alternative stable states, experiments and models are essential for a good diagnosis.

...read moreread less

2,464 citations

1
2
3
4
…
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200

Collapse

References

PDF

Open Access

More filters

Book•

The Logic of Scientific Discovery

[...]

Karl Popper

01 Jan 1934

TL;DR: The Open Society and Its Enemies as discussed by the authors is regarded as one of Popper's most enduring books and contains insights and arguments that demand to be read to this day, as well as many of the ideas in the book.

...read moreread less

Abstract: Described by the philosopher A.J. Ayer as a work of 'great originality and power', this book revolutionized contemporary thinking on science and knowledge. Ideas such as the now legendary doctrine of 'falsificationism' electrified the scientific community, influencing even working scientists, as well as post-war philosophy. This astonishing work ranks alongside The Open Society and Its Enemies as one of Popper's most enduring books and contains insights and arguments that demand to be read to this day.

...read moreread less

7,904 citations

Book•

The scientific image

[...]

William Demopoulos, Bas C. van Fraassen

01 Jan 1980

TL;DR: In this book van Fraassen develops an alternative to scientific realism by constructing and evaluating three mutually reinforcing theories.

...read moreread less

Abstract: In this book van Fraassen develops an alternative to scientific realism by constructing and evaluating three mutually reinforcing theories.

...read moreread less

3,468 citations

Journal Article•DOI•

Criticism and the Growth of Knowledge

[...]

Imre Lakatos, Alan Musgrave

01 Mar 1972-Philosophy of Science

TL;DR: The distinction between normal and revolutionary science hold water as mentioned in this papereyerabend, T. S. Kuhn and T. E. Toulmin have made a distinction between the two categories of science.

...read moreread less

Abstract: Preface Note on the third impression 1. Logic of discovery of psychology of research? T. S. Kuhn 2. Against 'Normal Science' J. W. N. Watkins 3. Does the distinction between normal and revolutionary science hold water? S. E. Toulmin 4. Normal science, scientific revolutions and the history of science L. Pearce Williams 5. Normal science and its dangers K. R. Popper 6. The nature of a paradigm Margaret Masterman 7. Falsification and the methodology of scientific research programmes I. Lakatos 8. Consolations for the specialist P. K. Feyerabend 9. Reflections on my critics T. S. Kuhn Index.

...read moreread less

3,434 citations

The scientific image

[...]

van Fraassen, C Bas

01 Jan 1980

3,226 citations

Journal Article•DOI•

Aspects of scientific explanation

[...]

Michael D. Resnik, Carl G. Hempel

01 Sep 1966-Philosophy and Phenomenological Research