Evaluating the use of “goodness-of-fit” Measures in hydrologic and hydroclimatic model validation

doi:10.1029/1998WR900018

Home
/
Papers
/
Evaluating the use of “goodness-of-fit” Measures in hydrologic and hydroclimatic model validation

Journal Article•DOI•

Evaluating the use of “goodness-of-fit” Measures in hydrologic and hydroclimatic model validation

01 Jan 1999-Water Resources Research (Wiley-Blackwell)-Vol. 35, Iss: 1, pp 233-241

TL;DR: In this paper, the goodness-of-fit or relative error measures (including the coefficient of efficiency and the index of agreement) that overcome many of the limitations of correlation-based measures are discussed.

read less

Abstract: Correlation and correlation-based measures (e.g., the coefficient of determination) have been widely used to evaluate the “goodness-of-fit” of hydrologic and hydroclimatic models. These measures are oversensitive to extreme values (outliers) and are insensitive to additive and proportional differences between model predictions and observations. Because of these limitations, correlation-based measures can indicate that a model is a good predictor, even when it is not. In this paper, useful alternative goodness-of-fit or relative error measures (including the coefficient of efficiency and the index of agreement) that overcome many of the limitations of correlation-based measures are discussed. Modifications to these statistics to aid in interpretation are presented. It is concluded that correlation and correlation-based measures should not be used to assess the goodness-of-fit of a hydrologic or hydroclimatic model and that additional evaluation measures (such as summary statistics and absolute error measures) should supplement model evaluation tools.

...read moreread less

Citations

PDF

Open Access

More filters

Journal Article•DOI•

Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations

[...]

Daniel N. Moriasi¹, Jeffrey G. Arnold¹, M. W. Van Liew, Ronald L. Bingner¹, R. D. Harmel¹, Tamie L. Veith - Show less +2 more•Institutions (1)

United States Department of Agriculture¹

01 Jan 2007-Transactions of the ASABE

TL;DR: In this paper, the authors present guidelines for watershed model evaluation based on the review results and project-specific considerations, including single-event simulation, quality and quantity of measured data, model calibration procedure, evaluation time step, and project scope and magnitude.

...read moreread less

Abstract: Watershed models are powerful tools for simulating the effect of watershed processes and management on soil and water resources. However, no comprehensive guidance is available to facilitate model evaluation in terms of the accuracy of simulated data compared to measured flow and constituent values. Thus, the objectives of this research were to: (1) determine recommended model evaluation techniques (statistical and graphical), (2) review reported ranges of values and corresponding performance ratings for the recommended statistics, and (3) establish guidelines for model evaluation based on the review results and project-specific considerations; all of these objectives focus on simulation of streamflow and transport of sediment and nutrients. These objectives were achieved with a thorough review of relevant literature on model application and recommended model evaluation methods. Based on this analysis, we recommend that three quantitative statistics, Nash-Sutcliffe efficiency (NSE), percent bias (PBIAS), and ratio of the root mean square error to the standard deviation of measured data (RSR), in addition to the graphical techniques, be used in model evaluation. The following model evaluation performance ratings were established for each recommended statistic. In general, model simulation can be judged as satisfactory if NSE > 0.50 and RSR < 0.70, and if PBIAS + 25% for streamflow, PBIAS + 55% for sediment, and PBIAS + 70% for N and P. For PBIAS, constituent-specific performance ratings were determined based on uncertainty of measured data. Additional considerations related to model evaluation guidelines are also discussed. These considerations include: single-event simulation, quality and quantity of measured data, model calibration procedure, evaluation time step, and project scope and magnitude. A case study illustrating the application of the model evaluation guidelines is also provided.

...read moreread less

9,386 citations

Cites background or methods from "Evaluating the use of “goodness-of-..."

...Several graphical techniques are also described briefly because graphical techniques provide a visual comparison of simulated and measured constituent data and a first overview of model performance (ASCE, 1993) and are essential to appropriate model evaluation (Legates and McCabe, 1999)....
[...]
...Although r and R2 have been widely used for model evaluation, these statistics are oversensitive to high extreme values (outliers) and insensitive to additive and proportional differences between model predictions and measured data (Legates and McCabe, 1999)....
[...]
...The index of agreement can detect additive and proportional differences in the observed and simulated means and variances; however, d is overly sensitive to extreme values due to the squared differences (Legates and McCabe, 1999)....
[...]
...A number of publications have addressed model evaluation statistics (Willmott, 1981; ASCE, 1993; Legates and McCabe, 1999), but they do not include recently developed statistics (e....
[...]
...50(3): 885−900 a relative model evaluation assessment, and error indices quantify the deviation in the units of the data of interest (Legates and McCabe, 1999)....
[...]

Journal Article•DOI•

Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling

[...]

Hoshin V. Gupta¹, Harald Kling¹, Koray K. Yilmaz¹, Guillermo F. Martinez¹•Institutions (1)

University of Arizona¹

20 Oct 2009-Journal of Hydrology

TL;DR: A diagnostically interesting decomposition of NSE is presented, which facilitates analysis of the relative importance of its different components in the context of hydrological modelling, and it is shown how model calibration problems can arise due to interactions among these components.

...read moreread less

3,147 citations

Cites background or methods from "Evaluating the use of “goodness-of-..."

...…of model skill, there has been a long and vivid discussion about the suitability of NSE (McCuen and Snyder, 1975; Martinec and Rango, 1989; Legates and McCabe, 1999; Krause et al., 2005; McCuen et al., 2006; Schaefli and Gupta, 2007; Jain and Sudheer, 2008) and several authors have…...
[...]
...For such situations, various authors have recommended the use of the seasonal or climatological mean as a baseline model (Garrick et al., 1978; Murphy, 1988; Martinec and Rango, 1989; Legates and McCabe, 1999; Schaefli and Gupta, 2007)....
[...]
...…or reporting results based on calibration with NSE, information about the correlation, bias, and variability of flows should also be given (interestingly, this was already proposed by Legates and McCabe (1999), although they did not discuss the interrelation between NSE and its three components)....
[...]

Journal Article•DOI•

Physiographically sensitive mapping of climatological temperature and precipitation across the conterminous United States

[...]

Christopher Daly¹, Michael D. Halbleib¹, Joseph I. Smith¹, Wayne Gibson¹, Matthew K. Doggett¹, George H. Taylor¹, Jan Curtis², Phillip P. Pasteris² - Show less +4 more•Institutions (2)

Oregon State University¹, Natural Resources Conservation Service²

01 Dec 2008-International Journal of Climatology

TL;DR: In this paper, the authors used the PRISM (Parameter-elevation relationships on independent slopes model) interpolation method to develop data sets that reflected, as closely as possible, the current state of knowledge of spatial climate patterns in the United States.

...read moreread less

Abstract: Spatial climate data sets of 1971–2000 mean monthly precipitation and minimum and maximum temperature were developed for the conterminous United States These 30-arcsec (∼800-m) grids are the official spatial climate data sets of the US Department of Agriculture The PRISM (Parameter-elevation Relationships on Independent Slopes Model) interpolation method was used to develop data sets that reflected, as closely as possible, the current state of knowledge of spatial climate patterns in the United States PRISM calculates a climate–elevation regression for each digital elevation model (DEM) grid cell, and stations entering the regression are assigned weights based primarily on the physiographic similarity of the station to the grid cell Factors considered are location, elevation, coastal proximity, topographic facet orientation, vertical atmospheric layer, topographic position, and orographic effectiveness of the terrain Surface stations used in the analysis numbered nearly 13 000 for precipitation and 10 000 for temperature Station data were spatially quality controlled, and short-period-of-record averages adjusted to better reflect the 1971–2000 period PRISM interpolation uncertainties were estimated with cross-validation (C-V) mean absolute error (MAE) and the 70% prediction interval of the climate–elevation regression function The two measures were not well correlated at the point level, but were similar when averaged over large regions The PRISM data set was compared with the WorldClim and Daymet spatial climate data sets The comparison demonstrated that using a relatively dense station data set and the physiographically sensitive PRISM interpolation process resulted in substantially improved climate grids over those of WorldClim and Daymet The improvement varied, however, depending on the complexity of the region Mountainous and coastal areas of the western United States, characterized by sparse data coverage, large elevation gradients, rain shadows, inversions, cold air drainage, and coastal effects, showed the greatest improvement The PRISM data set benefited from a peer review procedure that incorporated local knowledge and data into the development process Copyright © 2008 Royal Meteorological Society

...read moreread less

2,447 citations

Cites background from "Evaluating the use of “goodness-of-..."

...Willmott et al., 1985; Legates and McCabe, 1999 )....
[...]

Journal Article•DOI•

Comparison of different efficiency criteria for hydrological model assessment

[...]

Peter Krause¹, Douglas P. Boyle², F. Bäse¹•Institutions (2)

University of Jena¹, Desert Research Institute²

16 Dec 2005-Advances in Geosciences

TL;DR: In this paper, the utility of several efficiency criteria is investigated in three examples using a simple observed streamflow hydrograph, and the selection and use of specific efficiency criteria and interpretation of the results can be a challenge for even the most experienced hydrologist since each criterion may place different emphasis on different types of simulated and observed behaviours.

...read moreread less

Abstract: . The evaluation of hydrologic model behaviour and performance is commonly made and reported through comparisons of simulated and observed variables. Frequently, comparisons are made between simulated and measured streamflow at the catchment outlet. In distributed hydrological modelling approaches, additional comparisons of simulated and observed measurements for multi-response validation may be integrated into the evaluation procedure to assess overall modelling performance. In both approaches, single and multi-response, efficiency criteria are commonly used by hydrologists to provide an objective assessment of the "closeness" of the simulated behaviour to the observed measurements. While there are a few efficiency criteria such as the Nash-Sutcliffe efficiency, coefficient of determination, and index of agreement that are frequently used in hydrologic modeling studies and reported in the literature, there are a large number of other efficiency criteria to choose from. The selection and use of specific efficiency criteria and the interpretation of the results can be a challenge for even the most experienced hydrologist since each criterion may place different emphasis on different types of simulated and observed behaviours. In this paper, the utility of several efficiency criteria is investigated in three examples using a simple observed streamflow hydrograph.

...read moreread less

2,375 citations

Cites background or methods from "Evaluating the use of “goodness-of-..."

...As a result larger values in a time series are strongly overestimated whereas lower values are neglected (Legates and McCabe, 1999)....
[...]
...4.1 Results of example 1 In example 1, the value of the coefficient of determination r2, is 1.0 while the value of the weighted coefficient,wr2, is 0.7 reflecting the poor simulation better thanr2 alone....
[...]
...2.3 Index of agreementd The index of agreementd was proposed by Willmot (1981) to overcome the insensitivity ofE and r2 to differences in the observed and predicted means and variances (Legates and McCabe, 1999)....
[...]
...…the arithmetic mean of the entire observed; and in the remaining model simulations (3 to 136), the observed hydrograph values were progressively substituted for the arithmetic mean of the entire observed hydrograph until the last model simulation (number 136) was the actual observed hydrograph....
[...]

Journal Article•DOI•

SWAT: Model Use, Calibration, and Validation

[...]

Jeffrey G. Arnold¹, Daniel N. Moriasi¹, Philip W. Gassman², Karim C. Abbaspour³, Michael J. White¹, Raghavan Srinivasan⁴, C. Santhi, R. D. Harmel¹, A. van Griensven⁵, M. W. Van Liew⁶, Narayanan Kannan, Manoj Jha⁷ - Show less +8 more•Institutions (7)

United States Department of Agriculture¹, Iowa State University², Swiss Federal Institute of Aquatic Science and Technology³, Texas A&M University⁴, UNESCO-IHE Institute for Water Education⁵, University of Nebraska–Lincoln⁶, North Carolina Agricultural and Technical State University⁷

01 Jan 2012-Transactions of the ASABE

TL;DR: The SWAT-CUP tool as discussed by the authors is a semi-distributed river basin model that requires a large number of input parameters, which complicates model parameterization and calibration, and is used to provide statistics for goodness-of-fit.

...read moreread less

Abstract: SWAT (Soil and Water Assessment Tool) is a comprehensive, semi-distributed river basin model that requires a large number of input parameters, which complicates model parameterization and calibration. Several calibration techniques have been developed for SWAT, including manual calibration procedures and automated procedures using the shuffled complex evolution method and other common methods. In addition, SWAT-CUP was recently developed and provides a decision-making framework that incorporates a semi-automated approach (SUFI2) using both manual and automated calibration and incorporating sensitivity and uncertainty analysis. In SWAT-CUP, users can manually adjust parameters and ranges iteratively between autocalibration runs. Parameter sensitivity analysis helps focus the calibration and uncertainty analysis and is used to provide statistics for goodness-of-fit. The user interaction or manual component of the SWAT-CUP calibration forces the user to obtain a better understanding of the overall hydrologic processes (e.g., baseflow ratios, ET, sediment sources and sinks, crop yields, and nutrient balances) and of parameter sensitivity. It is important for future calibration developments to spatially account for hydrologic processes; improve model run time efficiency; include the impact of uncertainty in the conceptual model, model parameters, and measured variables used in calibration; and assist users in checking for model errors. When calibrating a physically based model like SWAT, it is important to remember that all model input parameters must be kept within a realistic uncertainty range and that no automatic procedure can substitute for actual physical knowledge of the watershed.

...read moreread less

2,200 citations

Cites methods from "Evaluating the use of “goodness-of-..."

..., 1997); (2) multiple evaluation techniques (ASCE, 1993; Legates and McCabe, 1999; Boyle et al., 2000); (3) calibrating all constituents to be evaluated; and (4) verification that other important model outputs are reasonable....
[...]

1
2
3
4
…
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200

Collapse

References

PDF

Open Access

More filters

Journal Article•DOI•

River flow forecasting through conceptual models part I — A discussion of principles☆

[...]

J.E. Nash¹, J.V. Sutcliffe¹•Institutions (1)

National University of Ireland, Galway¹

01 Apr 1970-Journal of Hydrology

TL;DR: In this article, the principles governing the application of the conceptual model technique to river flow forecasting are discussed and the necessity for a systematic approach to the development and testing of the model is explained and some preliminary ideas suggested.

...read moreread less

19,601 citations

Journal Article•DOI•

An approach toward a rational classification of climate.

[...]

C. W. Thornthwaite

01 Jan 1948-Geographical Review

6,391 citations

Journal Article•DOI•

On the validation of models

[...]

Cort J. Willmott¹•Institutions (1)

University of Delaware¹

01 Jul 1981-Physical Geography

TL;DR: In this paper, it is suggested that the correlation coefficieness between observed and simulated variates is not as good as observed variates, and that correlation can be improved.

...read moreread less

Abstract: Traditional methods of evaluating geographic models by statistical comparisons between observed and simulated variates are criticized. In particular, it is suggested that the correlation coefficien...

...read moreread less

3,761 citations

Journal Article•DOI•

A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation

[...]

Bradley Efron¹, Gail Gong²•Institutions (2)

Stanford University¹, Carnegie Mellon University²

01 Feb 1983-The American Statistician

TL;DR: This paper reviewed the nonparametric estimation of statistical error, mainly the bias and standard error of an estimator, or the error rate of a prediction rule, at a relaxed mathematical level, omitting most proofs, regularity conditions and technical details.

...read moreread less

Abstract: This is an invited expository article for The American Statistician. It reviews the nonparametric estimation of statistical error, mainly the bias and standard error of an estimator, or the error rate of a prediction rule. The presentation is written at a relaxed mathematical level, omitting most proofs, regularity conditions, and technical details.

...read moreread less

3,146 citations

Journal Article•DOI•

An Approach Toward a Rational Classification of Climate

[...]

C. W. Thornthwaite

01 Jul 1948-Soil Science

2,200 citations