Construction of correlation functions in two and three dimensions
01 Jan 1999-Quarterly Journal of the Royal Meteorological Society (John Wiley & Sons, Ltd)-Vol. 125, Iss: 554, pp 723-757
TL;DR: In this paper, the authors focus on the construction of simply parametrized covariance functions for data-assimilation applications and provide a self-contained, rigorous mathematical summary of relevant topics from correlation theory.
Abstract: This article focuses on the construction, directly in physical space, of simply parametrized covariance functions for data-assimilation applications. A self-contained, rigorous mathematical summary of relevant topics from correlation theory is provided as a foundation for this construction. Covariance and correlation functions are defined, and common notions of homogeneity and isotropy are clarified. Classical results are stated, and proven where instructive. Included are smoothness properties relevant to multivariate statistical-analysis algorithms where wind/wind and wind/mass correlation models are obtained by differentiating the correlation model of a mass variable. the Convolution Theorem is introduced as the primary tool used to construct classes of covariance and cross-covariance functions on three-dimensional Euclidean space R3. Among these are classes of compactly supported functions that restrict to covariance and cross-covariance functions on the unit sphere S2, and that vanish identically on subsets of positive measure on S2. It is shown that these covariance and cross-covariance functions on S2, referred to as being space-limited, cannot be obtained using truncated spectral expansions. Compactly supported and space-limited covariance functions determine sparse covariance matrices when evaluated on a grid, thereby easing computational burdens in atmospheric data-analysis algorithms.
Convolution integrals leading to practical examples of compactly supported covariance and cross-covariance functions on R3 are reduced and evaluated. More specifically, suppose that gi and gj are radially symmetric functions defined on R3 such that gi(x) = 0 for |x| > di and gj(x) = 0 for |xv > dj, O di + dj and |x - y| > 2di, respectively, Additional covariance functions on R3 are constructed using convolutions over the real numbers R, rather than R3. Families of compactly supported approximants to standard second- and third-order autoregressive functions are constructed as illustrative examples. Compactly supported covariance functions of the form C(x,y) := Co(|x - y|), x,y ∈ R3, where the functions Co(r) for r ∈ R are 5th-order piecewise rational functions, are also constructed. These functions are used to develop space-limited product covariance functions B(x, y) C(x, y), x, y ∈ S2, approximating given covariance functions B(x, y) supported on all of S2 × S2.
Citations
More filters
Earth System Research Laboratory1, University of Colorado Boulder2, Met Office3, National Oceanic and Atmospheric Administration4, ETH Zurich5, University of Bern6, University of East Anglia7, Rovira i Virgili University8, University Corporation for Atmospheric Research9, South African Weather Service10, British Antarctic Survey11, University of Milan12, Norwegian Meteorological Institute13, University of Lisbon14, Environment Canada15, National Center for Atmospheric Research16
TL;DR: The Twentieth Century Reanalysis (20CR) dataset as discussed by the authors provides the first estimates of global tropospheric variability, and of the dataset's time-varying quality, from 1871 to the present at 6-hourly temporal and 2° spatial resolutions.
Abstract: The Twentieth Century Reanalysis (20CR) project is an international effort to produce a comprehensive global atmospheric circulation dataset spanning the twentieth century, assimilating only surface pressure reports and using observed monthly sea-surface temperature and sea-ice distributions as boundary conditions. It is chiefly motivated by a need to provide an observational dataset with quantified uncertainties for validations of climate model simulations of the twentieth century on all time-scales, with emphasis on the statistics of daily weather. It uses an Ensemble Kalman Filter data assimilation method with background ‘first guess’ fields supplied by an ensemble of forecasts from a global numerical weather prediction model. This directly yields a global analysis every 6 hours as the most likely state of the atmosphere, and also an uncertainty estimate of that analysis.
The 20CR dataset provides the first estimates of global tropospheric variability, and of the dataset's time-varying quality, from 1871 to the present at 6-hourly temporal and 2° spatial resolutions. Intercomparisons with independent radiosonde data indicate that the reanalyses are generally of high quality. The quality in the extratropical Northern Hemisphere throughout the century is similar to that of current three-day operational NWP forecasts. Intercomparisons over the second half-century of these surface-based reanalyses with other reanalyses that also make use of upper-air and satellite data are equally encouraging.
It is anticipated that the 20CR dataset will be a valuable resource to the climate research community for both model validations and diagnostic studies. Some surprising results are already evident. For instance, the long-term trends of indices representing the North Atlantic Oscillation, the tropical Pacific Walker Circulation, and the Pacific–North American pattern are weak or non-existent over the full period of record. The long-term trends of zonally averaged precipitation minus evaporation also differ in character from those in climate model simulations of the twentieth century. Copyright © 2011 Royal Meteorological Society and Crown Copyright.
3,043 citations
TL;DR: In this paper, an ensemble adjustment Kalman filter is proposed to estimate the probability distribution of the state of a model given a set of observations using Monte Carlo approximations to the nonlinear filter.
Abstract: A theory for estimating the probability distribution of the state of a model given a set of observations exists. This nonlinear filtering theory unifies the data assimilation and ensemble generation problem that have been key foci of prediction and predictability research for numerical weather and ocean prediction applications. A new algorithm, referred to as an ensemble adjustment Kalman filter, and the more traditional implementation of the ensemble Kalman filter in which “perturbed observations” are used, are derived as Monte Carlo approximations to the nonlinear filter. Both ensemble Kalman filter methods produce assimilations with small ensemble mean errors while providing reasonable measures of uncertainty in the assimilated variables. The ensemble methods can assimilate observations with a nonlinear relation to model state variables and can also use observations to estimate the value of imprecisely known model parameters. These ensemble filter methods are shown to have significant advantag...
1,660 citations
TL;DR: In this paper, the EnSRF algorithm is proposed, which uses the traditional Kalman gain for updating the ensemble mean but uses a reduced loss to update deviations from the EnKF mean.
Abstract: The ensemble Kalman filter (EnKF) is a data assimilation scheme based on the traditional Kalman filter update equation. An ensemble of forecasts are used to estimate the background-error covariances needed to compute the Kalman gain. It is known that if the same observations and the same gain are used to update each member of the ensemble, the ensemble will systematically underestimate analysis-error covariances. This will cause a degradation of subsequent analyses and may lead to filter divergence. For large ensembles, it is known that this problem can be alleviated by treating the observations as random variables, adding random perturbations to them with the correct statistics. Two important consequences of sampling error in the estimate of analysis-error covariances in the EnKF are discussed here. The first results from the analysis-error covariance being a nonlinear function of the backgrounderror covariance in the Kalman filter. Due to this nonlinearity, analysis-error covariance estimates may be negatively biased, even if the ensemble background-error covariance estimates are unbiased. This problem must be dealt with in any Kalman filter‐based ensemble data assimilation scheme. A second consequence of sampling error is particular to schemes like the EnKF that use perturbed observations. While this procedure gives asymptotically correct analysis-error covariance estimates for large ensembles, the addition of perturbed observations adds an additional source of sampling error related to the estimation of the observation-error covariances. In addition to reducing the accuracy of the analysis-error covariance estimate, this extra source of sampling error increases the probability that the analysis-error covariance will be underestimated. Because of this, ensemble data assimilation methods that use perturbed observations are expected to be less accurate than those which do not. Several ensemble filter formulations have recently been proposed that do not require perturbed observations. This study examines a particularly simple implementation called the ensemble square root filter, or EnSRF. The EnSRF uses the traditional Kalman gain for updating the ensemble mean but uses a ‘‘reduced’’ Kalman gain to update deviations from the ensemble mean. There is no additional computational cost incurred by the EnSRF relative to the EnKF when the observations have independent errors and are processed one at a time. Using a hierarchy of perfect model assimilation experiments, it is demonstrated that the elimination of the sampling error associated with the perturbed observations makes the EnSRF more accurate than the EnKF for the same ensemble size.
1,477 citations
Cites background from "Construction of correlation functio..."
...4.10 in Gaspari and Cohn (1999), while not strictly constraining the ensemble to satisfy Eq....
[...]
TL;DR: In this paper, a distance-dependent reduction of background error covariance estimates in an ensemble Kalman filter is demonstrated, by performing an elementwise multiplication of the background covariance matrix with a correlation function with local support.
Abstract: The usefulness of a distance-dependent reduction of background error covariance estimates in an ensemble Kalman filter is demonstrated. Covariances are reduced by performing an elementwise multiplication of the background error covariance matrix with a correlation function with local support. This reduces noisiness and results in an improved background error covariance estimate, which generates a reduced-error ensemble of model initial conditions. The benefits of applying the correlation function can be understood in part from examining the characteristics of simple 2 3 2 covariance matrices generated from random sample vectors with known variances and covariance. These show that noisiness in covariance estimates tends to overwhelm the signal when the ensemble size is small and/or the true covariance between the sample elements is small. Since the true covariance of forecast errors is generally related to the distance between grid points, covariance estimates generally have a higher ratio of noise to signal with increasing distance between grid points. This property is also demonstrated using a twolayer hemispheric primitive equation model and comparing covariance estimates generated by small and large ensembles. Covariances from the large ensemble are assumed to be accurate and are used a reference for measuring errors from covariances estimated from a small ensemble. The benefits of including distance-dependent reduction of covariance estimates are demonstrated with an ensemble Kalman filter data assimilation scheme. The optimal correlation length scale of the filter function depends on ensemble size; larger correlation lengths are preferable for larger ensembles. The effects of inflating background error covariance estimates are examined as a way of stabilizing the filter. It was found that more inflation was necessary for smaller ensembles than for larger ensembles.
1,084 citations
Cites methods from "Construction of correlation functio..."
...We applied the 5th-order function in Gaspari and Cohn (1999) as discussed in section 3c to an EnKF with 25, 100, and 400 members....
[...]
...To define the correlation matrix S, we used a fifthorder function of Gaspari and Cohn (1999), which is similar to a Gaussian function in shape but compactly supported, that is, correlations decreased to zero at a finite radius....
[...]
...To define the correlation matrix S, we used a 5thorder function of Gaspari and Cohn (1999), which is similar in shape to the Gaussian function, but tapers to zero at a finite distance. The controlling parameter is lc, a correlation length scale for the function. See Hamill et al. (2001) for more description....
[...]
...The filtering used a fifth-order function in Gaspari and Cohn (1999) as discussed in section 3c. a. Analysis errors as function of filter length scale Figures 6a,b present the time average ensemble mean error for the sparse network (46 observation locations; Fig....
[...]
...To define the correlation matrix S, we used a 5thorder function of Gaspari and Cohn (1999), which is similar in shape to the Gaussian function, but tapers to zero at a finite distance....
[...]
TL;DR: It is shown that tapering the correct covariance matrix with an appropriate compactly supported positive definite function reduces the computational burden significantly and still leads to an asymptotically optimal mean squared error.
Abstract: Interpolation of a spatially correlated random process is used in many scientific areas. The best unbiased linear predictor, often called a kriging predictor in geostatistical science, requires the solution of a (possibly large) linear system based on the covariance matrix of the observations. In this article, we show that tapering the correct covariance matrix with an appropriate compactly supported positive definite function reduces the computational burden significantly and still leads to an asymptotically optimal mean squared error. The effect of tapering is to create a sparse approximate linear system that can then be solved using sparse matrix algorithms. Monte Carlo simulations support the theoretical results. An application to a large climatological precipitation dataset is presented as a concrete and practical illustration.
757 citations
Cites background from "Construction of correlation functio..."
...Indeed, a very effective use of covariance tapering is well known in the data assimilation literature for numerical weather prediction (Gaspari and Cohn 1999)....
[...]
...Wu (1995), Wendland (1995, 1998), Gaspari and Cohn (1999), and Gneiting (2002) gave several procedures to construct compactly supported covariance functions with arbitrary degree of differentiability at the origin and at the support length....
[...]
...…(E-mail: nychka@ucar.edu). c© 2006 American Statistical Association, Institute of Mathematical Statistics, and Interface Foundation of North America Journal of Computational and Graphical Statistics, Volume 15, Number 3, Pages 502–523 DOI: 10.1198/106186006X132178 502 and interactive computing…...
[...]
...Douglas Nychka is Senior Scientist, Geophysical Statistics Project, National Center for Atmospheric Research, Boulder, CO 80307–3000 (E-mail: nychka@ucar.edu). c© 2006 American Statistical Association, Institute of Mathematical Statistics, and Interface Foundation of North America Journal of Computational and Graphical Statistics, Volume 15, Number 3, Pages 502–523 DOI: 10.1198/106186006X132178 502 and interactive computing environment....
[...]
References
More filters
Book•
[...]
TL;DR: In this article, the authors present results of both classic and recent matrix analyses using canonical forms as a unifying theme, and demonstrate their importance in a variety of applications, such as linear algebra and matrix theory.
Abstract: Linear algebra and matrix theory are fundamental tools in mathematical and physical science, as well as fertile fields for research. This new edition of the acclaimed text presents results of both classic and recent matrix analyses using canonical forms as a unifying theme, and demonstrates their importance in a variety of applications. The authors have thoroughly revised, updated, and expanded on the first edition. The book opens with an extended summary of useful concepts and facts and includes numerous new topics and features, such as: - New sections on the singular value and CS decompositions - New applications of the Jordan canonical form - A new section on the Weyr canonical form - Expanded treatments of inverse problems and of block matrices - A central role for the Von Neumann trace theorem - A new appendix with a modern list of canonical forms for a pair of Hermitian matrices and for a symmetric-skew symmetric pair - Expanded index with more than 3,500 entries for easy reference - More than 1,100 problems and exercises, many with hints, to reinforce understanding and develop auxiliary themes such as finite-dimensional quantum systems, the compound and adjugate matrices, and the Loewner ellipsoid - A new appendix provides a collection of problem-solving hints.
23,986 citations
Book•
01 Jan 1965
TL;DR: This chapter discusses the concept of a Random Variable, the meaning of Probability, and the axioms of probability in terms of Markov Chains and Queueing Theory.
Abstract: Part 1 Probability and Random Variables 1 The Meaning of Probability 2 The Axioms of Probability 3 Repeated Trials 4 The Concept of a Random Variable 5 Functions of One Random Variable 6 Two Random Variables 7 Sequences of Random Variables 8 Statistics Part 2 Stochastic Processes 9 General Concepts 10 Random Walk and Other Applications 11 Spectral Representation 12 Spectral Estimation 13 Mean Square Estimation 14 Entropy 15 Markov Chains 16 Markov Processes and Queueing Theory
13,886 citations
6,899 citations
Book•
01 Mar 1990
TL;DR: In this paper, a theory and practice for the estimation of functions from noisy data on functionals is developed, where convergence properties, data based smoothing parameter selection, confidence intervals, and numerical methods are established which are appropriate to a number of problems within this framework.
Abstract: This book serves well as an introduction into the more theoretical aspects of the use of spline models. It develops a theory and practice for the estimation of functions from noisy data on functionals. The simplest example is the estimation of a smooth curve, given noisy observations on a finite number of its values. Convergence properties, data based smoothing parameter selection, confidence intervals, and numerical methods are established which are appropriate to a number of problems within this framework. Methods for including side conditions and other prior information in solving ill posed inverse problems are provided. Data which involves samples of random variables with Gaussian, Poisson, binomial, and other distributions are treated in a unified optimization context. Experimental design questions, i.e., which functionals should be observed, are studied in a general context. Extensions to distributed parameter system identification problems are made by considering implicitly defined functionals.
6,120 citations