The purpose of this paper is to provide a comprehensive presentation and interpretation of the Ensemble Kalman Filter (EnKF) and its numerical implementation. The EnKF has a large user group, and numerous publications have discussed applications and theoretical aspects of it. This paper reviews the important results from these studies and also presents new ideas and alternative interpretations which further explain the success of the EnKF. In addition to providing the theoretical framework needed for using the EnKF, there is also a focus on the algorithmic formulation and optimal numerical implementation. A program listing is given for some of the key subroutines. The paper also touches upon specific issues such as the use of nonlinear measurements, in situ profiles of temperature and salinity, and data which are available with high frequency in time. An ensemble based optimal interpolation (EnOI) scheme is presented as a cost-effective approach which may serve as an alternative to the EnKF in some applications. A fairly extensive discussion is devoted to the use of time correlated model errors and the estimation of model bias.

/pdf/the-ensemble-kalman-filter-theoretical-formulation-and-2xx6out3z9.pdf

The Ensemble Kalman Filter: theoretical formulation and practical implementation

The Twentieth Century Reanalysis (20CR) project is an international effort to produce a comprehensive global atmospheric circulation dataset spanning the twentieth century, assimilating only surface pressure reports and using observed monthly sea-surface temperature and sea-ice distributions as boundary conditions. It is chiefly motivated by a need to provide an observational dataset with quantified uncertainties for validations of climate model simulations of the twentieth century on all time-scales, with emphasis on the statistics of daily weather. It uses an Ensemble Kalman Filter data assimilation method with background ‘first guess’ fields supplied by an ensemble of forecasts from a global numerical weather prediction model. This directly yields a global analysis every 6 hours as the most likely state of the atmosphere, and also an uncertainty estimate of that analysis.

The 20CR dataset provides the first estimates of global tropospheric variability, and of the dataset's time-varying quality, from 1871 to the present at 6-hourly temporal and 2° spatial resolutions. Intercomparisons with independent radiosonde data indicate that the reanalyses are generally of high quality. The quality in the extratropical Northern Hemisphere throughout the century is similar to that of current three-day operational NWP forecasts. Intercomparisons over the second half-century of these surface-based reanalyses with other reanalyses that also make use of upper-air and satellite data are equally encouraging.

It is anticipated that the 20CR dataset will be a valuable resource to the climate research community for both model validations and diagnostic studies. Some surprising results are already evident. For instance, the long-term trends of indices representing the North Atlantic Oscillation, the tropical Pacific Walker Circulation, and the Pacific–North American pattern are weak or non-existent over the full period of record. The long-term trends of zonally averaged precipitation minus evaporation also differ in character from those in climate model simulations of the twentieth century. Copyright © 2011 Royal Meteorological Society and Crown Copyright.

/pdf/the-twentieth-century-reanalysis-project-5fx9pcs9x0.pdf

The Twentieth Century Reanalysis Project

The Ensemble Kalman Filter: Theoretical formulation and practical implementation

This comprehensive text and reference work on numerical weather prediction, first published in 2002, covers not only methods for numerical modeling, but also the important related areas of data assimilation and predictability. It incorporates all aspects of environmental computer modeling including an historical overview of the subject, equations of motion and their approximations, a modern and clear description of numerical methods, and the determination of initial conditions using weather observations (an important science known as data assimilation). Finally, this book provides a clear discussion of the problems of predictability and chaos in dynamical systems and how they can be applied to atmospheric and oceanic systems. Professors and students in meteorology, atmospheric science, oceanography, hydrology and environmental science will find much to interest them in this book, which can also form the basis of one or more graduate-level courses.

/pdf/atmospheric-modeling-data-assimilation-and-predictability-1qxqadvqdq.pdf

Atmospheric Modeling, Data Assimilation and Predictability

The possibility of performing data assimilation using the flow-dependent statistics calculated from an ensemble of short-range forecasts (a technique referred to as ensemble Kalman filtering) is examined in an idealized environment. Using a three-level, quasigeostrophic, T21 model and simulated observations, experiments are performed in a perfect-model context. By using forward interpolation operators from the model state to the observations, the ensemble Kalman filter is able to utilize nonconventional observations. In order to maintain a representative spread between the ensemble members and avoid a problem of inbreeding, a pair of ensemble Kalman filters is configured so that the assimilation of data using one ensemble of shortrange forecasts as background fields employs the weights calculated from the other ensemble of short-range forecasts. This configuration is found to work well: the spread between the ensemble members resembles the difference between the ensemble mean and the true state, except in the case of the smallest ensembles. A series of 30-day data assimilation cycles is performed using ensembles of different sizes. The results indicate that (i) as the size of the ensembles increases, correlations are estimated more accurately and the root-meansquare analysis error decreases, as expected, and (ii) ensembles having on the order of 100 members are sufficient to accurately describe local anisotropic, baroclinic correlation structures. Due to the difficulty of accurately estimating the small correlations associated with remote observations, a cutoff radius beyond which observations are not used, is implemented. It is found that (a) for a given ensemble size there is an optimal value of this cutoff radius, and (b) the optimal cutoff radius increases as the ensemble size increases.

/pdf/data-assimilation-using-an-ensemble-kalman-filter-technique-24neq9pvfa.pdf

Data Assimilation Using an Ensemble Kalman Filter Technique

An ensemble Kalman filter may be considered for the 4D assimilation of atmospheric data. In this paper, an efficient implementation of the analysis step of the filter is proposed. It employs a Schur (elementwise) product of the covariances of the background error calculated from the ensemble and a correlation function having local support to filter the small (and noisy) background-error covariances associated with remote observations. To solve the Kalman filter equations, the observations are organized into batches that are assimilated sequentially. For each batch, a Cholesky decomposition method is used to solve the system of linear equations. The ensemble of background fields is updated at each step of the sequential algorithm and, as more and more batches of observations are assimilated, evolves to eventually become the ensemble of analysis fields. A prototype sequential filter has been developed. Experiments are performed with a simulated observational network consisting of 542 radiosonde and 615 satellite-thickness profiles. Experimental results indicate that the quality of the analysis is almost independent of the number of batches (except when the ensemble is very small). This supports the use of a sequential algorithm. A parallel version of the algorithm is described and used to assimilate over 100 000 observations into a pair of 50-member ensembles. Its operation count is proportional to the number of observations, the number of analysis grid points, and the number of ensemble members. In view of the flexibility of the sequential filter and its encouraging performance on a NEC SX-4 computer, an application with a primitive equations model can now be envisioned.

/pdf/a-sequential-ensemble-kalman-filter-for-atmospheric-data-30hy8xln76.pdf

A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation

For many aspects of numerical weather prediction it is important to have good error statistics. Here one can think of applications as diverse as data assimilation, model improvement, and medium-range forecasting. In this paper, a method for producing these statistics from a representative ensemble of forecast states at the appropriate forecast time is proposed and examined. To generate the ensemble, an attempt is made to simulate the process of error growth in a forecast model. For different ensemble members the uncertain elements of the forecasts are perturbed in different ways. First the authors attempt to obtain representative initial perturbations. For each perturbation, an independent 6-h assimilation cycle is performed. For this the available observations are randomly perturbed. The perturbed observations are input to the statistical interpolation assimilation scheme, giving a perturbed analysis. This analysis is integrated for 6 h with a perturbed version of the T63 forecast model, using p...

A System Simulation Approach to Ensemble Prediction

An ensemble Kalman filter (EnKF) has been implemented for atmospheric data assimilation. It assimilates observations from a fairly complete observational network with a forecast model that includes a standard operational set of physical parameterizations. To obtain reasonable results with a limited number of ensemble members, severe horizontal and vertical covariance localizations have been used. It is observed that the error growth in the data assimilation cycle is mainly due to model error. An isotropic parameterization, similar to the forecast-error parameterization in variational algorithms, is used to represent model error. After some adjustment, it is possible to obtain innovation statistics that agree with the ensemble-based estimate of the innovation amplitudes for winds and temperature. Currently, no model error is added for the humidity variable, and, consequently, the ensemble spread for humidity is too small. After about 5 days of cycling, fairly stable global filter statistics are ob...

/pdf/atmospheric-data-assimilation-with-an-ensemble-kalman-filter-5bz7pslogy.pdf

Atmospheric Data Assimilation with an Ensemble Kalman Filter: Results with Real Observations

An ensemble Kalman filter (EnKF) has been implemented at the Canadian Meteorological Centre to provide an ensemble of initial conditions for the medium-range ensemble prediction system. This demonstrates that the EnKF can be used for operational atmospheric data assimilation.

We show how the EnKF relates to the Kalman filter. In particular, to make the ensemble approximation feasible, we have to use a fairly small ensemble with many less members than either the number of model coordinates, or the number of independent observations, or the (unknown) dimension of the dynamical system. To nevertheless obtain good results, we must (i) counter the tendency of the ensemble spread to underestimate the true error, and (ii) localize the ensemble covariances. The localization is severe and leads to imbalance in the initial conditions.

The operational EnKF is used to investigate to what extent our system respects the underlying hypotheses of both the Kalman filter and its ensemble approximation. In particular, we quantify the imbalance in the initial conditions and the magnitude of the model-error component. The occurrence of imbalance constrains the ways in which time interpolation can be performed and in which parametrized model error can be added. With this study we hope to obtain and provide guidance for further improvements to the EnKF. Copyright © 2005 Royal Meteorological Society

/pdf/ensemble-kalman-filtering-32snxfshew.pdf

Herschel L. Mitchell

Papers

Data Assimilation Using an Ensemble Kalman Filter Technique

A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation

A System Simulation Approach to Ensemble Prediction

Atmospheric Data Assimilation with an Ensemble Kalman Filter: Results with Real Observations

Ensemble Kalman filtering