Objective quality control of observations using Bayesian methods. Theory, and a practical implementation

TLDR: In this paper, the authors provide a theoretical framework for the quality control of data from a large variety of types of observations, with different accuracies and reliabilities, and apply Bayes' theorem to derive the well-known formula for the combination of data with errors.

Abstract: This work attempts to provide a theoretical framework for the quality control of data from a large variety of types of observations, with different accuracies and reliabilities. Bayes' theorem is introduced, and is used in a simple example with Gaussian error distributions to derive the well-known formula for the combination of data with errors. A simple model is proposed whereby the error in each datum is either from a known Gaussian distribution, or a gross error, in which case the observation gives no useful information. Bayes' theorem is applied to this, and it is shown that usual operational practice, which is to reject outlying data and to treat the rest as if their errors are Gaussian, is a reasonable approximation to the correct Bayesian analysis. Appropriate rejection criteria are derived in terms of the observational error and the prior probability of a gross error. These ideas have been implemented in a computer program to check pressure, wind, temperature and position data from ships, weather ships, buoys and coastal synoptic reports. Historical information on the accuracies and reliabilities of various classifications of observation is used to provide prior estimates of observational errors and the prior probabilities of gross error. The latter are then updated in the light of information from a current forecast, and from nearby observations (allowing for the inaccuracies and possible gross errors in these) to give new estimates. The final probabilities can be used to reject or accept the data in an objective analysis. Results from trials of this system are given. It is shown to be possible using an archive generated by the system to update the prior error statistics necessary to make the method truly objective. Some practical case studies are shown, and compared with careful human quality control.