35
Australian Meteorological and Oceanographic Journal 62 (2012) 35–37
Quarterly numerical weather prediction
model performance summary—October to
December 2011
(Manuscript received April 2012)
Xiaoxi Wu
National Meteorological and Oceanographic Centre, Bureau of Meteorology, Australia
Introduction
This summary, covering the three-month period from
October to December 2011, continues the series reporting
on the performances of Numerical Weather Prediction
(NWP) models used operationally in the Australian Bureau
of Meteorology.
NWP models—October to December 2011
Local Models
The Bureau’s tropical cyclone model ACCESS-TC was
made operational on 11 November 2011. ACCESS-TC is
the successor to the Bureau’s old tropical cyclone model
TCLAPS which was turned off in August 2010. For the
details regarding ACCESS-TC, please refer to: http://www.
bom.gov.au/australia/charts/bulletins/apob90.pdf
No changes have been reported for the Bureau’s
operational global model ACCESS-G and the limited area
models ACCESS-R, ACCESS-T and ACCESS-A during this
verification period.
Details on the configurations of the Bureau’s models are
described in an earlier summary (Wu and Bridge 2010). For
more details about the ACCESS systems, please refer to:
http://www.bom.gov.au/australia/charts/bulletins/apob83.
pdf and http://www.bom.gov.au/nmoc/access/NWPData.
shtml.
Overseas Models
The following four operational global models which are run
by overseas forecast centres are verified in this article. The
European Centre Spectral Prognosis (ECSP) refers to the
European Centre for Medium-Range Weather Forecasts
(ECMWF) system, UKGC to the Unified Model from the UK
Met Office, United States Aviation Model (USAVN) to the
Global Forecast System (GFS) from National Centers for
Environmental Prediction (NCEP) and Japan Meteorological
Agency Global Spectral Model (JMAGSM) to the global
assimilation and forecast model from JMA.
On 25 October 2011 JMAGSM increased the inner loop
horizontal resolution from T159 (~80km) to TL319 (~60km).
On 15 Nov 2011 ECMWF introduced a new version of
the NWP system called Cycle 37r3. The new cycle includes
a collection of improvements to the forecast models and
the data assimilation system. The cycle also contains hourly
post-processing of model data to 90 hours in support of the
BC (Boundary Conditions) Optional Programme. The main
meteorological changes implemented in this cycle include
modification of the entrainment/detrainment of convection;
modification of the supersaturation and deposition rate for
clouds; modification of the surface roughness; bias correction
of aircraft temperature observations; cycling of stratospheric
model error (for the weak-constraint 4D-Var); assimilation
of accumulated rainfall from NEXRAD radar data from
the United States; assimilation of ozone observations from
infrared radiances; use of the latest version of the NWP-SAF
(Satellite Application Facilities) radiative transfer model;
and retuning of cloud detection for the advanced infrared
sounder data.
For further information on the improvements made to
overseas NWP assimilation and forecast models refer to web
references given below. Details on the configurations of the
assimilation and forecast models are described in an earlier
summary (Lee 2005).
Verification method
A description of the S1 skill-score, as applied in NMOC, can
be found in the paper by Skinner (1995). All results have
been calculated within NMOC Melbourne, where each of the
models was verified against its own analysis. From the large
number of objective verification results routinely produced,
the statistics presented here cover only the mean sea level
pressure (MSLP) and 500 hPa geopotential height fields
over the irregular Australian verification area (Miao 2003).
It is noted that this particular verification grid has southerly
points that are outside the ACCESS-T’s southern domain
boundary and, hence, the ACCESS-T scores are not strictly
Corresponding author address: Lixin Qi, National Meteorological and
Oceanographic Centre, Bureau of Meteorology, GPO Box 1289, Mel-
bourne Vic. 3001, Australia
email: L.Qi@bom.gov.au
36 Australian Meteorological and Oceanographic Journal 62:1 March 2012
comparable with those from ACCESS-G/R. Also the results
for the 0000 and 1200 UTC base-times have been combined.
For the locally run, limited-area models, the verified forecast
periods go out to a maximum of 72 hours and for the global
models to a maximum of 192 hours.
Review of performance—October to
December 2011
Figures 1 to 3 are the plots covering the verifying period
from October to December 2011.
Local models (ACCESS-G, ACCESS-R, ACCESS-T)
The intercomparison of the S1 skill scores of the MSLP
forecasts for the three local models covering the verifying
period October to December 2011 is shown in Fig. 1(a). The
S1 skill-scores are averaged over the three-month period
for various forecast periods ranging from 0 to 72 hours. S1
skill-score comparison of the 500 hPa geopotential height
forecasts is shown in Fig. 1(b). In general, the coarser-
resolution global model outperforms the finer-resolution
limited area models. This result is partly due to the later
data cut-off of the assimilation for the global models. It is
also due to the disadvantage suffered by the limited area
models which obtain their initial first guess and boundary
conditions from the earlier run of the global model forecasts.
Forecasts from earlier runs tend to be poorer than forecasts
produced from later runs. One other contributing factor
for the better-than-expected scores for the global models
is the verification method used here, which disadvantages
finer resolution models through ‘double penalty’ scoring.
For example, a location error of a deep low pressure system
from a more realistic high resolution forecast is counted
once for misplacing the low where the verifying analysis
does not have it and twice for not placing it where the
verifying analysis does. Care needs to be taken to filter out
Fig. 1(a) MSLP S1 skill-score comparison, for different fore-
cast periods, between ACCESS-G, ACCESS-R and
ACCESS-T (October to December 2011).
Fig. 1(b): 500 hPa geopotential height S1 skill-score compari-
son, for different forecast periods, between ACCESS-
G, ACCESS-R and ACCESS-T (October to December
2011).
Fig. 2(a): MSLP S1 skill-score comparison, for different forecast
periods, between ACCESS-G, ECSP, UKGC, USAVN,
and JMAGSM (October to December 2011).
Fig. 2(b): 500 hPa geopotential height S1 skill-score compari-
son, for different forecast periods, between ACCESS-
G, ECSP, UKGC, USAVN and JMAGSM (October to
December 2011).
Wu: NWP summary October to December 2011 37
scales below which a verification method was not intended
to measure if models that are run at different resolutions are
to be objectively compared.
Global models (ACCESS-G, ECSP, UKGC, USAVN,
JMAGSM)
The Bureau’s new operational global spectral model
ACCESS-G and the four global models from overseas NWP
centres are operationally used by forecasters. The outputs
from the models are also postprocessed to produce various
objective guidance products used in and outside of the
Bureau. Hence their forecast performance is of great interest
to the forecasters and other users. The S1 skill scores for
MSLP and 500 hPa geopotential height forecasts for the
period October to December 2011 are presented in Figs 2(a)
and 2(b). Anomaly correlations for the MSLP forecasts are
shown in Fig. 3.
From this quarterly summary, the low resolution USAVN
model at the 2.5° x 5.0° grid points has been replaced by
the high-resolution model at 0.5°x0.5° grid points. With the
same verification method as other models presented in this
summary, the high-resolution US forecasts and analyses are
regridded on the common 2.5° x 2.5° verification grid before
the verification. It is then verified at the forecast hours up
to 192 hours which is the same valid forecast period as
ACCESS-G and JMAGSM.
Assuming the commonly used cut-off of 60 per cent as the
criterion for useful forecasts (Murphy 1989), for the October
to December 2011 quarter the anomaly correlation scores
for the ACCESS-G, ECMWF, JMAGSM and USAVN show
useful skill to beyond seven days. ACCESS-G has similar
skill as JMAGSM and USAVN at the short term up to two
days but becomes less skillful than those two models for the
longer term up to six days, before becoming similarly skillful
to USAVN again at day seven and day eight.
References
Lee, J. 2005. Quarterly numerical weather prediction model performance
summary – July to September 2005. Aust. Meteorol. Mag., 54, 253-61.
Miao, Y. 2003. Numerical prediction model performance summary July to
September 2002. Aust. Meteorol. Mag., 52, 73-5.
Murphy, A. and Epstein E.S. 1989. Skill Scores and Correlation Coeffi-
cients in Model Verification. Mon. Weather Rev., 117, 572-81.
Skinner, W. 1995. Numerical prediction model performance summary
April to June 1995. Aust. Meteorol. Mag., 44, 309-12.
Wu, X. and Bridge, C. 2010. Quarterly numerical weather prediction mod-
el performance summaries April to June 2010 and July to September
2010. Aust. Met. Oceanogr. J., 60, 301-5.
Web reference:
For ECMWF:
http://www.ecmwf.int/publications/newsletters
http://www.ecmwf.int/products/data/technical/model_id/index.html
For UKMO:
http://www.metoffice.gov.uk/research/nwp/publications/
For NCEP:
http://www.emc.ncep.noaa.gov/gmb/STATS/html/model_changes.html
For JMA:
http://ddb.kishou.go.jp
For ACCESS:
http://www.bom.gov.au/australia/charts/bulletins/apob83.pdf
http://www.bom.gov.au/nmoc/access/NWPData.shtml
For Verify:
http://synopticview.co.uk/metpy_verify.html
Fig. 3 Anomaly correlation of MSLP comparison, for dif-
ferent forecast periods, between ACCESS-G, ECSP,
UKGC, USAVN and JMAGSM (October to December
2011).
38 Australian Meteorological and Oceanographic Journal 62:1 March 2012