University of Nebraska - Lincoln University of Nebraska - Lincoln
DigitalCommons@University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln
Publications, Agencies and Staff of the U.S.
Department of Commerce
U.S. Department of Commerce
2011
ICOADS Release 2.5: extensions and enhancements to the ICOADS Release 2.5: extensions and enhancements to the
surface marine meteorological archive surface marine meteorological archive
Scott D. Woodruff
NOAA/ESRL (PSD3)
, Scott.D.Woodruff@noaa.gov
Steven J. Worley
National Center for Atmospheric Research
Sandra J. Lubker
NOAA Earth System Research Laboratory
Zaihua Ji
National Center for Atmospheric Research
J. Eric Freeman
Sourcecorp
See next page for additional authors
Follow this and additional works at: https://digitalcommons.unl.edu/usdeptcommercepub
Part of the Environmental Sciences Commons
Woodruff, Scott D.; Worley, Steven J.; Lubker, Sandra J.; Ji, Zaihua; Freeman, J. Eric; Berry, David I.; Brohan,
Philip; Kent, Elizabeth C.; Reynolds, Richard W.; Smith, Shawn R.; and Wilkinson, Clive, "ICOADS Release
2.5: extensions and enhancements to the surface marine meteorological archive" (2011).
Publications,
Agencies and Staff of the U.S. Department of Commerce
. 332.
https://digitalcommons.unl.edu/usdeptcommercepub/332
This Article is brought to you for free and open access by the U.S. Department of Commerce at
DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in Publications, Agencies and
Staff of the U.S. Department of Commerce by an authorized administrator of DigitalCommons@University of
Nebraska - Lincoln.
Authors Authors
Scott D. Woodruff, Steven J. Worley, Sandra J. Lubker, Zaihua Ji, J. Eric Freeman, David I. Berry, Philip
Brohan, Elizabeth C. Kent, Richard W. Reynolds, Shawn R. Smith, and Clive Wilkinson
This article is available at DigitalCommons@University of Nebraska - Lincoln: https://digitalcommons.unl.edu/
usdeptcommercepub/332
INTERNATIONAL JOURNAL OF CLIMATOLOGY
Int. J. Climatol. 31: 951–967 (2011)
Published online 2 March 2010 in Wiley Online Library
(wileyonlinelibrary.com) DOI: 10.1002/joc.2103
ICOADS Release 2.5: extensions and enhancements to the
surface marine meteorological archive
Scott D. Woodruff,
a
*
†
Steven J. Worley,
b
Sandra J. Lubker,
a†
Zaihua Ji,
b
J. Eric Freeman,
c,d†
David I. Berry,
e
Philip Brohan,
f‡
Elizabeth C. Kent,
e
Richard W. Reynolds,
g
Shawn R. Smith
h
and Clive Wilkinson
d,i
a
NOAA Earth System Research Laboratory, Boulder, CO 80305, USA
b
National Center for Atmospheric Research, Boulder, CO 80307, USA
c
Sourcecorp, Dallas, TX 75204, USA
d
NOAA Climate Database Modernization Program (CDMP), Asheville, NC 28801, USA
e
National Oceanography Centre, Southampton, SO14 3ZH, UK
f
Met Office Hadley Centre, Exeter, Devon EX1 3PB, UK
g
NOAA National Climatic Data Center, Asheville, NC 28801, USA
h
Center for Ocean-Atmospheric Prediction Studies, Florida State University, Tallahassee, FL 32306, USA
i
Climatic Research Unit, University of East Anglia, Norwich NR4 7TJ, UK
ABSTRACT: Release 2.5 of the International Comprehensive Ocean-Atmosphere Data Set (ICOADS) is a major update
(covering 1662–2007) of the world’s most extensive surface marine meteorological data collection. Building on extensive
national and international partnerships, many new and improved contributing datasets have been processed into a uniform
format and combined with the previous Release 2.4. The new data range from early non-instrumental ship observations to
measurements initiated in the twentieth century from buoys and other automated platform types. Improvements to existing
data include replacing preliminary Global Telecommunication System (GTS) receipts with more reliable, delayed mode
reports for post-1997 data, and in the processing and quality control (QC) of humidity observations. Over the entire period
of record, spatial and temporal coverage has been enriched and data and metadata quality has been improved. Along with
the observations, now updated monthly in near real time, Release 2.5 includes quality-controlled monthly summary products
for 2
°
latitude × 2
°
longitude (since 1800) and 1
°
× 1
°
boxes (since 1960), together with multiple options for access to
the data and products. The measured and estimated data in Release 2.5 are subject to many technical changes, multiple
archive sources, and historical events throughout the more than three-century record. Some of these data characteristics
are highlighted, including known unresolved errors and inhomogeneities, which may impact climate and other research
applications. Anticipated future directions for ICOADS aim to continue adding scientific value to the observations, products,
and metadata, as well as strengthen the cooperative enterprise through expanded linkages to international initiatives and
organisations. Copyright
2010 Royal Meteorological Society and Crown Copyright.
KEY WORDS marine meteorological data; ship data; buoy data; data rescue; sea surface temperature; sea level pressure;
humidity; metadata
Received 2 July 2009; Accepted 10 January 2010
1. Introduction
The International Comprehensive Ocean-Atmosphere
Data Set (ICOADS) contains marine surface data span-
ning the past three centuries. The observational archive
and monthly summary products were first made available
25 years ago (Slutz et al., 1985; Woodruff et al., 1987).
Each subsequent release (see Appendix) has brought
* Correspondence to: Scott D. Woodruff, NOAA/ESRL (PSD3), 325
Broadway, Boulder, CO 80305 USA.
E-mail: Scott.D.Woodruff@noaa.gov
†
The contributions of Scott D. Woodruff, Sandra J. Lubker and J. Eric
Freeman to this article were prepared as part of their official duties as
United States Federal Government employees.
‡
The contribution of P Brohan was written in the course of his
employment at the Met Office, UK and is published with the permission
of the Controller of HMSO and the Queen’s Printer for Scotland.
notable improvements that have benefited research,
education, and commercial operations. The continuing
objectives that remain are to collect as much original
estimated (non-instrumental) and measured in situ data
as possible; to treat each observation systematically –
preserving data source identification and measurement
metadata with each record, and converting units and cod-
ing schemes to a uniform set – to perform basic qual-
ity checks; and to freely distribute the data and prod-
ucts worldwide. As it contains observations from many
different observing systems encompassing the evolu-
tion of measurement technology over hundreds of years,
ICOADS is probably the most complete and heteroge-
neous collection of surface marine data in existence.
The expanding national and international community
providing data, and otherwise contributing to ICOADS
Copyright 2010 Royal Meteorological Society and Crown Copyright.
This article is a U.S. government work, and is not subject to copyright in the United States.
952 S. D. WOODRUFF et al.
development, is reviewed in Section 2 of this paper.
Section 3 describes ICOADS Release 2.5 (1662–2007)
and the continuing monthly observational updates for
2008 onwards. Section 4 discusses selected data char-
acteristics of Release 2.5 and highlights some outstand-
ing data issues. Sections 5 and 6 briefly review access
to ICOADS data and products and community-derived
products respectively. Section 7 covers future goals and
challenges for ICOADS and conclusions are in Section 8.
2. The national and international contributing
community
ICOADS has grown from a US-centric effort between
the National Oceanic and Atmospheric Administration
(NOAA) – its Earth System Research Laboratory (ESRL)
and National Climatic Data Center (NCDC) – and the
National Center for Atmospheric Research (NCAR) to
an expanded United States and international partner-
ship. This growth has added value to ICOADS in many
ways. New data sources have been uncovered and con-
tributed; expert knowledge has enabled data improve-
ments spanning the early sailing fleets, ship and measure-
ments metadata, and quantification of data uncertainty;
to name just a few examples. Data provision, colla-
tion, and distribution remain the responsibility of the
founding partners, but other countries and international
organisations – including the Joint World Meteorological
Organisation (WMO)–Intergovernmental Oceanographic
Commission (IOC) Technical Commission for Oceanog-
raphy and Marine Meteorology (JCOMM) – now make
noteworthy contributions.
From the operational and research areas, ICOADS
incorporates observations from a variety of existing
activities. Real-time Global Telecommunication System
(GTS) data are provided by the NOAA National Centers
for Environmental Prediction (NCEP), marine surface
meteorological observations and near-surface profile tem-
peratures are contributed by the NOAA National Oceano-
graphic Data Center (NODC) through the World Ocean
Database (WOD), and environmental US moored buoy
and Coastal Marine-Automated Network (C-MAN) data
are provided by the NOAA National Data Buoy Cen-
ter (NDBC).
Data from tropical moored buoy arrays – specifically,
the Tropical Atmosphere Ocean/TRIangle Trans-Ocean
buoy Network (TAO/TRITON) array in the Pacific
(McPhaden et al., 1998) and the Prediction and Research
Moored Array in the Atlantic (PIRATA) (Bourl
`
es et al.,
2008) – are provided by NDBC, the NOAA Pacific
Marine Environmental Laboratory (PMEL) and the Japan
Agency for Marine-Earth Science and Technology (JAM-
STEC). The Center for Ocean-Atmospheric Predic-
tion Studies (COAPS) at Florida State University con-
tributes high-resolution measurements from some ocean
research vessels (R/Vs) (Gould and Smith, 2006). Inte-
grated Science Data Management (ISDM), of Fisheries
and Oceans – Canada, provides quality controlled (QCd)
drifting (plus some moored) buoy data.
The JCOMM global collecting centres (GCCs) dis-
tribute worldwide delayed mode (DM) (i.e. paper or
electronic logbook) Voluntary Observing Ship (VOS)
data on a quarterly basis, currently from about 26 con-
tributing members (Woodruff et al., 2009). The National
Oceanography Centre, Southampton – UK, contributes
quality-checked WMO (1955) ship metadata (Kent et al.,
2007). Beyond the most active recent and ongoing major
programs noted above, there are many others, including
substantial contributions from Europe and Asia (see e.g.
Worley et al., 2005 and Woodruff et al., 2005).
In the historical arena, the RECovery of Logbooks
And International Marine data (RECLAIM; Wilkinson
et al., 2010), Climatological database for the World’s
Oceans (CLIWOC) (Garc
´
ıa-Herrera et al., 2005), and
Atmospheric Circulation Reconstruction over the Earth
initiative (ACRE; http://www.met-acre.org/) are all multi-
national efforts that have augmented ICOADS with sev-
enteenth to twentieth century data and metadata. For
example, in recent joint projects, the UK Met Office and
NOAA Climate Data Modernization Program (CDMP)
(Dupigny-Giroux et al., 2007) have provided digitised
and quality-checked data from logbooks (e.g. Brohan
et al., 2008) and historical publications held in UK
archives.
Continuity and coordination for ICOADS have been
promoted by two series of meetings that began over
a decade ago: The JCOMM Workshops on Advances
in Marine Climatology (CLIMAR) were held in Van-
couver, Canada, 1999, Brussels, Belgium, 2003, Parker
et al., 2004 and Gydnia, Poland, 2008 (Charpentier et al.,
2008). Alternating approximately biennially with CLI-
MAR, the Workshops on Advances in the Use of Histor-
ical Marine Climate Data (MARCDAT) have been held in
Boulder, USA, 2002, and Exeter, UK, 2005 (Kent et al.,
2007). Sponsors of past meetings have included WMO,
the Global Climate Observing System (GCOS), and the
Climate Variability and Predictability Project (CLIVAR).
These meetings have done much to guide the develop-
ment of ICOADS.
3. Release 2.5
Release 2.5 (R2.5) data processing was completed in
July 2009. This marks a major update where 238 million
(M) reports from the preceding Release 2.4 (R2.4) were
merged together with new data, subjected to duplicate
analysis and elimination, quality checked against clima-
tologies, and preconditioned to fix specific problems and
remove suspect data. The resultant output was then com-
pared with R2.4. R2.5 has 261M records in the Interna-
tional Maritime Meteorological Archive (IMMA) format
covering 1662–2007. This archive is a blend of DM and
GTS data, where DM records are preferentially selected
in cases where duplicates are detected.
The IMMA format (Woodruff, 2007) has been rec-
ommended as a standard to JCOMM and is ideal for
marine surface data spanning many centuries. It is an
Copyright 2010 Royal Meteorological Society and Crown Copyright. Int. J. Climatol. 31: 951–967 (2011)
THE INTERNATIONAL COMPREHENSIVE OCEAN-ATMOSPHERE DATA SET 953
ASCII-based format with a fixed ‘core’ set of the most
commonly reported meteorological variables sufficient
for most users, also including the time, location, and
individual platform identification (e.g. ship callsign or
WMO buoy number, if available). In addition to the
IMMA core, an arbitrary number of data ‘attachments’
can be made. Ship metadata, original data records, QC
flags, special ICOADS source-tracking information, plat-
form type, and a variety of other ancillary metadata are
preserved in these attachments.
A new procedure to extend ICOADS to near real time
is also coincident with R2.5 providing IMMA records
based on the best available GTS receipts (currently based
exclusively on NCEP data). This forward extension to
ICOADS is updated monthly, and lags real-time by 2 to
6 weeks. These data are subjected to basic quality checks
and the IMMA core fields are populated to the maximum
extent possible. This product is replacing an existing
near-real-time ASCII-formatted product containing only
a limited selection of data fields.
R2.5 greatly improved ICOADS by extending the
period of record backward to 1662 (i.e. by 122 years,
albeit with sparse coverage), and forward through the
end of 2007, and generally increasing the number of
available records over the whole period. Twelve major
new or replacement historical data sources were incor-
porated (Figure 1) and numerous other data sources
reprocessed (the Appendix provides R2.5 data com-
position details). Some new sources were specifically
digitised for ICOADS, e.g. US Marine Meteorologi-
cal Journals (MMJ), 1878–1894, 1.8M records. A few
data sources replaced earlier versions, e.g. the World
Ocean Database 2005 (WOD05), 1772–2005, 7.9M.
Others such as the UK Marine Data Bank (MDB)
(previously blended into periods outside 1950–1979;
Woodruff et al., 2003) partially duplicated reports already
in ICOADS, but also provided some unique or improved
contributions.
Figure 1. Major historical data sources added to ICOADS Release 2.5.
Horizontal green lines illustrate the time range of the original data
sources. The annual numbers of reports are plotted as curves, blue for
the previous R2.4, and red for R2.5. For clarity, the vertical scale is
truncated at 9M; years 2005–2007 have 13M, 15M, and 16M total
reports (not visible) in R2.5, respectively. Data coverage prior to 1800
is very sparse.
Based purely on the number of records, regardless of
which parameters might be available, R2.5 has increased
the global spatial coverage. Figure 2 illustrates important
increases in global sampling in R2.5 distributed over the
full period since the 1860s. The 1880s shows outstanding
improvement (up to 9%), primarily due to addition of
the MMJ collection (Figure 1), which provides extensive
observations throughout the diurnal cycle (plus accom-
panying detailed instrumental metadata and instructions).
Deutscher Wetterdienst (DWD) historical archive subsets
helped increase coverage from the late 1800s through
1914, and the 2003 Edition of the Japanese Kobe col-
lection accounts for major increases around the 1920s.
The UK Royal Navy Ship’s Logs (Brohan et al., 2008)
helped fill in the data-sparse period (1938–1947) around
World War II, and the UK MDB helped enhance coverage
particularly during the 1960s and 1970s.
While these increases had limited impact on total
ocean area sampled (still peaking at about 60% around
1980), they should add confidence to long-term climate
estimates. Focusing on the 1880s, for example, three
important variables – sea surface temperature (SST), sea
level pressure (SLP), and relative humidity (RH) – show
greatly increased global coverage relative to Release 1
and to the previous update of that period (Figure 3). The
nearly void RH coverage prior to R2.5 was addressed by
a QC modification discussed in Section 6.
4. Data characteristics and unresolved issues
An important objective with each new Release is to
replace GTS data with higher quality DM data. Normally
this has limited effect on temporal and spatial coverage,
but occasionally DM data can more appreciably augment
coverage. This is illustrated for the 1998–2006 period,
where global coverage increased 2% or more with respect
to R2.4 (Figure 2). This effect arises largely because no
DM ship data had been added to ICOADS for that period
(owing to project resource limitations) since 2002.
This impact can be segregated into three broad plat-
form categories: observations from ships, moored buoys,
and drifting buoys. In R2.4, all ship data from 1998 and
onward were derived from GTS receipts, whereas in R2.5
about 50% of data in this period are from DM collections
(Figure 4). Here DM data from the GCCs (and other
sources) have replaced the GTS data for 1998 onward
and additional new data have been added from historical
sources throughout the period of record (e.g. numbers
of reports increased 30% or more around World War
II through blending the UK Royal Navy Ship’s Logs;
Figure 4).
The differences in moored buoys between R2.4 and
R2.5 are relatively minor when contrasting annual record
counts and data sources (Figure 5). It is noteworthy
though that minor decreases in R2.5 have been achieved
with better duplicate elimination, e.g. 2005–2006. The
mixture of sources has not changed much, but the tropi-
cal moored buoys were replaced with a new DM version
Copyright 2010 Royal Meteorological Society and Crown Copyright. Int. J. Climatol. 31: 951–967 (2011)
