We present an overview of key aspects of the Atmospheric Radiation Measurement (ARM) Program Climate Research Facility (ACRF) data quality assurance program. Processes described include instrument deployment and cali- bration; instrument and facility maintenance; data collection and processing infrastructure; data stream inspection and as- sessment; problem reporting, review and resolution; data archival, display and distribution; data stream reprocessing; en- gineering and operations management; and the roles of value-added data processing and targeted field campaigns in speci- fying data quality and characterizing field measurements. The paper also includes a discussion of recent directions in ACRF data quality assurance. A comprehensive, end-to-end data quality assurance program is essential for producing a high-quality data set from measurements made by automated weather and climate networks. The processes developed dur- ing the ARM Program offer a possible framework for use by other instrumentation- and geographically-diverse data col- lection networks and highlight the myriad aspects that go into producing research-quality data.

/pdf/an-overview-of-arm-program-climate-research-facility-data-4yy4l7h0md.pdf

An Overview of ARM Program Climate Research Facility Data Quality Assurance

This report documents key aspects of the Atmospheric Radiation Measurement (ARM) Climate Research Facility (ACRF) data quality assurance program as it existed in 2008. The performance of ACRF instruments, sites, and data systems is measured in terms of the availability, usability, and accessibility of the data to a user. First, the data must be available to users; that is, the data must be collected by instrument systems, processed, and delivered to a central repository in a timely manner. Second, the data must be usable; that is, the data must be inspected and deemed of sufficient quality for scientific research purposes, and data users must be able to readily tell where there are known problems in the data. Finally, the data must be accessible; that is, data users must be able to easily find, obtain, and work with the data from the central repository. The processes described in this report include instrument deployment and calibration; instrument and facility maintenance; data collection and processing infrastructure; data stream inspection and assessment; the roles of value-added data processing and field campaigns in specifying data quality and haracterizing the basic measurement; data archival, display, and distribution; data stream reprocessing; and engineering and operations management processes and procedures. Future directions in ACRF data quality assurance also are presented.

/pdf/quality-assurance-of-arm-program-climate-research-facility-4r62mya3iv.pdf

Quality Assurance of ARM Program Climate Research Facility Data

The IoT advancement generates a massive volume and a variety of data at unprecedented velocity. There is a need to interpret and to convert such data into valuable information, enabling the discovery of knowledge as well as the development of new technologies and data products. From the Data Science research field, Visual Analytics (VA) techniques seek to support in the analysis and understanding of large datasets using statistical techniques and data mining, aided by visualization techniques. This paper demonstrates how VA techniques can be applied in Data Science projects enabling users to reason over a dataset generated by IoT devices, what we call as Data Understandability. A practical case is presented in order to provide an overview of the data management procedures and how a set of Visual Analytics tools developed in the context of The U. S. Department of Energy Atmospheric Radiation Measurement (ARM) Program are helping users to visualize and analyze the collected data from a climate sensor network spread throughout the world.

Visual Analytics Improving Data Understandability in IoT Projects: An Overview of the U. S. DOE ARM Program Data Science Tools

Within the Atmospheric Radiation Measurement (ARM) instrument network, several different systems often measure the same quantity at the same site. For example, several ARM instruments measure time-series profiles of the atmosphere that were previously available only from balloon-borne radiosonde systems. These instruments include the Radar Wind Profilers (RWP) with Radio-Acoustic Sounding Systems (RASS), the Atmospheric Emitted Radiance Interferometer (AERI), the Microwave Radiometer Profiler (MWRP), and the Raman Lidar (RL). ARM researchers have described methods for direct cross-comparison of time-series profiles (Coulter and Lesht 1996; Turner et al. 1996) and we have extended this concept to the development of methods for automated quality control (QC) of ARM datastreams.

/pdf/instrument-cross-comparisons-and-automated-quality-control-4d3og1obsj.pdf

Instrument Cross-Comparisons and Automated Quality Control of Atmospheric Radiation Measurement Data

The Atmospheric Radiation Measurement (ARM) Data Quality Office (DQO) has made a number of improvements and additions over the past year to its main tool for inspecting and assessing ARM data quality—the Data Quality Health and Status (DQ HandS) system (http://dq.arm.gov/). Among the improvements and additions, some of which are shown below, are the inclusion of ARM Mobile Facility (AMF) data; a new plot browser to facilitate the viewing of DQ HandS diagnostic plots; an improved method for writing and databasing weekly data quality assessment reports; a new automated daily alert, an improved method for searching ARM report databases (see Doty and Wagener’s abstract in this proceeding); addition of more instrument and value-added products output, and creation of a development version of DQ HandS that allows the present system to become a true production tool.

/pdf/improvements-to-and-status-of-the-data-quality-health-and-2tgm41iqpl.pdf

K. Doty

Papers

Improvements To and Status of the Data Quality Health and Status System

Improvements To and Status of the Data Quality Health and Status System