Independent Validation of National Satellite-Based Land-Use Regression Models for Nitrogen Dioxide Using Passive Samplers

TLDR: In this paper, satellite observations of nitrogen dioxide (NO2) in land-use regression (LUR) models can improve their predictive ability, but requires rigorous evaluation and validation.

Abstract: Including satellite observations of nitrogen dioxide (NO2) in land-use regression (LUR) models can improve their predictive ability, but requires rigorous evaluation. We used 123 passive NO2 samplers sited to capture within-city and near-road variability in two Australian cities (Sydney and Perth) to assess the validity of annual mean NO2 estimates from existing national satellite-based LUR models (developed with 68 regulatory monitors). The samplers spanned roadside, urban near traffic (≤100 m to a major road), and urban background (>100 m to a major road) locations. We evaluated model performance using R2 (predicted NO2 regressed on independent measurements of NO2), mean-square-error R2 (MSE-R2), RMSE, and bias. Our models captured up to 69% of spatial variability in NO2 at urban near-traffic and urban background locations, and up to 58% of variability at all validation sites, including roadside locations. The absolute agreement of measurements and predictions (measured by MSE-R2) was similar to their c...

Figures

Table 3. Validation statistics for the surface and column models. RMSE = root-mean-square error; FB = fractional bias. Other abbreviations are 251 defined in the main text. 252

Table 2. Percentiles of annual NO2 concentrations (ppb) measured at validation sites. * Any negative concentrations following subtraction of 244 field blank values were randomly assigned a value between zero and the limit of detection (2.0 ppb) in the Perth study (see Dirgawati et al. 26). 245

Figure 2a (left) and 2b (right). Measured vs. predicted annual mean NO2 at 123 validation sites (roadside, urban near traffic, and urban 303 background combined) for the surface (2a) and column (2b) models. The dashed line is the line of best fit (see Table 3 for fit statistics). The solid 304 line is the line of agreement. Symbols denote different measurement campaigns: solid circles = Perth 1; hollow squares = Sydney 1; hollow 305 diamonds = Sydney 2. 306

Figure 1. The two Australian cities (Sydney and Perth) where validation measurements were performed. The left panel shows Perth and the right 165 shows Sydney. The map shows the 123 sites used in the main analysis, denoted as red triangles. Major roads are also shown. See Figure S5 in 166 the supporting information for maps of predicted NO2 in the study areas. The outlines were created using census data published by the 167 Australian Bureau of Statistics and roads were generated from data supplied by the Australian Public Sector Mapping Agencies.22,37 168

Figure 3a (left) and 3b (right). Measured vs. predicted annual mean NO2 at 98 urban near traffic and urban background validation sites combined 308 for the surface (3a) and column (3b) models. The dashed line is the line of best fit (see Table 3 for fit statistics). The solid line is the line of 309 agreement. Symbols denote different measurement campaigns: solid circles = Perth 1; hollow squares = Sydney 1; hollow diamonds = Sydney 2. 310

Table 1. Details of each of the three sampling campaigns used for validation. 170

Full text

Independent validation of national satellite-based land-use regression models for nitrogen 1
dioxide using passive samplers 2
3
Luke D. Knibbs
1*
,
Craig P. Coorey
2
, Matthew J. Bechle
3
, Christine T. Cowie
4,5,6
, Mila 4
Dirgawati
7
, Jane S. Heyworth
7
, Guy B. Marks
4,5
, Julian D. Marshall
3
, Lidia Morawska
8
, 5
Gavin Pereira
9
, Michael G. Hewson
10
6
7
1
School of Public Health, The University of Queensland, Herston, QLD 4006, Australia 8
2
School of Medicine, The University of Queensland, Herston, QLD 4006, Australia 9
3
Department of Civil and Environmental Engineering, University of Washington, Seattle, 10
WA 98195, USA 11
4
South Western Sydney Clinical School, The University of New South Wales, Liverpool, 12
NSW 2170, Australia 13
5
Ingham Institute of Medical Research, Liverpool, NSW 2170, Australia 14
6
Woolcock Institute of Medical Research, University of Sydney, Glebe, NSW 2037, Australia 15
7
School of Population Health, The University of Western Australia, Crawley, WA 6009, 16
Australia 17
8
International Laboratory for Air Quality and Health, Queensland University of Technology, 18
Brisbane, QLD 4001, Australia 19
9
School of Public Health, Curtin University, Perth, WA 6000, Australia 20
10
School of Geography, Planning and Environmental Management, The University of 21
Queensland, St. Lucia, QLD 4067, Australia 22
* Corresponding author (e: l.knibbs@uq.edu.au; ph: +61 7 3365 5409; fax: +61 7 3365 5540)23
1

Abstract 24
Including satellite observations of nitrogen dioxide (NO
2
) in land-use regression (LUR) 25
models can improve their predictive ability, but requires rigorous evaluation. We used 123 26
passive NO
2
samplers sited to capture within-city and near-road variability in two Australian 27
cities (Sydney and Perth) to assess the validity of annual mean NO
2
estimates from existing 28
national satellite-based LUR models (developed with 68 regulatory monitors). The samplers 29
spanned roadside, urban near traffic (≤100 m to a major road), and urban background (>100 30
m to a major road) locations. We evaluated model performance using R
2
(predicted NO
2
31
regressed on independent measurements of NO
2
), mean-square-error R
2
(MSE-R
2
), RMSE, 32
and bias. Our models captured up to 69% of spatial variability in NO
2
at urban near-traffic 33
and urban background locations, and up to 58% of variability at all validation sites, including 34
roadside locations. The absolute agreement of measurements and predictions (measured by 35
MSE-R
2
) was similar to their correlation (measured by R
2
). Few previous studies have 36
performed independent evaluations of national satellite-based LUR models, and there is little 37
information on the performance of models developed with a small number of NO
2
monitors. 38
We have demonstrated that such models are a valid approach for estimating NO
2
exposures 39
in Australian cities. 40
41
42
43
44
45
46
47
48
2

Introduction 49
Land-use regression (LUR) is frequently used for estimating exposure to outdoor air pollution 50
in epidemiological studies. LUR models use features of the built and natural environment, 51
such as road length, impervious surfaces, and tree cover, to capture spatial variability in 52
pollutant concentrations measured at fixed locations. This allows concentrations at 53
unmeasured locations to be estimated.
1
Several recent studies have shown that the predictive 54
ability of LUR models for nitrogen dioxide (NO
2
), quantified as R
2
, increases by 2 to 15 55
percentage points when satellite-observed tropospheric NO
2
is included as a predictor 56
variable.
2-7
These models aim to leverage the best attributes of satellite observations (e.g., 57
large spatial coverage) and LUR models (e.g., local-scale predictors) to improve performance 58
and coverage compared with either technique alone. 59
60
The spatial coverage offered by satellite data makes it suitable for national or multi-national 61
applications, and satellite-based LUR models have been developed for the USA,
2,7,8
Canada,
6
62
Australia,
5
Western Europe,
3
and the Netherlands.
4
A single national satellite model can offer 63
a simpler and consistent way to assign exposures to geographically dispersed study subjects 64
compared with separate non-satellite LUR models for each city, which are costly and time-65
intensive to develop.
9
Some national models can also offer comparable predictive ability and 66
spatial resolution to city-scale models.
2,7
67
68
LUR models can overfit, particularly when the number of measurement sites is small and the 69
number of potential predictors is large.
10-12
Validation is therefore important for assessing 70
how well they perform when applied beyond the data sets used to develop them (e.g., at the 71
home addresses of subjects in an epidemiological study).
12,13
Numerous LUR validation 72
studies have focused on city-scale models (e.g.,
11,14,15
). In contrast, there is little information 73
3

on validation of satellite-based national NO
2
models,
2,3,7,8
especially in countries with limited 74
ground-based monitoring.
6
Validation of these models is particularly important because they 75
are implemented at a nation-wide scale, which encompasses a wide range of land-use 76
conditions that may differ from the sites used to develop the models. 77
78
In this study, we sought to perform an independent validation of Australian national satellite-79
based LUR models for NO
2
. Through this, we wanted to determine if our models were 80
suitable for estimating residential NO
2
exposures in epidemiological studies. We also aimed 81
to add to the limited literature on satellite-based LUR evaluation by exploring the ability of 82
national models developed with a relatively small number of monitoring sites to predict NO
2
83
concentrations at sites selected to capture within-city and near-road variability. 84
85
Experimental Materials and Methods 86
Models being evaluated 87
We previously described our satellite-based LUR models for NO
2
,
5
which were developed 88
using data from 68 continuous regulatory chemiluminescence monitors throughout Australia 89
(population = 23.5 million; area = 7.7 million km
2
; ~0.3 NO
2
monitors/100,000 persons). 90
Two models using different satellite predictors were developed. One model included the 91
tropospheric column abundance of NO
2
molecules observed by the OMI spectrometer aboard 92
the Aura satellite as a predictor (molecules × 10
15
per cm
2
; ‘column model’). The other model 93
included the estimated NO
2
concentration at ground-level (ppb; ‘surface model’), based on 94
applying a surface-to-column ratio from the Weather Research and Forecasting model 95
coupled with Chemistry (WRF-Chem). Using eight and nine land-use predictor variables, our 96
column and surface models respectively explained 81% (RMSE = 1.4 ppb) and 79% (RMSE 97
4

= 1.4 ppb) and of spatial variability in measured annual mean NO
2
in Australia during 2006-98
11. 99
100
Measurements used for validation 101
In this study, we sought a data set independent of that used in our LUR models’ development 102
to rigorously assess their performance. Because we had previously used most available 103
regulatory air monitoring data for development, we contacted all investigators who had 104
performed NO
2
monitoring as part of epidemiological studies between 2006 and 2014. Our 105
initial inclusion criteria were that: (a) NO
2
had been measured anywhere in Australia 106
provided that repeated, precise coordinates were collected (i.e, to 5 decimal places); (b) 107
measurements ran for at least two weeks, and; (c) a validated measurement method with 108
documented quality assurance procedures was used. We received data from five studies, 109
which, to our knowledge, represented all NO
2
monitoring that met the inclusion criteria. 110
Together, these studies included 174 measurement sites across three of Australia’s six states. 111
112
After preliminary screening we imposed additional, more stringent, inclusion criteria for the 113
studies. Namely, we required three repeated measurements of 14 days’ duration each that 114
spanned different seasons. We aimed to ensure that measurements from different studies 115
captured seasonal variation in NO
2
, were of comparable duration, and able to be converted to 116
an estimated annual mean using standard methods. These criteria were informed by the well-117
described European Study of Cohorts for Air Pollution Health Effects (ESCAPE) protocol for 118
LUR development.
16
Based on this, we excluded two studies comprising 43 measurement 119
sites. 120
121
5

Frequently asked questions

Q1. What have the authors contributed in "Independent validation of national satellite-based land-use regression models for nitrogen" ?

The authors evaluated model performance using R ( predicted NO2 31 regressed on independent measurements of NO2 ), mean-square-error R ( MSE-R ), RMSE, 32 and bias. 38 the authors have demonstrated that such models are a valid approach for estimating NO2 exposures 39 in Australian cities. 

Q2. What was the main source of NO2 at the site?

The site was located 120 m from a hospital that 156 emitted a moderate amount of NOX per year (~5000 kg), but the main source of NO2 was 157 more likely to be traffic emissions because it was also a roadside site. 

Q3. How many sites were used to validate their models?

448449 Although their LUR models were developed using continuous regulatory chemiluminescence 450 monitors the authors validated them using data from Ferm-type and Ogawa passive samplers. 

Q4. How many m distances were used to determine the concentration of NO2?

Where measurements were done across more than 177 one year, the authors averaged the predicted NO2 concentrations to match the measurement period. 

Q5. What was the distance threshold used to capture sites immediately influenced by vehicle emissions?

The 15 144 m distance threshold was selected to capture sites immediately influenced by vehicle 145 emissions, while the 100 m threshold was selected because it represents the approximate half-146life in the decay of NO2 away from a road. 

Q6. What is the mean mean NO2 at roadside sites?

The surface 259 and column models captured 36% (MSE-R2 = -18%) and 29% (MSE-R2 = -13%), 260 respectively, of spatial variability at roadside sites. 

Q7. What is the importance of the absolute agreement between pollutant measurements and LUR model predictions?

The absolute agreement between pollutant measurements and LUR model predictions is 367 important when models are used to assign exposures in epidemiological studies. 

Q8. What is the mean mean NO2 at the 123 validation sites?

The surface and column models captured 58% 237 (MSE-R2 = 51%) and 55% (MSE-R2 = 52%), respectively, of spatial variability in annual 238 mean NO2 at the 123 validation sites overall (Figures 2a and 2b). 

Q9. How many sites are available to develop the models?

Their findings indicate that satellite-based LUR models provide a valid, consistent, and 469 cost-effective method for assigning NO2 exposures, even when the number of sites available 470 to develop them is limited. 

Q10. What might be the reason why Hystad et al. did not validate their model?

It might also reflect that their 343 model had fewer variables (4 predictors vs. 8 and 9 predictors in their models) and was not 344 geared towards detecting emissions attributable to heavy industry and biomass combustion, 345 which the authors noted may have affected their results. 

Q11. Why is the LUR validation study more comparable with their overall results?

336 Because of the diverse siting of validation sites in their study, its results are more comparable 337 with their overall validation results at 123 sites (i.e., including roadside sites). 

Q12. How many industrial point sources of NOX were within 250 m of a site?

There was only one industrial point source of NOX within 250 m of a site, based on the 155 Australian National Pollutant Inventory.