Extraction of Airways from CT (EXACT’09)
Pechin Lo
1
, Bram van Ginneken
2
,
Joseph M. Reinhardt
3
, and Marleen de Bruijne
1,4
1
Image Group, Department of Computer Science, University of Copenhagen,
Denmark, pechin@diku.dk,
2
Image Sciences Institute, University Medical Center Utrecht, The Netherlands,
3
Department of Biomedical Engineering, Iowa Institute for Biomedical Imaging,
The University of Iowa, USA,
4
Biomedical Imaging Group Rotterdam, Departments of Radiology & Medical
Informatics, Erasmus MC, Rotterdam, The Netherlands
Abstract. This paper describes a framework for evaluating airway ex-
traction algorithms in a standardized manner and establishing reference
segmentations that can be used for future algorithm development. Be-
cause of the sheer difficulty of constructing a complete reference standard
manually, we propose to construct a reference using results from the algo-
rithms being compared, by splitting each airway tree segmentation result
into individual branch segments that are subsequently visually inspected
by trained observers. Using the so constructed reference, a total of seven
performance measures covering different aspects of segmentation quality
are computed. We evaluated 15 airway tree extraction algorithms from
different research groups on a diverse set of 20 chest CT scans from
subjects ranging from healthy volunteers to patients with severe lung
disease, who were scanned at different sites, with several different CT
scanner models, and using a variety of scanning protocols and recon-
struction parameters.
1 Introduction
Analysis of the airways in volumetric computed tomography (CT) scans plays
an important role in the diagnosis and monitoring of lung diseases. While airway
segmentation is a key component for such analysis, the accuracy and reliability
of existing algorithms for segmenting the airway are still unknown because of
the lack of a common test database, a reference, and standardized means for
comparison.
The aim of this paper is to develop a framework for evaluating airway extrac-
tion algorithms in a standardized manner and to establish a database with refer-
ence segmentations that can be used for future algorithm development. Because
of the sheer difficulty of establishing a complete reference standard manually,
we propose to instead construct a reference using results from the algorithms to
compare. Airway tree segmentations were first subdivided into their individual
branches, which can be easily visualized and were subsequently evaluated by
EXACT'09
-175-
trained observers. This way, manual annotation is avoided and the trained ob-
server is given the easier task of deciding whether a segmented branch is correct
or wrong, thus separating wrongly labeled regions from correctly labeled regions.
We focus the evaluation on extracting the most complete airway trees, progress-
ing into higher generation airways and extracting as many branches as possible.
The exact airway shape and dimensions were not taken into account; branches
were considered to be correct as long as there was no significant leakage outside
the airway walls. Since several annotations of the same images were evaluated,
the evaluation process can be accelerated by automatically accepting branches
that have high overlap with previously accepted branches. Finally, a reference
segmentation was established by taking the union of all correct branches.
A diverse set of chest CT scans was contributed by eight different institutions.
A total of 40 scans were selected, with 20 of these scans designated as training
set, reserved for algorithm training and/or parameter tuning, and the remaining
20 as test set. The images were selected to include a large variety of acquisition
conditions and pathologies.
The comparative study was organized as a challenge at the 2nd International
Workshop on Pulmonary Image Analysis, which was held in conjunction with
MICCAI 2009. Invitations to participate were sent out to several mailing lists
and to authors of published papers on airway segmentation. Due to the time
required for manual evaluation, participants were asked to submit results from
a single method only. A total of 22 teams registered to download the data, and
15 teams [1–15] submitted their results, ten in the fully automated category
and five in the semi-automated category. All results submitted by participating
teams were used to establish the reference used for evaluation.
2 Data
A total of 75 chest CT scans were contributed by eight different sites. Scans
were taken on several different CT scanner models, using a variety of scanning
protocols and reconstruction parameters. The condition of the scanned subjects
varied widely, ranging from healthy volunteers to patients showing severe pathol-
ogy of the airways or lung parenchyma. Among the contributed CT scans, 40
were selected to be included into the dataset for the challenge, which were further
divided into a training set and a test set. Care was taken to ensure that scans of
all eight sites were represented in both the training and test sets, all files were
anonymized properly, no scans of the same subject would be in both the training
and test sets, and both sets included the same number of scans of similar quality,
obtained at the same site and with similar characteristics. The 20 images in the
training set were named CASE01 to CASE20, and the 20 images in the test set
were named CASE21 to CASE40. Table 1 presents some characteristics of the
20 test cases.
-176-
EXACT'09
Table 1. Acquisition parameters of the 20 test cases. Dosage is presented as x-ray tube
current (kVp) and exposure (mAs) pair. The breathing status indicates full inspira-
tion (Insp.) or full expiration (Exp.). Contrast indicates whether intravenous contrast
was used during acquisition. The abbreviations for the scanner models are as follows:
Siemens Sensation (SS), Siemens Volume Zoom (SVZ), Philips Mx8000 IDT (PMI),
Philips Brilliance (PB), Toshiba Aquilion (TA) and GE LightSpeed (GEL). * indicates
that the scan is from the same subject as the previous scan.
Thickness Scanner Convolution Dosage Breath Contrast
(mm) Kernel state
CASE21 0.6 SS64 B50f 200/100 Exp. No
CASE22*
0.6 SS64 B50f 200/100 Insp. No
CASE23
0.75 SS64 B50f 200/100 Insp. No
CASE24
1 TA FC12 10/5 Insp. No
CASE25*
1 TA FC10 150/75 Insp. No
CASE26
1 TA FC12 10/5 Insp. No
CASE27*
1 TA FC10 150/75 Insp. No
CASE28
1.25 SVZ B30f 300/100 Insp. Yes
CASE29*
1.25 SVZ B50f 300/100 Insp. Yes
CASE30
1 PMI16 D 120/40 Insp. No
CASE31
1 PMI16 D 120/40 Insp. No
CASE32
1 PMI16 D 120/40 Insp. No
CASE33
1 SS16 B60f 321/200 Insp. No
CASE34
1 SS16 B60f 321/200 Insp. No
CASE35
0.625 GEL16 Std. 441/6209 Insp. No
CASE36
1 PB16P C 206/130 Insp. No
CASE37
1 PB16P B 64/30 Insp. No
CASE38*
1 PB16P C 51/20 Exp. No
CASE39
1 SS16 B70f 436/205 Insp. Yes
CASE40
1 SS16 B70s 162/105 Insp. No
EXACT'09
-177-
(a) (b) (c) (d) (e)
Fig. 1. Illustration of the process of dividing the airway tree into branches. (a) A seed
point is set at the root of a tree and the front propagation process starts. (b) Centroid
of the propagating front is stored as centerline at each time step. (c) The centerline
is stored up to the point before a bifurcation. (d) Bifurcation is detected as the front
splits. The individual fronts are used as seeds to perform front propagation in each of
the branches. (e) Front propagation proceeds in each branch.
3 Airway branch evaluation
3.1 Subdividing an airway tree into branches
An important component of our evaluation is the process of subdividing an
airway tree into its individual branch segments. This was done by detecting the
bifurcations using wave front propagation, similar to [16]. The key idea is that
a wave front propagating through a tree structure remains connected until it
encounters a bifurcation, and any side branches can thus be detected as new
disconnected components in the propagating front.
The front was propagated using a fast marching algorithm [17], with the
speed function set to 1 within the segmented structure and 0 outside. We mon-
itor the front through a set of “trial” points in the fast marching process. Con-
nected component analysis was applied to the trial points when the time stamp
from the fast marching algorithm increased by 1/ΔD, where ΔD is the distance
between two voxels. Propagation stops when multiple disconnected components
were detected in the front, whereupon the process was repeated on the individual
split fronts to obtain the branches at the next level. The process ends when all
marked regions of the tree have been evaluated. During the front propagation,
the centerlines were obtained by storing the centroid of the front at every step.
Figure 1 illustrates the algorithm.
3.2 Visual assessment
To enable visual inspection of extracted branches, each of the branches was
presented to the trained observers using a fixed number of slices through the
branch at different positions and orientations. Two different views were used,
a reformatted and a reoriented view. The reformatted view was obtained by
straightening the centerline of the branch segment, while the reoriented view
-178-
EXACT'09
was obtained by rotating the branch segment such that its main axis coincides
with the x-axis.
A total of eight slices were extracted from the reformatted view. The first
four slices were taken perpendicular to the centerline, distributed evenly from
the start to the end of the centerline. The other four slices were taken along the
centerline, at cut planes that were angled at 0
◦
,45
◦
,90
◦
and 135
◦
. A schematic
view of the cut planes for the different slices for the reformatted view is shown
in Figure 2(b), with slice examples in Figure 2(d).
For the reoriented view, a total of nine slices were extracted, three from
each axis. For the y-axis and z-axis, which are perpendicular to the axis of the
branch segment, the cut planes were placed at 15%, 50% and 85% of the width
of the branch measured in the respective axis. On the x-axis, the cut planes were
placed at 5%, 50% and 95% of the length of the branch. Figure 2(b) shows the
cut planes for the slices extracted from reoriented view, and examples of the
slices are shown in Figure 2(e).
3.3 Evaluation of individual branches
Based on the slices from the two views described in Section 3.2, the trained
observers were asked to assign one of the four following labels to each branch:
“correct”, “partly wrong”, “wrong” or “unknown”. A branch is “correct” if it
does not have leakage outside the airway wall. “Partly wrong” is assigned to
a branch if part of the branch lies well within the airway lumen, while part is
outside the walls. A branch is “wrong” if it does not contain airway lumen at
all. The “unknown” label is used when the observers are unable to determine
whether a branch is an airway or not.
The evaluation of each branch was performed in two phases. At phase one,
two observers are assigned to evaluate the branch. If both observers assign the
same label, the evaluation is complete. Otherwise the evaluation proceeds to
phase two, where three new observers are assigned. In this phase, the final label
assigned to the branch is the label that constitutes the majority vote among the
three new observers. In case of no majority, the branch is labeled as “unknown”.
The entire process was automated by web-based distribution of tasks to a pool
of ten trained observers, who labeled the 15 segmentations of each of the 20
test cases. The observers were all medical students who were familiar with CT
and chest anatomy. They were trained with a set of examples and subsequently
processed two complete segmentations of different scans to ensure their ratings
were reliable.
4 Establishing a reference
The reference segmentation for a CT scan was constructed by fusing branches
based on the labels assigned as described in Section 3, which involves evaluated
branches of segmented airway trees obtained from all participating teams.
EXACT'09
-179-
