General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright
owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
Users may download and print one copy of any publication from the public portal for the purpose of private study or research.
You may not further distribute the material or use it for any profit-making activity or commercial gain
You may freely distribute the URL identifying the publication in the public portal
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
Downloaded from orbit.dtu.dk on: Aug 10, 2022
Plane-Wave Imaging Challenge in Medical Ultrasound
Liebgott, Herve; Molares, Alfonso Rodriguez; Jensen, Jørgen Arendt; Cervenansky, F.; Jensen, Jørgen
Arendt; Bernard, O.
Published in:
Proceedings of 2016 IEEE International Ultrasonics Symposium.
Link to article, DOI:
10.1109/ULTSYM.2016.7728908
Publication date:
2016
Document Version
Peer reviewed version
Link back to DTU Orbit
Citation (APA):
Liebgott, H., Molares, A. R., Jensen, J. A., Cervenansky, F., Jensen, J. A., & Bernard, O. (2016). Plane-Wave
Imaging Challenge in Medical Ultrasound. In Proceedings of 2016 IEEE International Ultrasonics Symposium.
IEEE. https://doi.org/10.1109/ULTSYM.2016.7728908
Plane-Wave Imaging Challenge
in Medical Ultrasound
H. Liebgott
1
, A. Rodriguez-Molares
2
, F. Cervenansky
1
, J.A. Jensen
3
and O. Bernard
1
1
Univ Lyon, INSA-Lyon, Université Lyon 1, CNRS, Inserm, CREATIS UMR 5220, U1206, F-69621, LYON, France
2
Department of Circulation and Medical Imaging, Norwegian University of Science and Technology, Trondheim, Norway
3
Center for Fast Ultrasound Imaging, Department of Electrical Engineering, Technical University of Denmark, DK-2800 Lyngby, Denmark
Abstract—: Plane-Wave imaging enables very high frame
rates, up to several thousand frames per second.
Unfortunately the lack of transmit focusing leads to
reduced image quality, both in terms of resolution and
contrast. Recently, numerous beamforming techniques
have been proposed to compensate for this effect, but
comparing the different methods is difficult due to the lack
of appropriate tools. PICMUS, the Plane-Wave Imaging
Challenge in Medical Ultrasound aims to provide these
tools. This paper describes the PICMUS challenge, its
motivation, implementation, and metrics.
Keywords—ultrasound, challenge, Plane-Wave, beamforming,
ultrafast.
I. INTRODUCTION
Researchers and engineers in the field of medical ultrasound
have always tried to improve image quality and frame rates.
This was already the objective with synthetic aperture imaging
(STA) [1]. More recently, our domain has undergone a
revolution with the emergence of Plane-Wave Imaging (PWI),
which has been applied to most fields in medical ultrasound
[2] yielding framerates as high as several thousands of images
per second. At first this increase in framerate was obtained at
the expense of reduced contrast and resolution. However, this
drawback was skilfully addressed by Coherent Plane-Wave
Compounding (CPWC) [3], which introduced a trade-off
between framerate and image quality. Note that compounding
is also used in STA.
Beamforming has therefore regained a lot of its past attention,
with the proposal of numerous beamforming techniques that
aim to increase image quality without losing framerate. In
addition, some design choices can have a significant impact on
the resulting quality, such as the choice of the steering angles,
apodization, or probe design. On the other hand, more
sophisticated techniques often come at the cost of increased
processing time or computing power, which may hinder real-
time operation with state-of-the-art technology.
Unfortunately most of the proposed techniques are only
compared to delay-and-sum. In some selected cases new
methods are tested against other contemporary techniques, but
understandably, the amount of time and fine tuning devoted to
these auxiliary implementations is rarely the same as used for
the proposed method. In addition, the testing conditions vary
from paper to paper (simulation and experimental parameters,
probe and scanner settings, imaged medium, etc.), which
makes it even more difficult to generalize the results. Finally
the metrics used for the assessment of image quality can also
vary from one study to other. Altogether, this makes plane-
wave imaging a good topic for a research challenge with a
consistent set of performance metrics.
Challenges are regularly organized and hosted by medical
imaging conferences like IEEE International Symposium on
Biomedical Imaging [http://biomedicalimaging.org/2016/] or
Medical Image Computing and Computer Assisted
Intervention MICCAI [http://www.miccai2016.org/en/]. The
idea is to provide to the participants a number of datasets to be
processed by their method. The results are then uploaded to a
web platform and evaluated with a set of predefined metrics.
So the methods can be objectively compared.
For the first time in history a challenge is organized by the
IEEE International Ultrasonics Symposiun, to be held in
Tours, France the 21 September of 2016. In the remaining we
describe the design and implementation of the first edition of
the Plane-Wave Imaging Challenge in Medical Ultrasound
(PICMUS).
II. PICMUS
The aim of PICMUS is to provide the community with a tool
to objectively compare newly proposed beamforming
methods. The challenge includes:
x a website with all available information,
x the challenge platform (MIDAS),
x two simulated CPWC datasets in HDF5 format,
x two experimental CPWC datasets in HDF5 format,
x example code, in MATLAB, on how to interact with
the datasets, and
x tests for contrast, resolution, geometrical distortion,
and speckle appearance.
These features are further described in the following sections.
Description of the datasets
PICMUS has been designed for PWI and CPWC beamforming
techniques. Other beam shapes, such as focused beams,
diverging waves, and synthetic transmit aperture are not
included.
Each dataset contains 75 steered Plane-Waves covering the
angle span from -16° to 16°. Each dataset is available in RF
(modulated) and IQ (demodulated) format. The dataset were
stored in HDF5 (Hierarchical Data Format).
Two datasets were generated with Field II [4, 5]. The
parameters used in the simulation, shown in Table 1, were fine
tuned to correspond as much as possible to the experimental
setup. A first dataset consisted of isolated scatterers distributed
vertically and horizontally over an anechoic background (see
Figure 3a). A second simulated dataset included a number of
anechoic cysts, also distributed vertically and horizontally,
over fully developed speckle (see Figure 3c).
Table 1: Imaging parameters used both in simulations and
for the experiments
Pitch
0.30 mm
Element width
0.27 mm
Element height
5 mm
Elevation focus
20 mm
Number of elements
128
Aperture width
38.4 mm
Transmit frequency
5.208 MHz
Sampling frequency
20.832 MHz
Pulse bandwidth
67%
Excitation
2.5 cycles
Two experimental datasets were provided. Data were acquired
using a Verasonics Vantage 256 research scanner and a L11
probe (Verasonics Inc., Redmond, WA). The datasets were
recorded on a CIRS Multi-Purpose Ultrasound Phantom
(Model 040GSE) in the regions indicated in Figure 1. A first
dataset contained several wires against speckle background
(see Figure 3b). A second dataset contained two anechoic
cysts against speckle background (see Figure 3d).
Besides, a collection of MATLAB classes were provided to
the participants specifying how to interact with the datasets,
together with a reference implementation of delay-and-sum
(DAS) beamforming. The participants were then asked to
beamform the four datasets on a specific grid of points, and to
supply the envelope image before any kind of compression.
Figure 1
Schematic of the upper part of the CIRS Model
040GSE Phantom used for the experiments. The highlighted left
region was acquired for contrast evaluation and the right region
for resolution evaluation.
Description of the metrics
Contrast and resolution are, by far, the most used metrics to
assess image quality. But it is also important to check for
geometrical distortion and good speckle statistics.
To estimate resolution the full width at half maximum
(FWHM) was evaluated both in axial and lateral directions.
The FWHM obtained for all scatterers in the image were
averaged to obtain the average axial and lateral resolution in
both simulations and experiments (Figures 3a and 3c).
Contrast was estimated with the classical expression for
contrast to noise ratio (CNR),
10
22
20log
/2
in out
in out
CNR
PP
VV
§·
¨¸
¨¸
©¹
(1)
where μ
in
and μ
out
are the mean gray level inside and outside
the anechoic cystic region, and
V
in
and
V
out
are the gray level
standard deviation inside and outside the anechoic cystic
region. As for the resolution, the CNR for all the cysts in the
image were averaged leading to one CNR for the simulations
and one for the experiments (Figures 3b and 3d).
Geometrical accuracy was verified by investigating the
position of the points scatterers (Figures 3a and 3c). For the
simulations the true position of the scatterers was used as
reference. For the experiments, the provided DAS
implementation was used as reference. Geometrical distortion
was penalyzed if the maximum distance of any scatterer from
its theoretical position was greater than one wavelength.
Considering that speckle is an intrinsic characteristic of
ultrasound images, it was decided to penalize methods that
removed speckle. It is well-know that the intensity of fully
developed speckle follows a Rayleigh distribution. For a set of
predefined regions, the Kolmogorov–Smirnov (KS) test was
applied. The speckle quality test was considered positive, if
the tested regions obtained a significance level α = 0.05 in the
KS test.
Description of the competition and the ranking
Research challenges are inherently competitive. Healthy
competition has proven itself as a robust and reliable way to
drive research and spur discussion. To achieve this it is
necessary to combine the metrics into a single value, so that
participants can be ranked. Yet, the direct comparison and
discussion of the individual metrics is, most probably, of more
scientific value.
Four categories have been set depending on the number of
plane-wave selected by the participants
x Category I: 1 plane-waves,
x Category II: 11 plane-waves,
x Category III: 75 plane-waves,
x Category IV: arbitrary number of plane-waves.
In all categories, except in category III which uses all of them,
the angles can be chosen arbitrarily . In category IV the score
is normalized by the number of Plane-Waves used with the
following expression
_
_
1
5
all metrics
pts
final score
NB
§·
¨¸
¨¸
¨¸
©¹
¦
(2)
where NB is the number of Plane-Waves used by the
participant and pts is the number of points received by the
participants for a given metric.
Material available
All the information is available at the website
https://www.creatis.insa-lyon.fr/Challenge/IEEE_IUS_2016/.
It describes the datasets, the metrics, provides a link to the
data and the code, as well as some more general information
about the organization of the challenge.
Participants get access to the challenge through the web
platform http://challenge.creatis.insa-
lyon.fr/IEEE_IUS_2016/community/1#tabs-info. It is the
intention of the organisers to maintain the platform as long as
possible and even keep extending it in the future to support
comparison with other methods after the challenge.
The platform includes a collection of Matlab classes that
facilitate the interaction with the HDF5 dataset and illustrate
how the metrics are calculated.
III. E
XAMPLES
Figure 2 (respectively Figure 3) shows the images obtained
with the DAS method provided as reference, when a single 0
degrees plane-wave is transmitted (respectively when the 75
transmitted plane-waves are combined). The two Figures show
images of the four provided datasets. From left to right Figure
2 and Figure 3 show first the two images used for resolution
evaluation in simulations and experiments, and then the two
images used for contrast evaluation in simulations and
experiments. As expected using 75 plan-waves leads to better
contrast and resolution both in simulations and
experimentally.
a
b
c
d
Figure 2 Example of simulated (a and c) and experimental (b and d) images obtained with the DAS provided within PICMUS for the
resolution (a and b) and contrast (c and d) images with only one Plane-Wave. Axial and lateral dimensions are given in mm, and the
grey scale ranges from -60 to 0dB.
-10
0
10
10
20
30
40
-10
0
10
10
20
30
40
-10
0
10
10
20
30
40
-10
0
10
1
2
3
4
-60
-50
-40
-30
-20
-10
0
a
b
c
d
Figure 3 Example of simulated (a and c) and experimental (b and d) images obtained with the DAS provided within PICMUS for the
resolution (a and b) and contrast images (c and d) with all 75 Plane-Waves. Axial and lateral dimensions are given in mm, and the grey
scale ranges from -60 to 0dB.
IV. D
ISCUSSION AND CONLUSION
PICMUS is a tool to assess the relevance of new beamforming
methods and compare their performance with other state-of-
the-art methods in an objective and standardized way.
Unfortunately, due to practical reasons and several time
constrains, the scope of PICMUS was limited.
First of all the choice has been made to consider only plane
wave imaging. Other transmit strategies could also be
interesting topics for a challenge, like for example synthetic
aperture imaging. Also transmit apodization could have been
considered.
For instance, only static phantoms were considered. Motion
artefacts can dramatically affect the images quality of fast
moving objects. When motion is considered not only the
number of transmitted angles is important, but also the order
in which they have been transmitted.
Phase aberration was not covered by the provided datasets.
This could be easily included, both in simulations and
experimentally, by adding a phase aberration layer on top of
the phantoms introducing a small error in the phase. Equally
simple would have been to evaluate how sensitive the methods
were to sound speed errors.
Also PICMUS does not take into account the processing time,
which might be decisive in the assessment of the methods
applicability. One way of including this metric would be to
ask the participants to upload and run their code on the same
online platform. However, in some cases intellectual property
issues may have to be addressed. In addition, implementation
in different language codes, or architectures (for instance GPU
vs CPU based) may hinder a comparison of the algorithm’s
computation complexity.
In this first edition of the challenge we opted for a reduced
scope for the sake of clarity and conciseness. But future
editions could increase the scope and improve the metrics.
We believe utterly important to maintain the platform in time,
open for other users to use and contribute, supporting the
objective intercomparison of methods and promoting fruitful
discussion on the relevance of new ideas.
A
CKNOWLEDGMENT
The authors would like to thank Verasonics who supported
the challenge financially and particularly Mike Vega. Special
thanks to Peter Krakovski for his advice on data acquisition.
The authors would also like to thank the IEEE IUS
organization committee and technical program committee as
well as all members of the ultrasound community who
supported this effort.
This work was performed within the framework of the
Labex PRIMES (ANR-10-LABX-0063) of Université de Lyon,
within the program "Investissements d'Avenir" (ANR-11-
IDEX-0007) operated by the French Nation-al Research
Agency (ANR). Part of this work was also supported by the
Center for Innovative Ultrasound Solutions for health care,
maritime, and oil & gas, CIUS which is a Norwegian Research
Council appointed centre for research-based innovation.
R
EFERENCES
[1] J. T. Ylitalo and H. Ermert, "Ultrasound synthetic aperture
imaging: monostatic approach," IEEE Transactions on Ultrasonics,
Ferroelectrics, and Frequency Control, vol. 41, n°3, pp. 333-339,
1994.
[2] M. Tanter and M. Fink, "Ultrafast imaging in biomedical
ultrasound," IEEE Transactions on Ultrasonics, Ferroelectrics, and
Frequency Control, vol. 61, n°1, pp. 102-119, 2014.
[3] G. Montaldo, M. Tanter, J. Bercoff, N. Benech, and M. Fink,
"Coherent plane-wave compounding for very high frame rate
ultrasonography and transient elastography," IEEE Transactions on
Ultrasonics, Ferroelectrics and Frequency Control, vol. 56, n°3, pp.
489-506, 2009.
[4] J. A. Jensen and N. B. Svendsen, "Calculation of pressure fields
from arbitrarily shaped, apodized and excited ultrasound
transducers," IEEE Transactions on Ultrasonics, Ferroelectrics and
Frequency Control, vol. 39, n°2, pp. 262-267, March 1992 1992.
[5] J. A. Jensen, "Field: A Program for Simulating Ultrasound
Systems," Medical & Biological Engineering & Computing, vol. 34,
n°Supplement 1, Part 1, pp. 351-353, 1996.
-10
0
10
10
20
30
40
-10
0
10
10
20
30
40
-10
0
10
10
20
30
40
-10
0
10
10
20
30
40
-60
-50
-40
-30
-20
-10
0