DiFX-2: A More Flexible, Efficient, Robust, and Powerful Software Correlator
Author(s): A. T. Deller, W. F. Brisken, C. J. Phillips, J. Morgan, W. Alef, R. Cappallo, E.
Middelberg, J. Romney, H. Rottmann, S. J. Tingay, R. Wayth
Reviewed work(s):
Source:
Publications of the Astronomical Society of the Pacific,
Vol. 123, No. 901 (March
2011), pp. 275-287
Published by: The University of Chicago Press on behalf of the Astronomical Society of the Pacific
Stable URL: http://www.jstor.org/stable/10.1086/658907 .
Accessed: 12/03/2012 04:33
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
The University of Chicago Press and Astronomical Society of the Pacific are collaborating with JSTOR to
digitize, preserve and extend access to Publications of the Astronomical Society of the Pacific.
http://www.jstor.org
DiFX-2: A More Flexible, Efficient, Robust, and Powerful Software Correlator
A. T. D
ELLER
,
1,2
W. F. B
RISKEN
,
1
C. J. P
HILLIPS
,
3
J. M
ORGAN
,
4
W. A
LEF
,
5
R. C
APPALLO
,
6
E. M
IDDELBERG
,
7
J. R
OMNEY
,
1
H. R
OTTMANN
,
5
S. J. T
INGAY
,
4
AND
R. W
AYTH
4
Received 2010 September 20; accepted 2010 December 27; published 2011 February 23
ABSTRACT. Software correlation, where a correlation algorithm written in a high-level language such as C++ is
run on commodity computer hardware, has become increasingly attractive for small- to medium-sized and/or band-
width-constrained radio interferometers. In particular, many long-baseline arrays (which typically have fewer than
20 elements and are restricted in observing bandwidth by costly recording hardware and media) have utilized soft-
ware correlators for rapid, cost-effective, correlator upgrades to allow compatibility with new, wider-bandwidth,
recording systems and to improve correlator flexibility. The DiFX correlator, made publicly available in 2007,
has been a popular choice in such upgrades and is now used for production correlation by a number of observatories
and research groups worldwide. Here, we describe the evolution in the capabilities of the DiFX correlator over the
past three years, including a number of new capabilities, substantial performance improvements, and a large amount
of supporting infrastructure to ease use of the code. New capabilities include the ability to correlate a large number
of phase centers in a single correlation pass, the extraction of phase-calibration tones, correlation of disparate but
overlapping sub-bands, the production of rapidly sampled filter-bank and kurtosis data at minimal cost, and many
more. The latest version of the code is at least 15% faster than the original (and in certain situations, many times this
value). Finally, we also present detailed test results validating the correctness of the new code.
Online material: color figures
1. INTRODUCTION
Development of the Distributed FX (DiFX) software corre-
lator began in 2005, primarily for usage with the Australian
Long Baseline Array (LBA) as part of a sensitivity upgrade pro-
gram (Deller et al. 2007), where it entered production usage in
2006. It is an FX-style correlator (see, e.g., Chikada et al. 1987;
Thompson et al. 1994; Romney 1999) designed to run on mod-
ern CPUs under Linux or Mac OS X. The basic principles of
radio astronomy cross-correlator fundamentals will not be
rederived in this article, which focuses on the particular imple-
mentation of the DiFX software correlator. We direct the reader
to the preceding references for a thorough explanation of FX-
style correlator functionality and to Deller et al. (2007) for a
comprehensive description of the specific implementation of
this functionality in the DiFX code.
The DiFX code is accelerated using vector arithmetic li-
braries: specifically, the Intel Integrated Performance Primitives
(IPP) library,
8
and the distribution across multiple nodes is en-
abled using implementations of the Message Passing Interface.
9
The advantages brought by the adoption of a newer, more flex-
ible, correlator architecture were enumerated by Deller et al.
(2007) and included greater flexibility in the setting of correla-
tion parameters, lower cost, rapid development, ease of main-
tenance, and upgradability (both in hardware and software). In
keeping with this final point, development of the DiFX software
correlator has continued rapidly since its first public release in
2007. Since that time, many new features and performance im-
provements have been merged into the DiFX codebase, which
cumulatively are sufficient to merit a major version increment
for the DiFX package, which we designate DiFX-2. Where nec-
essary, any release of DiFX prior to DiFX-2 will be referred to
generically as DiFX-1.x, acknowledging that this time period
spanned a series of release numbers. Some of the features pre-
sented here were made available early in the DiFX-1.x series
and have been available for several years, while others were
added recently and are only available in DiFX-2.
1
National Radio Astronomy Observatory, PO Box O, Socorro, NM 87801.
2
UC Berkeley, 601 Campbell Hall, University of California at Berkeley,
Berkeley, CA 94720.
3
Australia Telescope National Facility, CSIRO Astronomy & Space Sciences,
PO Box 76, Epping, NSW, Australia.
4
International Centre for Radio Astronomy Research, Curtin University, GPO
Box U1987, Perth, WA, Australia.
5
Max-Planck-Institut für Radioastronomie, Postfach 20 24, 53010 Bonn,
Germany.
6
MIT Haystack Observatory, Westford, MA 01886.
7
Ruhr-Universität Bochum, Astronomisches Institut, NA 7/73, Universität-
stra. 150, D-44801 Bochum, Germany.
8
See http://software.intel.com/en‑us/intel‑ipp/.
9
See http://www.mcs.anl.gov/research/projects/mpi/.
275
P
UBLICATIONS OF THE
A
STRONOMICAL
S
OCIETY OF THE
P
ACIFIC
, 123:275–287, 2011 March
© 2011. The Astronomical Society of the Pacific. All rights reserved. Printed in U.S.A.
DiFX has been adopted by a number of leading VLBI facil-
ities in addition to the LBA. Specifically, the Very Long Base-
line Array (VLBA) operated by the National Radio Astronomy
Observatory (NRAO
10
) in the US has retired its hardware
correlator, which was designed in the 1980s, and migrated com-
pletely to DiFX. In addition, the Max Planck Institute for Radio -
astronomy (MPIfR) has begun routine operation of DiFX in
parallel with the existing Mark4 hardware correlator operations
and plans to phase out its Mark4 hardware correlator by the end
of 2010. The specific needs of the LBA, VLBA, and MPIfR
have driven the development of many of the new capabilities
of DiFX, including the features discussed in this article, which
allow entirely new areas of long-baseline science to be under-
taken. All of these developments have been made available to all
current and potential users.
Almost as important for new users of DiFX, considerable
effort has been made to improve the documentation and online
resources available for installing and testing DiFX.
11
Two mail-
ing lists are available to seek or disseminate information regard-
ing DiFX: one list reaches the entire DiFX community, while
the second is focused specifically on code developers. Prospec-
tive users of DiFX are directed to the online resources, where
further information is available on how to obtain the code from
the central repository. Finally, improved version control has
been implemented since the early days of DiFX-1.x, and tagged
releases of frozen code are made available on a regular basis.
The current stable version of DiFX-2 is DiFX-2.0.0, and the
current version of the previous series is DiFX-1.5.4.
In this article, we describe the new DiFX-2 functionality in
§ 2. The performance improvements which have been provided
in the accompanying code changes are listed and quantified in
§ 3. Additional code that provides functionality for DiFX-2 that
is not directly part of the correlator itself is described in § 4, and
the validation testing undertaken for DiFX-2 is described in § 5.
Future work is discussed in § 6, and our conclusions are pre-
sented in § 7.
2. NEW FEATURES
2.1. FITS-IDI and Mark4 Format Output
Initially, DiFX-1.x only supported the RP FITS
12
file format,
which was the format historically used by the LBA. However,
RPFITS is not a standard FITS format and has limited support in
most radio astronomy postprocessing packages. Accordingly, in
version 1.5.0 the ability to produce FITS-IDI format correlator
files (as produced by the VLBA and the Joint Institute for VLBI
in Europe [JIVE] hardware correlators) was added to DiFX.
Unlike the RPFITS files written by early versions of DiFX-
1.x, the FITS file is not directly written by the correlator, but
is translated from a DiFX binary output written by DiFX-2 after
correlation has completed. This eliminates the need to link large
FITS libraries into DiFX and simplifies and speeds the output
writing process. Support for the RPFITS format has been with-
drawn in DiFX-2.
Geodetic observers typically use specialized postproces-
sing software such as HOPS,
13
which is closely tied to the vis-
ibility data format produced by the Mark4 hardware correlator
(Whitney et al. 2004). In order to facilitate the use of DiFX-2 for
geodetic observations, an additional translation program has
been written to produce these Mark4 format visibility data sets
from the binary DiFX output. The ability to import DiFX format
output data directly into the HOPS geodetic postprocessing
package is new in DiFX-2.
2.2. Native Mark5 Interface
The Mark5 recording media series (Whitney 2003) is widely
used among VLBI networks, with the VLBA, the Euro pean
VLBI Network (EVN), the Korean VLBI Network (KVN),
and global geodetic arrays all utilizing the system. Initially,
however, DiFX-1.x could not read data directly from Mark5
disk modules, as it required the data to be accessible from a
standard Linux file. Correlating Mark5 data therefore required
a tedious intermediate step of exporting the files to a Linux file
system, which imposed additional overhead in time and storage
space. Since version 1.5.0, DiFX has had the ability to read
Mark5 disk modules natively, using the application interface
available from the Mark5 vendor. This eliminates the need to
export data from modules to standard files, streamlining the cor-
relation process and reducing the need for large amounts of stan-
dard disk storage. The support of VLBA and Mark4 data
formats on both standard Linux files and Mark5 modules has
been extended to include Mark5B, and limited support is al-
ready in place for the next-generation VDIF format described
by Whitney et al. (2009).
2.3. Phase-Calibration-Tone Extraction
Phase-calibration tones can be injected at the front end of a
radio astronomy antenna in order to provide a convenient means
to estimate instrumental delays. For applications such as ge-
odesy, phase-calibration tones are heavily relied upon. While
some radio telescope arrays extract, average, and store phase-
calibration-tone information at the antenna, others rely on the
correlator to perform this important function. Accordingly, a
flexible phase-calibration-tone extraction system has been added
to DiFX-2. The phase-calibration extraction in DiFX-2 can be
configured to extract any number of tones, unlike many existing
10
The National Radio Astronomy Observatory is a facility of the National
Science Foundation operated under cooperative agreement by Associated
Universities, Inc.
11
See http://cira.ivec.org/dokuwiki/doku.php/difx/start.
12
See http://www.atnf.csiro.au/computing/software/rpfits.html.
13
See http://www.haystack.mit.edu/tech/vlbi/hops.html.
276 DELLER ET AL.
2011 PASP, 123:275–287
hardware implementations such as those at the VLBA stations
(which provide two tones per sub-band). Preliminary results
comparing DiFX corrections with those extracted at the VLBA
stations show agreement to ≪1° (corresponding to ∼ femto-
seconds at 43 GHz) and also show that the computational over-
head of extracting all the phase-calibra tion tones present
14
is
∼5%. A detailed analysis of the performance of DiFX on geode-
tic observations, including the verification of DiFX-2 phase-
calibration extraction and the production of Mark4 format
visibility data, is deferred to a future publication (Morgan et al.
2011, in preparation).
2.4. Spectral Selection and Averaging
Once the data have been channelized (the “F” portion of the
FX algorithm), it is possible to discard segments of the spectrum
that hold no interest for the current observation. The main ap-
plication of such spectral selection is to zoom in on widely sep-
arated spectral features such as masers that are contained within
a wide bandwidth. Use of this new feature in DiFX-2, which we
generically term “zoom mode,” reduces the load on the cross-
multiply/accumulate (“X”) portion of the FX algorithm and,
more importantly, reduces the amount of data that must be re-
turned to the manage r node for long-term accumulation (see
Deller et al. 2007). This allows very high spectral resolution
to be obtained without overloading the correlator interconnect,
without generating unduly large amounts of data to be written to
intermediate disk results and later discarded (as was the case
with all versions of DiFX-1.x).
An alternate application of spectral selection, which we
generically term “band-matching,” facilitates the correlation
of heteroge neous recorded bands by subdividing wider bands
recorded at some antennas to match narrower bands recorded
at other antennas. This is a particularly useful feature for cor-
relating infrequently used antennas with nonstandard VLBI
back-end systems, which are unable to produce bands compa-
tible with other VLBI systems. A current example involves
geodetic correlations where 32 MHz bands recorded at most
stations are correlated against 16 MHz bands recorded at the
Plateau de Bure interferometer. The correlation of upper side-
band data with lower sideband data (covering the same spectral
range) is also supported.
Spectral averaging allows multiple spectral points to be aver-
aged after correlation, but before the visibilities are returned to
the manager node. This is useful in two cases. The first is when
the desired final spectral resolution is low. In this instance, gen-
erating such coarse spectral resolution directly by means of a
very short Fourier transform is not efficient, so use of an opti-
mally sized transform (typicall y with a length of ∼256 points) is
preferred, with spectral averaging before the visibilities are
returned to the manager. As with spectral selection, this saves
correlator interconnect and disk-space resources.
The second, and more important, usage of spectral averaging
is with the multiple-phase-center correlations described in the
following section. In order to avoid bandwidth decorrelation ef-
fects as described subsequently, very high spectral resolution is
required initially. After the visibilities have been shifted to the
desired phase center, however, they can be averaged to a stan-
dard VLBI resolution. Spectral averaging is critical for this ap-
plication, as the return of multiple copies of very high spectral
resolution visibilities would completely overwhelm the correla-
tor interconnect.
2.5. Multiple Simultaneous Phase Centers
Because of the very high fringe rates inherent in VLBI
observations, the use of standard frequency (hundreds of kilo-
hertz) and time resolution (seconds) leads to an extremely small
(several-arcsecond) field of view (see, e.g., Middelberg et al.
2010). We hereafter refer to a narrow field of view resulting
from standard VLBI correlation parameters as a “pencil beam.”
Thompson et al. (1994) provide a detailed explanation of the
challenges inherent in wide-field imaging. While improving
the temporal and spectral resolution allows somewhat wider
fields of view (and was indeed one of the main drivers of DiFX
development), this carries an increasingly problematic cost in
expanded data volume. Mapping even one-tenth of the primary
beam of the VLBA at 1.4 GHz with no more than 10% decorr-
elation due to time and bandwidth decorrelation requires 8 kHz
frequency channels and an integration time of 0.1 s—which
yields visibility data sets >2 Tbyte for a typical 12 hr VLBA
observation at current bandwidths.
An alternative to mapping large swaths of sky (which, in any
case, are almost entirely empty at VLBI resolution at centimter
frequencies) is to image small areas around known sources. This
can be accomplished by shifting the phase center of the correla-
tion (a uv shift) to the location of known sources and averaging
visibilities to obtain manageable-sized data sets, which can be
used to produce pencil beams at new locations (see, e.g., Lenc
et al. 2008). A uv shift is implemented by calculating the base-
line-based differential geometric delay between the desired and
applied phase center, converting to a phase rotation for each vis-
ibility by multiplying this delay by the associated sky frequency,
and rotating the visibility phases by this value. Morgan et al.
(2010) examine the problem of uv shifting in more detail, in-
cluding the detailed calculation of the necessary delay shifts.
The drawback of using this approach after correlation is the ne-
cessity of generating an initial visibility data set that is as large
as that required for a single large image. The intermediate data
volume problem is therefore comparable with that experienced
with the singl e-large-image approach, and the I/O cost of writ-
ing visibilities to, and reading from, disk is substantial.
If implemented within the correlator, however, the twin
problems of I/O and storage volume are solved, because the
14
Spaced at 1 MHz intervals, so typically 8 or 16 tones per sub-band.
DIFX-2: AN UPDATED AND IMPROVED SOFTWARE CORRELATOR 277
2011 PASP, 123:275–287
intermediate data products (the high spectral resolution visi-
bilities held at the processing nodes) do not need to be trans-
mitted from where they are calculated and are never written
to disk. Obtaining sufficiently high time resoluti on is trivially
implemented—the time division multiplexing within DiFX
(see Deller et al. 2007) already provides time resolution better
than that required in most cases, but the ability to uv shift and
average after a shorter, user-specified, time has also been imple-
mented. For P phase centers, the processing nodes transmit P
normal-sized (postaverage) collections of vi sibility results back
to the manager node, and P normal-sized data sets are
ultimately written to disk. The impact on performance of this
feature is relatively small, due to the fact that the uv shift/
average operations need only be carried out relatively infre-
quently (compared with the multiplications, additions, and
Fourier transforms required for the regular correlation process).
The visibility amplitudes and weights are corrected for time
and bandwidth decorrelation online, before the visibilities are
written to disk. The resultant P pencil beams can be reduced
and imaged using standard tools. Presently, these corrections
are not tabulated and saved, since no postprocessing software
exists that could parse and use this information. The information
is readily available, however, and could be formatted and writ-
ten out when suitable postprocessing becomes available.
Thus, as long as low-resolution finder catalogs are available,
VLBI-resolution surveying is possible with DiFX-2 with mini-
mal overhead. Figure 1 illustrates the use of a low-resolution
image and the VLBI data sets that would result from a multiple-
field correlator pass. Middelberg et al. (2010) have already used
this new capability to carry out pilot VLBI survey observations
in the Chandra Deep Field South, and these observations were
instrumental in the development and refinement of the new cor-
relation mode. Section 3 describes in detail the performance
impact of adding multiple phase centers to a correlation. Sec-
tion 5 shows the verification that uv shifted visibility data sets
have no residual phase or amplitude errors. This feature is new
in DiFX-2.
2.6. Correct Model Accountability
The RPFITS output format used initially by DiFX-1.x had no
means to store an accurate representation of the delay model
applied at the correlator. The transition to FITS-IDI in version
1.5.0 has made correct model accountability possible, and an
accurate representation of the applied delay model is now stored
in two binary tables—the IM table and the MC table. These ta-
bles store the same sampled model polynomials used by DiFX-2
and the applied clock model and can be used by postprocessing
software such as AIPS
15
to accurately make changes to the phase
center of the correlated data set or to correct for antenna position
errors, Earth orientation parameter errors, and the like.
2.7. New Data Monitoring Tools
2.7.1. Autocorrelation Filterbank Spigot
In order to facilitate searches for transient signals, an auto-
correlation “spigot” has been added to DiFX-2. This spigot
supplies the autocorrelations from each antenna at a user-
specified time and frequency resolution, by means of a UDP
(User Datagram Protocol) multicast message. The additional
computational load is negligible, since the antenna autocorrela-
tions are already calculated as a matter of course, and the addi-
tional load of sending the multicast messages is negligible for all
but very short integrations. The messages are sent with a simple
plain-text header, allowing (one or more) analysis programs to
capture, time-order, and inspect what are essentially N indepen-
dent but time-aligned filter-bank data streams, where N is the
number of antennas. This feature has been available since ver-
sion 1.5.1, but is considerably improved in DiFX-2. An example
of the two-dimensional dynamic spectrum that is obtained (for
each antenna) is shown in Figure 2.
At the VLBA, additional functionality has been added to
allow feedback from the analysis programs, allowing them to
request that small time ranges of baseband data be extracted
after the correlation has finished and written to disk elsewhere.
F
IG
.1.—Example of a representative finding field centered on 07
h
45
m
07:270
s
þ 33°40
0
37:52″ (from the FIRST [Faint Images of the Radio Sky at
Twenty cm] survey—http://sundog.stsci.edu/). The bold black ring shows the
31′ primary beam of a 25 m dish at 1600 MHz, and the small white rings show
the individual pencil beams that would be placed on known sources. The pencil-
beam diameter is displayed as 12″, at which point the cumulative time and
bandwidth decorrelation from 0.5 MHz channelization and 4 s averaging
reaches 10%.
15
See http://www.aips.nrao.edu/.
278 DELLER ET AL.
2011 PASP, 123:275–287
