1
Virus detection and identification in minutes using single-particle imaging and deep learning
Nicolas Shiaelis
1,*
, Alexander Tometzki
1
, Leon Peto
2,3
, Andrew McMahon
1
, Christof Hepp
1
, Erica Bickerton
4
,
Cyril Favard
5
, Delphine Muriaux
5,6
, Monique Andersson
3
, Sarah Oakley
3
, Alison Vaughan
2,7
, Philippa C.
Matthews
2
, Nicole Stoesser
2,8
, Derrick Crook
2,7,8
, Achillefs N. Kapanidis
1,*
and Nicole C. Robb
1,9,*
1
Biological Physics Research Group, Clarendon Laboratory, Department of Physics, University of Oxford,
Oxford, OX1 3PU, United Kingdom
2
Nuffield Department of Medicine, University of Oxford, Oxford, OX3 9DU, United Kingdom
3
Department of Microbiology, Oxford University Hospitals NHS Foundation Trust, Oxford, OX3 9DU,
United Kingdom
4
The Pirbright Institute, Ash Road, Pirbright, Woking, Surrey, GU24 0NF, United Kingdom
5
Membrane Domains and Viral Assembly, IRIM, UMR 9004 CNRS & University of Montpellier, 1919, route
de Mende, 34293 Montpellier, France
6
CEMIPAI, UMS 3725 CNRS & University of Montpellier, 1919, route de Mende, 34293 Montpellier,
France
7
NIHR Oxford Biomedical Research Centre, University of Oxford, Oxford, United Kingdom
8
NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance
at the University of Oxford in partnership with Public Health England, University of Oxford, Oxford,
United Kingdom
9
Warwick Medical School, University of Warwick, Coventry, CV4 7AL, United Kingdom
*To whom correspondence should be addressed: Nicole.Robb@warwick.ac.uk,
kapanidis@physics.ox.ac.uk, nicolas.shiaelis@st-annes.ox.ac.uk
Abstract
The increasing frequency and magnitude of viral outbreaks in recent decades, epitomized by the current
COVID-19 pandemic, has resulted in an urgent need for rapid and sensitive viral diagnostic methods. Here,
we present a methodology for virus detection and identification that uses a convolutional neural network
to distinguish between microscopy images of single intact particles of different viruses. Our assay achieves
labeling, imaging and virus identification in less than five minutes and does not require any lysis,
purification or amplification steps. The trained neural network was able to differentiate SARS-CoV-2 from
negative clinical samples, as well as from other common respiratory pathogens such as influenza and
seasonal human coronaviruses, with high accuracy. Single-particle imaging combined with deep learning
offers a promising alternative to traditional viral diagnostic methods, and has the potential for significant
impact.
All rights reserved. No reuse allowed without permission.
perpetuity.
preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in
The copyright holder for thisthis version posted October 17, 2020. ; https://doi.org/10.1101/2020.10.13.20212035doi: medRxiv preprint
NOTE: This preprint reports new research that has not been certified by peer review and should not be used to guide clinical practice.
2
Main
The SARS-CoV-2 betacoronavirus has infected millions of people in 2020, resulting in over a million deaths,
and causing worldwide social and economic disruption. Rapid, sensitive and accurate viral diagnosis is
fundamental to response efforts.
Current SARS-CoV-2 diagnostic methods include nucleic acid amplification tests, antigen detection, and
serology tests
1
. Reverse transcriptase polymerase chain reaction (RT-PCR) is considered the gold standard
for diagnosis; however, RT-PCR takes several hours to provide a result, is restricted to specialized
laboratories (as it requires viral lysis and RNA extraction), and can be limited by supply chain issues.
Isothermal nucleic acid amplification methods, such as loop-mediated isothermal amplification (RT-
LAMP), offer a promising alternative that does not require thermal cycling and can provide results within
an hour
2-7
; however, these methods are still subject to similar supply chain issues as RT-PCR. Rapid (<30
minutes) immunoassay-based antigen-detecting tests exist for some viruses (e.g., influenza), but generally
have low sensitivities
8
. There is thus an urgent need for new viral detection approaches, particularly ones
that can be deployed in non-laboratory settings.
To address this need, we developed a novel diagnostic method that relies on the detection of intact virus
particles using wide-field fluorescence imaging. Our method starts with near-instantaneous labeling of
enveloped viruses via cation-mediated binding of short fluorescent DNAs to the surface of virus particles
9
;
we subsequently surface-immobilise labelled particles, collect diffraction-limited images containing
thousands of labelled particles, and finally use image analysis and machine-learning to identify different
viruses in biological and clinical samples (Fig.1A). Our approach exploits the fact that distinct virus types
and strains have differences in surface chemistry, size, and shape, which in turn affects the fluorophore
distribution over the surface of different viruses. Such differences can be captured by convolutional neural
networks (CNNs), which have been used previously to classify super-resolved microscopy images of
heterogeneous virus populations into particle classes with distinct structural features
10
, and to detect
virus particles in transmission electron microscopy images
11
.
To demonstrate our ability to label, immobilize, and image coronavirus particles, we initially used
infectious bronchitis virus (IBV), an avian coronavirus (CoV). We labelled IBV using strontium chloride and
a mixture of green and red fluorescent DNAs (labelled with Cy3 or Atto647N fluorophores, respectively);
immobilized particles on a chitosan-coated glass slide; and imaged particles using total-internal-reflection
fluorescence microscopy (Sup.Fig.1). Fluorescent labelling was achieved within seconds via a single-step
addition of labelling mixture (see Methods), after which the viruses were immediately immobilized. The
resulting images contained particles with either single green or red fluorescence signals (shown as green
and red particles), as well as colocalised green and red fluorescence signals (shown as yellow particles)
(Fig.1B,C). In a virus-negative control, substantially fewer colocalised signals were observed (Fig.1B,D),
consistent with the fact that single red and green signals arose from free DNA, while the majority of
colocalised signals corresponded to coronavirus particles labelled with both colors. Colocalised signals in
the absence of virus likely occurred due to random coincidence or cation-mediated clustering of DNAs on
the surface. Omission of DNAs resulted in complete loss of the fluorescent signal (Fig.1B, left panels).
Prior to use for machine learning, individual image signals were isolated into bounding boxes (BBXs) using
segmentation of the field of view (FOV) through adaptive filtering. The raw FOVs from the red channel
(Fig.1Ei) were converted into a binary format (Fig.1Eii) and area filtering used to disregard objects with an
area smaller than 10 pixels (single fluorophores) or larger than 100 pixels (aggregates) (Fig.1Eiii). At the
same time, to enrich our sampling for viruses and exclude signals not arising from virus particles, the
All rights reserved. No reuse allowed without permission.
perpetuity.
preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in
The copyright holder for thisthis version posted October 17, 2020. ; https://doi.org/10.1101/2020.10.13.20212035doi: medRxiv preprint
3
location image (showing the green, red and yellow signals from both channels; Fig.1Eiv) was used to
identify colocalised signals (Fig.1Ev). This information was then combined with the signals identified in the
filtered binary image (Fig.1Eiii) to reject signals not meeting the colocalisation condition (Fig.1Evi; cyan
boxes) and retain signals meeting the colocalisation condition (Fig.1Evi-vii; red boxes). The segmentation
was fully automated, allowing each FOV to be processed in ~2 seconds. The mean number of colocalised
BBXs per FOV obtained when IBV was present was ~6-fold higher than when the virus was absent (Fig.1F).
Next, we tested whether our CNN architecture could differentiate between signals observed in virus-
positive and virus-negative samples, as well as between images of different viruses. Many respiratory
viruses, including SARS-CoV-2, influenza and seasonal human coronaviruses (hCoV), exhibit similar early
onset symptoms; it is thus crucial that diagnostic assays can differentiate between these different viruses.
As a proof-of-principle, we fluorescently labelled and imaged IBV and three laboratory-grown influenza A
strains: H3N2 A/Udorn/72 (Udorn), H3N2 A/Aichi/68 (X31), and H1N1 A/PR8/8/34 (PR8) (Fig.2A). These
viruses are similar in size and shape, and cannot be distinguished in diffraction-limited microscope images
of fluorescently labelled particles (Sup.Fig.2A). After image segmentation (Sup.Fig.2B) and examination of
the properties of the resulting BBXs, we observed that the four viruses exhibited small differences in
maximum pixel intensity, area, and semi-major-to-semi-minor-axis-ratio within the BBXs (Fig.2B-D); e.g.,
IBV appears brighter than influenza, whereas Udorn occupies a larger area than the other viruses. Such
features are not easily identifiable by manual analysis, however these and other image features such as
pixel correlations, can be exploited by deep-learning algorithms for classification purposes
12,13
.
To classify different viruses, we constructed a 15-layer CNN (Fig.2E, see legend for detail). Four
independent datasets of each virus strain and a virus-negative control were randomly divided into a
training dataset (70%) and a validation dataset (30%). The datasets used for both training and validation
of the model consisted of data collected from three different days of experiments to enhance the ability
of the trained models to classify data from future datasets. The network was trained using different
combinations of all four viruses and the negative control, using ~3000 BBXs per sample. To validate our
network, we checked if the network could differentiate IBV virus samples from negative controls
consisting of only strontium chloride and DNA; the first data point in the network validation session was
at 50% accuracy (as expected for a completely random classification of objects into two categories),
followed by an initial rapid increase in validation accuracy as the network detected the most obvious
parameters, followed by a slower increase as the number of iterations increased (Sup.Fig.3A). This was
accompanied by a similar decrease in the Loss Function (Sup.Fig.3B); the entire training and validation
process took 12 minutes to complete (Sup.Fig.3C).
Results of the network validation are shown as a confusion matrix, commonly used to visualize
performance measures for classification problems (Fig.2F). The rows correspond to the predicted class
(Output Class), the columns to the true class (Target Class), and the far-right, bottom cell represents the
overall validation accuracy (hereafter, accuracy) of the model for each classified particle. The trained
network was able to differentiate positive and negative IBV samples with high accuracy (91.4%), sensitivity
(91.9%) and specificity (90.9%) (Fig.2G). Of note, these probabilities refer to single virus particles in the
sample and not the whole sample; the probability of correctly identifying a sample with hundreds or
thousands of IBV virus particles will therefore approach 100%.
Next, we tested the network’s ability to distinguish between different virus types and strains by training
the network on data from IBV and influenza Udorn, X31 and PR8 strains. The network easily distinguished
between IBV and influenza, with an accuracy of 95.5% for IBV vs. Udorn (Fig.2H) and 94.3% for IBV vs. PR8
(Sup.Fig.4A). The network was able to differentiate between two strains of influenza (Udorn and X31),
All rights reserved. No reuse allowed without permission.
perpetuity.
preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in
The copyright holder for thisthis version posted October 17, 2020. ; https://doi.org/10.1101/2020.10.13.20212035doi: medRxiv preprint
4
albeit with a lower accuracy (68.8%), reflecting the greater homogeneity between strains of the same
virus (Sup.Fig.4B). The network was also able to distinguish between IBV and a pooled dataset consisting
of the virus-negative control and three influenza strains (92.2%) (Sup.Fig.4C).
Having validated our assay on laboratory-grown viruses, we next assessed clinical samples. Throat swabs
from patients negative for virus, or positive for SARS-CoV-2, seasonal hCoVs (OC43, HKU1 or NL63) or
human influenza A (as determined by RT-PCR) were inactivated with formaldehyde before being labelled
and immobilised (see Methods). Images were captured on four different days, with data from days 1-3
used to train and validate the network (Table1, Sup.Fig.5). The trained network was able to distinguish
between virus-positive and virus-negative samples with excellent accuracy, distinguishing between SARS-
CoV-2-positive and negative BBXs with an accuracy of ~70% (Fig.3A), between Flu A and negative BBXs
with an accuracy of ~87% (Fig.3B), and between seasonal hCoV and negative samples with an accuracy of
~78% (Sup.Fig.6A). The decrease in accuracy (compared to the laboratory-grown viruses) reflected the
greater heterogeneity and complexity of clinical samples (e.g., varied storage conditions, wide range of
virus concentrations, presence of residual cellular material, different sampling techniques). In spite of
these issues, the network could also distinguish SARS-CoV-2 from seasonal hCoVs with a validation
accuracy of ~73% (Fig.3C), and SARS-CoV-2 from Flu A with a validation accuracy of ~70% (Sup.Fig.6B),
potentially useful in diagnosing co-circulating infections. Having trained and validated the network, data
acquired on day 4 were then used to test the ability of the CNN to categorise the same samples imaged
on a different day (Sup.Table1). The network was able to classify more than 50% of BBXs correctly in 8 of
10 samples tested for seasonal hCoVs vs. negative, and in 8 of 9 samples tested for SARS-CoV-2 vs. hCoV;
results can be further improved by increasing the number of samples used for training.
We then tested our network’s ability to diagnose independent clinical samples never seen before by the
trained network; blind positive or negative samples were imaged on day 4 and assessed by the trained
network within a few seconds. The output number of BBXs classified as positive or negative for each
sample, and their associated probability values, were compared to the cumulative probability distribution
functions (PDFs) expected for either positive or negative samples. When a SARS-CoV-2 RT-PCR-negative
sample was analysed by the trained network, the large majority of BBXs were classified as negative (125
vs 29), and the associated probability of the sample being negative was overwhelmingly higher than the
probability of it being positive (0.97 vs 1.64x10
-63
) (Fig.3D), as expected. Similarly, when a SARS-CoV-2 RT-
PCR-positive sample was analysed, the large majority of BBXs were classified as positive (148 vs 25) and
the associated probability of the sample being positive was overwhelmingly higher than the probability of
it being negative (1.00 vs 2.49x10
-52
) (Fig.3E). Similar results were also obtained for an hCoV OC43 clinical
sample (Sup.Fig.7).
We also estimated the limit of detection (LOD) of our assay by testing the ability of the network to
accurately detect increasing IBV concentrations (Sup.Fig.8). Images were analysed by the trained network,
and the number of particles classified as positive was fitted linearly with increasing virus concentration.
The LOD was estimated as 6x10
4
PFU/mL, a sensitivity that, as expected, was significantly lower than that
of amplification-based methods like RT-PCR (~10
2
PFU/mL
14
); however, we anticipate that the sensitivity
should increase substantially with better immobilisation, pre-concentration of virus particles, optimised
labelling, and reduced surface binding of free DNA.
Our work demonstrates how single-particle fluorescence microscopy combined with deep learning can
help to rapidly detect and classify viruses, including coronaviruses. Our approach of instantaneous
labelling, rapid automated imaging, pre-processing and deep learning classifies viruses within minutes,
avoiding the need for viral lysis or amplification and the associated cost, tedium and supply-chain issues.
All rights reserved. No reuse allowed without permission.
perpetuity.
preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in
The copyright holder for thisthis version posted October 17, 2020. ; https://doi.org/10.1101/2020.10.13.20212035doi: medRxiv preprint
5
The non-specific detection of intact viral particles (rather than genome fragments) can report directly on
infectivity, and has the advantages of speed (results within 2-5 minutes), the ability to detect multiple
virus types in a single labelling step, and robustness against potential mutations in the viral genome. Our
algorithms are extremely versatile and can be trained to differentiate between many different viruses,
independently of how they are labelled, immobilized and imaged. Given its simplicity and rapid nature,
our technology could also be used outside of specialized laboratories, such as in airports, workplaces and
care homes. These unique capabilities should enable extremely rapid, mobile, and real-time analysis of
patient and community samples during pandemic situations.
Methods
Laboratory grown virus strains and DNAs. The influenza strains (H1N1 A/Puerto Rico/8/1934 (PR8), H3N2
A/Udorn/72 (Udorn) and H3N2 A/Aichi/68 (X31)) used in this study have been described previously
9
.
Briefly, PR8 and Udorn were grown in Madin-Darby bovine kidney (MDBK) or Madin-Darby canine kidney
(MDCK) cells and X31 was grown in embryonated chicken eggs. The cell culture supernatant or allantoic
fluid was collected and the viruses were titred by plaque assay. Titres of PR8, Udorn and X31 were
1.05x10
8
plaque forming units (PFU)/mL, 1.0x10
7
PFU/ mL and 4.5x10
8
PFU/mL respectively. The
coronavirus IBV (Beau-R strain)
15
was grown in embryonated chicken eggs and titred by plaque assay
(1x10
6
PFU/mL). Viruses were inactivated by addition of 2% formaldehyde before use.
Single-stranded oligonucleotides labelled with either red or green dyes were purchased from IBA
(Germany). The ‘red’ DNA was modified at the 5’ end with ATTO647N (5’
ACAGCACCACAGACCACCCGCGGATGCCGGTCCCTACGCGTCGCTGTCACGCTGGCTGTTTGTCTTCCTGCC 3’)
and the ‘green’ DNA was modified at the 3’ end with Cy3 (5’
GGGTTTGGGTTGGGTTGGGTTTTTGGGTTTGGGTTGGGTTGGGAAAAA 3’).
Clinical samples. Ethical approval was obtained for the use of anonymised oro- or nasopharyngeal
specimens from patients for the diagnosis of influenza and other respiratory pathogens, including SARS-
CoV-2 (North West-Greater Manchester South Research Ethics Committee [REC], REC Ref:19/NW/0730).
Specimens were maintained in Copan Universal Transport Medium (UTM) before being inactivated in a
4% final concentration of formaldehyde (Pierce) for 30 minutes at room temperature. Samples were
confirmed as SARS-CoV-2-positive or negative using either the Public Health England 2019-nCoV real-time
RT-PCR RdRp gene assay or RealStar SARS-CoV-2 RT-PCR Kit (Altona diagnostics). Testing for other
respiratory pathogens and sub-typing of seasonal human coronavirus (hCoV) samples as OC43, HKU1 or
NL63 strains was conducted using the BioFire FilmArray Respiratory Panel (Biomerieux, Marcy-L’Etoile,
France) and Cepheid Xpert Xpress Flu/RSV (Cepheid, Sunnyvale, CA, USA).
Sample preparation. Glass slides were treated with 0.015 mg/mL chitosan (a linear polysaccharide) in 0.1
M acetic acid for 30 min before being washed thrice with MilliQ water. Unless otherwise stated, virus
stocks (typically 10 µL) were diluted in 0.23 M CaCl
2
or SrCl
2
(as described in the figure legends) and 1 nM
of each fluorescently-labelled DNA in a final volume of 20 µL, before being added to the slide surface.
Virus labelling with CaCl
2
has been described previously
9
; SrCl
2
provides similar results. For laboratory
grown virus stocks, negatives were taken using Minimal Essential Media (Gibco) or allantoic fluid from
uninfected eggs in place of the virus.
Imaging. Images were captured using a wide-field fluorescence microscope, as previously described
9
. The
sample was imaged using total internal reflection fluorescence (TIRF) microscopy. The laser illumination
was focused at a typical angle of 53° with respect to the normal. Movies of 5 frames per field of view (FOV)
(measuring 75 x 49 µm) were taken at a frequency of 33 Hz and exposure time of 30 ms, with laser
All rights reserved. No reuse allowed without permission.
perpetuity.
preprint (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in
The copyright holder for thisthis version posted October 17, 2020. ; https://doi.org/10.1101/2020.10.13.20212035doi: medRxiv preprint